[26.04_linux-nvidia-bos] Backport CXL reset save/restore support

[kobak2026]

BugLink: https://bugs.launchpad.net/ubuntu/+source/linux-nvidia-7.0/+bug/2153819

Summary

Backport Srirangan's CXL reset plus DVSEC/HDM save-restore series for the CXL Type-2 stack.

This series:

• Reverts the older monolithic CXL reset implementation.
• Adds PCI CXL DVSEC and HDM decoder state save/restore across reset.
• Adds CXL reset orchestration, sibling coordination, memory quiesce/cache flush handling, and the cxl_reset PCI sysfs interface.
• Adds ABI documentation for /sys/bus/pci/devices/.../cxl_reset.
• Preserves PCI_DVSEC_CXL_CACHE_CAPABLE for the existing CXL.cache ATS dependency.

Dependency Note

This PR depends on the CXL Type-2 base branch and is intended to be reviewed/applied after that branch lands.

The branch is currently stacked locally on that dependency; the dependency branch is not pushed to NVIDIA/NV-Kernels.

Validation

• Full kernel build passed on the remote validation host.
• Built and booted validation kernel: 7.0.0-dgx16137-cxlreset.
• Read-only CXL reset baseline passed for all four target BDFs.
• Destructive cxl_reset validation passed on all four target BDFs.
• CXLCtl and captured CXL range fields were preserved across reset.
• CXLSta2 reported ResetComplete+ ResetError-.
• dmesg deltas were clean; no host/device recovery was required.
• After rebasing onto the refreshed dependency branch, git diff --check is clean.

[github-actions]

PR Validation Report

Patchscan ✅ No Missing Fixes

All cherry-picked commits checked — no missing upstream fixes found.

PR Lint ❌ Errors found

Details

Checking 38 commits...

Cherry-pick digest:
E: 52975ec822b9 ("NVIDIA: VR: SAUCE: PCI: Add CXL DVSEC re"): backport trailer order: ORDER: move [Name: note] before the backporter Signed-off-by and after (backported from ...)
E: 82adedeffcd7 ("NVIDIA: VR: SAUCE: cxl: Move HDM decoder"): backport trailer order: ORDER: move [Name: note] before the backporter Signed-off-by and after (backported from ...)
┌──────────────┬──────────────────────────────────────────────────────────────────┬────────────┬─────────┬───────────────────────────┐
│ Local        │ Referenced upstream / Patch subject                              │ Patch-ID   │ Subject │ SoB chain                 │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 9bf30cf9cb99 │ [SAUCE] documentation: abi: add cxl pci cxl_reset sysfs attribut │ N/A        │ N/A     │ smadhava, jan, bfigg, kob │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ c6db6ce8e793 │ [SAUCE] cxl: add cxl_reset sysfs interface for pci devices       │ N/A        │ N/A     │ smadhava, jan, bfigg, kob │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 6e9e72b4b2a2 │ [SAUCE] cxl: add cxl dvsec reset sequence and flow orchestration │ N/A        │ N/A     │ smadhava, jan, bfigg, kob │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 3a245b9b22c3 │ [SAUCE] cxl: add multi-function sibling coordination for cxl res │ N/A        │ N/A     │ smadhava, jan, bfigg, kob │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ baa527d0e4a3 │ [SAUCE] cxl: add memory offlining and cache flush helpers        │ N/A        │ N/A     │ smadhava, jan, bfigg, kob │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ d1c9d1b51460 │ [SAUCE] pci: export pci_dev_save_and_disable() and pci_dev_resto │ N/A        │ N/A     │ smadhava, jan, bfigg, kob │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 52975ec822b9 │ pci: add cxl dvsec reset and capability register definitions     │ noted      │ found   │ ORDER: move [Name: note]  │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 42ade8ec191c │ [SAUCE] pci: add hdm decoder state save/restore                  │ N/A        │ N/A     │ smadhava, jan, bfigg, kob │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 969356c44c97 │ [SAUCE] pci: add cxl dvsec state save/restore across resets      │ N/A        │ N/A     │ smadhava, jan, bfigg, kob │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 9a09ee09f074 │ [SAUCE] pci: add virtual extended cap save buffer for cxl state  │ N/A        │ N/A     │ smadhava, jan, bfigg, kob │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 82adedeffcd7 │ cxl: move hdm decoder and register map definitions to include/cx │ no-match   │ not fou │ ORDER: move [Name: note]  │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 07705ee1b952 │ [SAUCE] pci: add cxl dvsec control, lock, and range register def │ N/A        │ N/A     │ smadhava, jan, bfigg, kob │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ e90b1ee9e1bf │ [SAUCE] revert "nvidia: vr: sauce: cxl: add support for cxl rese │ N/A        │ N/A     │ kobak                     │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 5268ad904265 │ [SAUCE] [config] add pci_cxl annotation for cxl state save/resto │ N/A        │ N/A     │ jan, bfigg, kobak         │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ e3aadf637883 │ [SAUCE] [config] enable cxl dax and kmem built-in for cxl memory │ N/A        │ N/A     │ jan, bfigg, kobak         │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ a2eba9dce2b6 │ [SAUCE] [config] cxl config annotations for type-2 device and ra │ N/A        │ N/A     │ jan, bfigg, kobak         │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 7a6aebb18177 │ cxl/region: support multi-level interleaving with smaller granul │ noted      │ found   │ ok, backporter: kobak     │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 8cb4feab45af │ [SAUCE] dax/hmem: reintroduce soft reserved ranges back into the │ N/A        │ N/A     │ schofiel, lizhijia, Koral │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ c0db8c8c8fda │ [SAUCE] dax/hmem, cxl: defer and resolve ownership of soft reser │ N/A        │ N/A     │ williams, Koralaha, kobak │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 2e433461671f │ [SAUCE] dax: add deferred-work helpers for dax_hmem and dax_cxl  │ N/A        │ N/A     │ Koralaha, kobak           │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 3a6c6f56dc2c │ cxl/region: add helper to check soft reserved containment by cxl │ noted      │ found   │ ok, backporter: kobak     │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 3a52fc67323f │ [SAUCE] dax: track all dax_region allocations under a global res │ N/A        │ N/A     │ williams, Koralaha, kobak │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 79356d9fd78d │ [SAUCE] dax/cxl, hmem: initialize hmem early and defer dax_cxl b │ N/A        │ N/A     │ williams, Koralaha, kobak │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ d8ce89e1116e │ [SAUCE] cxl/region: skip decoder reset on detach for autodiscove │ N/A        │ N/A     │ Koralaha, kobak           │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 77ac7f9272ba │ [SAUCE] dax/hmem: gate soft reserved deferral on dev_dax_cxl     │ N/A        │ N/A     │ williams, Koralaha, kobak │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 1d651dbee5bc │ [SAUCE] dax/hmem: request cxl_acpi and cxl_pci before walking so │ N/A        │ N/A     │ williams, Koralaha, kobak │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 0fdb2010fcfc │ sfc: support pio mapping based on cxl                            │ noted      │ found   │ ok, backporter: kobak     │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ d6acd1ddf9f6 │ [SAUCE] cxl: avoid dax creation for accelerators                 │ N/A        │ N/A     │ alucerop, kobak           │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 388dc8c0c90a │ cxl: attach region to an accelerator/type2 memdev                │ noted      │ found   │ ok, backporter: kobak     │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 9c9123639c35 │ [SAUCE] sfc: create type2 cxl memdev                             │ N/A        │ N/A     │ alucerop, kobak           │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 79654508fdaf │ [SAUCE] cxl: prepare memdev creation for type2                   │ N/A        │ N/A     │ alucerop, kobak           │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ cb8c41246ed1 │ [SAUCE] cxl/sfc: initialize dpa without a mailbox                │ N/A        │ N/A     │ alucerop, kobak           │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ ba9972ccf965 │ cxl/sfc: map cxl regs                                            │ noted      │ found   │ ok, backporter: kobak     │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 513a2a51e219 │ [SAUCE] sfc: add cxl support                                     │ N/A        │ N/A     │ alucerop, kobak           │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ d3059e8eab00 │ d537d953c478 cxl/pci: Remove redundant cxl_pci_find_port() call  │ match      │ match   │ preserved + kobak added   │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 8b61bd238bbb │ 58f28930c7fb cxl: Move pci generic code from cxl_pci to core/cxl │ match      │ match   │ preserved + kobak added   │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 2bc7fd2fd558 │ 005869886d1d cxl: export internal structs for external Type2 dri │ match      │ match   │ preserved + kobak added   │
├──────────────┼──────────────────────────────────────────────────────────────────┼────────────┼─────────┼───────────────────────────┤
│ 645dd5c6be9f │ 9a775c07bb04 cxl: support Type2 when initializing cxl_dev_state  │ match      │ match   │ preserved + kobak added   │
└──────────────┴──────────────────────────────────────────────────────────────────┴────────────┴─────────┴───────────────────────────┘

Lint: all checks passed.

[nirmoy]

Boro review

Latest watcher review: open review

Head: 9bf30cf9cb99

This comment is maintained by nv-pr-bot. It is updated when the GitHub watcher publishes a newer review.

[nvmochs]

@kobak2026 Two comments...

0d3aece NVIDIA: VR: SAUCE: cxl: Add CXL DVSEC reset sequence and flow orchestration

cxl_reset_prepare_memdev() only checks !endpoint before walking &endpoint->dev, but cxlmd->endpoint is initialized to ERR_PTR(-ENXIO) until endpoint attach succeeds. A memdev can exist with endpoint still an error pointer, so writing 1 to cxl_reset can dereference an encoded error pointer. Please match cxl_reset_flush_cpu_caches() and use if (!endpoint || IS_ERR(endpoint)) return 0;.

---

fb0ce0b NVIDIA: SAUCE: Revert "NVIDIA: VR: SAUCE: cxl: add support for cxl reset\”

Is there a reason why the original commit’s SHA is absent from the commit message?

Nit: Why are there backslashes in the title?

[clsotog]

Same comment as Matt for fb0ce0b

And extra comment
drivers/cxl/core/pci.c:1126-1137 / drivers/cxl/core/pci.c:1154-1155
If krealloc() fails while collecting sibling CXL functions, the callback returns 1, but pci_walk_bus() returns void and discards that failure. The reset then proceeds at drivers/cxl/core/pci.c:1336-1339 with only a partial sibling list quiesced. For a reset that affects all CXL cache/mem functions, that can leave an active sibling running
through the reset.

[kobak2026]

@nirmoy
Pushback itmes

• drivers/cxl/mem.c: to_cxl_memdev_state() already checks cxlds->type != CXL_DEVTYPE_CLASSMEM and returns NULL before container_of(). The poison debugfs/sysfs guards are effective for DEVMEM devices.
• drivers/net/ethernet/sfc/efx_cxl.c: devm_cxl_dev_state_create() wraps _devm_cxl_dev_state_create(), which uses devm_kzalloc() and returns NULL, not ERR_PTR; if (!cxl) is correct.
• drivers/net/ethernet/sfc/efx_cxl.c: the same allocation is devm-managed, so early returns after allocation do not leak cxl.
• drivers/net/ethernet/sfc/efx_cxl.c: I did not find a runtime efx_ef10_dimension_resources() path racing CXL region removal; current unmap clears queue piobuf under TX locks. If there is a concrete reset/resource-redimension caller, point me at it and we can lock that path.
• drivers/cxl/core/region.c: devm_cxl_add_region() is only called with CXL_DECODER_HOSTONLYMEM in the current tree, so the non-HOSTONLYMEM no-attach leak is not reachable. The real attach failure cleanup path was fixed.

[kobak2026]

@jamieNguyenNVIDIA @clsotog @nvmochs
Updated the branch for the latest review feedback.

Changes:

• Propagated sibling collection allocation failures so reset aborts instead of proceeding with a partial sibling list.
• Added the missing ERR_PTR-safe endpoint handling and kept the endpoint lock held across reset preparation.
• Fixed the NULL callback path in CXL memory offlining.
• Adjusted the region attach cleanup path to avoid taking the region write lock while the read lock is held.
• Updated the revert commit message with the original reverted SHA and cleaned up the subject.

Validation:

• Full remote kernel build passed after these fixes.
• kernelrelease: 7.0.0-dgx16137-cxlreset
• No install/reboot/reset rerun was done for this update.

[jamieNguyenNVIDIA]

@kobak2026: The code changes are looking good now.

The only remaining issues I can spot have to do with these 8 commit messages:

8 commits missing provenance trailers
                                                                                                                        
  777eae3dfe6e  Dan Williams         dax/hmem: Request cxl_acpi and cxl_pci before walking Soft Reserved
  abe65b93d89d  Dan Williams         dax/hmem: Gate Soft Reserved deferral on DEV_DAX_CXL
  cacbb42691fc  Smita Koralahalli    cxl/region: Skip decoder reset on detach for autodiscovered regions
  300526c8b348  Dan Williams         dax/cxl, hmem: Initialize hmem early and defer dax_cxl binding                     
  6b90d4fa5cdd  Smita Koralahalli    dax: Track all dax_region allocations under a global resource tree                 
  6001a57f1ad5  Smita Koralahalli    dax: Add deferred-work helpers for dax_hmem and dax_cxl coordination
  2dcd553bf5df  Smita Koralahalli    dax/hmem, cxl: Defer and resolve ownership of Soft Reserved
  827f5acfea4f  Smita Koralahalli    dax/hmem: Reintroduce Soft Reserved ranges back into the iomem tree

These need a (backported/cherry-picked from ) trailer added.

[kobak2026]

@jamieNguyenNVIDIA thanks. fixed.

[jamieNguyenNVIDIA]

Acked-by: Jamie Nguyen <jamien@nvidia.com>

[clsotog]

Acked-by: Carol L Soto <csoto@nvidia.com>

[nvmochs]

@kobak2026 Thanks for addressing my feedback from previous review.

I re-reviewed the latest changes with codex and it is flagging the locking changes that were made in response to comments from Carol and Jamie's feedback that can now trigger a deadlock.

Here are the details:

There are two separate changes in drivers/cxl/core/pci.c. Here is the expanded before/after.

Before

Old cxl_do_reset() in 1877f75 looked like this:

  memdev = bus_find_device(...);
  if (memdev) {
          cxlmd = to_cxl_memdev(memdev);
          guard(device)(&cxlmd->dev);
  } /* cxlmd->dev lock is released here */

  mutex_lock(&cxl_reset_mutex);
  pci_dev_lock(pdev);

  if (cxlmd) {
          rc = cxl_reset_prepare_memdev(cxlmd);
          if (rc)
                  goto out_unlock;

          cxl_reset_flush_cpu_caches(cxlmd);
  }

  pci_dev_save_and_disable(pdev);
  cxl_pci_functions_reset_prepare(&ctx);

  rc = cxl_dev_reset(pdev, dvsec);

  cxl_pci_functions_reset_done(&ctx);
  pci_dev_restore(pdev);

Two key points:

• The guard(device)(&cxlmd->dev) was scoped only to the if (memdev) { ... } block, so it was released before reset began.
• Memory offlining / cache flush happened before pci_dev_save_and_disable(pdev).

After

New code in b9745d6 split this into cxl_do_reset() and __cxl_do_reset():

  static int cxl_do_reset(struct pci_dev *pdev)
  {
          ...
          struct cxl_memdev *cxlmd = to_cxl_memdev(memdev);

          guard(device)(&cxlmd->dev);
          return __cxl_do_reset(pdev, cxlmd, dvsec);
  }

Because this guard is in function scope, cxlmd->dev stays locked until __cxl_do_reset() returns.

  mutex_lock(&cxl_reset_mutex);
  pci_dev_lock(pdev);

  pci_dev_save_and_disable(pdev);
  rc = cxl_pci_functions_reset_prepare(&ctx);
  if (rc)
          goto out_restore;

  if (cxlmd) {
          rc = cxl_reset_prepare_memdev(cxlmd);
          if (rc)
                  goto out_reset_done;

          cxl_reset_flush_cpu_caches(cxlmd);
  }

  rc = cxl_dev_reset(pdev, dvsec);

  out_reset_done:
          cxl_pci_functions_reset_done(&ctx);

  out_restore:
          pci_dev_restore(pdev);

Issue 1: self-deadlock

The new outer guard(device)(&cxlmd->dev) is still held when pci_dev_restore(pdev) runs.

pci_dev_restore() calls the PCI driver’s reset_done() callback:

if (err_handler && err_handler->reset_done)
err_handler->reset_done(dev);

For the CXL PCI driver, that is cxl_reset_done(), which does:

guard(device)(&cxlmd->dev);

So the same thread already holds cxlmd->dev from cxl_do_reset(), then tries to lock it again inside cxl_reset_done(). That deadlocks.

Issue 2: reset ordering

The old flow was:

offline memory
flush CPU caches
disable PCI function
disable siblings
reset
restore

The new flow is:

disable PCI function
disable siblings
offline memory
flush CPU caches
reset
restore

The problem is that pci_dev_save_and_disable(pdev) clears PCI Command, including memory decode and bus master. That now happens before cxl_reset_prepare_memdev() tries to offline CXL-backed memory. If that memory is still online, this is backwards: the CXL memory path should be quiesced before disabling/resetting the device that backs it.

So the issue resides in the new refactor around cxl_do_reset() / __cxl_do_reset() in drivers/cxl/core/pci.c: Koba fixed the endpoint race by extending the memdev lock, but extended it too far, and also moved pci_dev_save_and_disable() ahead of memory preparation.

Instead, the memdev lock should be kept only around the endpoint-dependent preparation, not around the entire reset/restore flow.

A better shape would be:

  mutex_lock(&cxl_reset_mutex);
  pci_dev_lock(pdev);

  if (cxlmd) {
          guard(device)(&cxlmd->dev);

          rc = cxl_reset_prepare_memdev(cxlmd);
          if (rc)
                  goto out_unlock;

          cxl_reset_flush_cpu_caches(cxlmd);
  } /* cxlmd->dev lock releases here */

  pci_dev_save_and_disable(pdev);

  rc = cxl_pci_functions_reset_prepare(&ctx);
  if (rc)
          goto out_restore;

  rc = cxl_dev_reset(pdev, dvsec);

  cxl_pci_functions_reset_done(&ctx);

  out_restore:
          pci_dev_restore(pdev);

  out_unlock:
          pci_dev_unlock(pdev);
          mutex_unlock(&cxl_reset_mutex);

But one more detail matters: if sibling prepare succeeds, cxl_pci_functions_reset_done() must run. So the final structure should track whether siblings were prepared:

  bool siblings_prepared = false;

  mutex_lock(&cxl_reset_mutex);
  pci_dev_lock(pdev);

  if (cxlmd) {
          guard(device)(&cxlmd->dev);

          rc = cxl_reset_prepare_memdev(cxlmd);
          if (rc)
                  goto out_unlock;

          cxl_reset_flush_cpu_caches(cxlmd);
  }

  pci_dev_save_and_disable(pdev);

  rc = cxl_pci_functions_reset_prepare(&ctx);
  if (rc)
          goto out_restore;

  siblings_prepared = true;

  rc = cxl_dev_reset(pdev, dvsec);

  if (siblings_prepared)
          cxl_pci_functions_reset_done(&ctx);

  out_restore:
          pci_dev_restore(pdev);

  out_unlock:
          pci_dev_unlock(pdev);
          mutex_unlock(&cxl_reset_mutex);

Conceptually:

• Keep cxlmd->dev locked while reading/walking cxlmd->endpoint.
• Release cxlmd->dev before pci_dev_restore(), because reset_done() may lock it.
• Keep memory offline/cache flush before pci_dev_save_and_disable().
• Still propagate sibling collection failure and abort before reset.
• Restore the target PCI function on any path after pci_dev_save_and_disable().

So the fix is not “don’t hold the memdev lock”; it is “hold it only across cxl_reset_prepare_memdev() and cxl_reset_flush_cpu_caches(), then release it before PCI restore callbacks run.”

[kobak2026]

@nvmochs thanks
Fixed in the latest push. I folded the change into the CXL reset flow commit.
Summary:

• Scoped the cxlmd->dev lock only around endpoint-dependent memory preparation
  and CPU cache flush.
• Released that lock before pci_dev_restore() can invoke the CXL
  reset_done() callback.
• Restored the reset ordering so CXL memory offlining/cache flush happens before
  pci_dev_save_and_disable().
• Kept sibling reset cleanup conditional on successful sibling prepare.
  Validation:
• Pre-push arm64 nvidia-bos whole-kernel build passed on the pushed head.

[jamieNguyenNVIDIA]

My ACK still stands:

Acked-by: Jamie Nguyen <jamien@nvidia.com>

[clsotog]

Acked-by: Carol L Soto <csoto@nvidia.com>

[kobak2026]

@nvmochs I rebased to PR1

Verified your finding. cxlmd->endpoint can still be
ERR_PTR(-ENXIO), and cxl_reset_done() was only checking for
non-NULL before calling cxl_endpoint_decoder_reset_detected().
That made the restore/reset_done path still vulnerable even
after the reset prepare/cache flush paths were fixed.

Fixed in the reset orchestration commit by:

• reading cxlmd->endpoint under guard(device)(&cxlmd->dev)
• returning early on !endpoint || IS_ERR(endpoint)
• only calling cxl_endpoint_decoder_reset_detected() for a
valid endpoint

I also amended the related commit description with:

[koba: Guard reset_done() against NULL/ERR_PTR memdev
endpoints before decoder reset detection.]

Validation:

• focused static guard/order check: PASS
• git diff --check: PASS
• scripts/checkpatch.pl --strict --terse: PASS
• remote arm64 nvidia-bos full build: PASS

Pushed in 2462338.

[jamieNguyenNVIDIA]

Re-adding my ACK. Matt's comment looks to be addressed in v4.

Acked-by: Jamie Nguyen <jamien@nvidia.com>

[nvmochs]

@kobak2026

Looks like these were mistakenly left in during the rebase?

7d436d86bf5a NVIDIA: SAUCE: dax/hmem, cxl: Defer and resolve ownership of Soft Reserved memory ranges
bd27785772fa NVIDIA: SAUCE: dax: Add deferred-work helpers for dax_hmem and dax_cxl coordination

[jamieNguyenNVIDIA]

@kobak2026

Looks like these were mistakenly left in during the rebase?

7d436d86bf5a NVIDIA: SAUCE: dax/hmem, cxl: Defer and resolve ownership of Soft Reserved memory ranges
bd27785772fa NVIDIA: SAUCE: dax: Add deferred-work helpers for dax_hmem and dax_cxl coordination

I believe that was on purpose. Both of these are included in pr#426 (the dependency series).

[nvmochs]

@kobak2026
Looks like these were mistakenly left in during the rebase?

7d436d86bf5a NVIDIA: SAUCE: dax/hmem, cxl: Defer and resolve ownership of Soft Reserved memory ranges
bd27785772fa NVIDIA: SAUCE: dax: Add deferred-work helpers for dax_hmem and dax_cxl coordination

I believe that was on purpose. Both of these are included in pr#426 (the dependency series).

Here is what I see...

2462338e3d1e (kobanvk/bug-DGX-16137/cxl-backport-26.04-bos-nvpr) NVIDIA: VR: SAUCE: Documentation: ABI: Add CXL PCI cxl_reset sysfs attribute
7c745e181a30 NVIDIA: VR: SAUCE: cxl: Add cxl_reset sysfs interface for PCI devices
31a7fb4259aa NVIDIA: VR: SAUCE: cxl: Add CXL DVSEC reset sequence and flow orchestration
2ce02b595597 NVIDIA: VR: SAUCE: cxl: Add multi-function sibling coordination for CXL reset
d6ae2502e342 NVIDIA: VR: SAUCE: cxl: Add memory offlining and cache flush helpers
72d3d93f4c9f NVIDIA: VR: SAUCE: PCI: Export pci_dev_save_and_disable() and pci_dev_restore()
d62f7889f5d6 NVIDIA: VR: SAUCE: PCI: Add CXL DVSEC reset and capability register definitions
154db0c73a8e NVIDIA: VR: SAUCE: PCI: Add HDM decoder state save/restore
ff45db1361fb NVIDIA: VR: SAUCE: PCI: Add cxl DVSEC state save/restore across resets
5540bcc62acb NVIDIA: VR: SAUCE: PCI: Add virtual extended cap save buffer for CXL state
b482b6992a86 NVIDIA: VR: SAUCE: cxl: Move HDM decoder and register map definitions to include/cxl/cxl.h
b89497a0d17a NVIDIA: VR: SAUCE: PCI: Add CXL DVSEC control, lock, and range register definitions
1fef6831b64d NVIDIA: SAUCE: Revert "NVIDIA: VR: SAUCE: cxl: add support for cxl reset"


7d436d86bf5a NVIDIA: SAUCE: dax/hmem, cxl: Defer and resolve ownership of Soft Reserved memory ranges
bd27785772fa NVIDIA: SAUCE: dax: Add deferred-work helpers for dax_hmem and dax_cxl coordination

^^ These two commits are already present in the dependency series below (i.e. they exist twice on this branch)


5268ad904265 (kobanvk/bug-DGX-16136/cxl-backport-26.04-bos) NVIDIA: VR: SAUCE: [Config] Add PCI_CXL annotation for CXL state save/restore
e3aadf637883 NVIDIA: VR: SAUCE: [Config] Enable CXL DAX and KMEM built-in for CXL memory access
a2eba9dce2b6 NVIDIA: VR: SAUCE: [Config] CXL config annotations for Type-2 device and RAS support
7a6aebb18177 NVIDIA: VR: SAUCE: cxl/region: Support multi-level interleaving with smaller granularities for lower levels
8cb4feab45af NVIDIA: VR: SAUCE: dax/hmem: Reintroduce Soft Reserved ranges back into the iomem tree
c0db8c8c8fda NVIDIA: VR: SAUCE: dax/hmem, cxl: Defer and resolve ownership of Soft Reserved memory ranges
2e433461671f NVIDIA: VR: SAUCE: dax: Add deferred-work helpers for dax_hmem and dax_cxl coordination
3a6c6f56dc2c NVIDIA: VR: SAUCE: cxl/region: Add helper to check Soft Reserved containment by CXL regions
3a52fc67323f NVIDIA: VR: SAUCE: dax: Track all dax_region allocations under a global resource tree
79356d9fd78d NVIDIA: VR: SAUCE: dax/cxl, hmem: Initialize hmem early and defer dax_cxl binding
d8ce89e1116e NVIDIA: VR: SAUCE: cxl/region: Skip decoder reset on detach for autodiscovered regions
77ac7f9272ba NVIDIA: VR: SAUCE: dax/hmem: Gate Soft Reserved deferral on DEV_DAX_CXL
1d651dbee5bc NVIDIA: VR: SAUCE: dax/hmem: Request cxl_acpi and cxl_pci before walking Soft Reserved ranges
0fdb2010fcfc NVIDIA: VR: SAUCE: sfc: support pio mapping based on cxl
d6acd1ddf9f6 NVIDIA: VR: SAUCE: cxl: Avoid dax creation for accelerators
388dc8c0c90a NVIDIA: VR: SAUCE: cxl: attach region to an accelerator/type2 memdev
9c9123639c35 NVIDIA: VR: SAUCE: sfc: create type2 cxl memdev
79654508fdaf NVIDIA: VR: SAUCE: cxl: Prepare memdev creation for type2
cb8c41246ed1 NVIDIA: VR: SAUCE: cxl/sfc: Initialize dpa without a mailbox
ba9972ccf965 NVIDIA: VR: SAUCE: cxl/sfc: Map cxl regs
513a2a51e219 NVIDIA: VR: SAUCE: sfc: add cxl support
d3059e8eab00 cxl/pci: Remove redundant cxl_pci_find_port() call
8b61bd238bbb cxl: Move pci generic code from cxl_pci to core/cxl_pci
2bc7fd2fd558 cxl: export internal structs for external Type2 drivers
645dd5c6be9f cxl: support Type2 when initializing cxl_dev_state

It's not that big of a deal, but we'll need to be clear to Brad which commits to pick from this PR (he can't just blindly apply all the commits).

No issues with the recent updates to the reset and save/restore patches. I verified that it resolves my last finding.

Acked-by: Matthew R. Ochs <mochs@nvidia.com>

[nvmochs]

Merged, closing PR.

28b628e1e58c (nresolute/nvidia-bos-next) NVIDIA: VR: SAUCE: Documentation: ABI: Add CXL PCI cxl_reset sysfs attribute
c46d624f2a73 NVIDIA: VR: SAUCE: cxl: Add cxl_reset sysfs interface for PCI devices
0b863ed53738 NVIDIA: VR: SAUCE: cxl: Add CXL DVSEC reset sequence and flow orchestration
d1153b9e307c NVIDIA: VR: SAUCE: cxl: Add multi-function sibling coordination for CXL reset
14b5b2f40916 NVIDIA: VR: SAUCE: cxl: Add memory offlining and cache flush helpers
dec4b0067323 NVIDIA: VR: SAUCE: PCI: Export pci_dev_save_and_disable() and pci_dev_restore()
e52f4e02aa42 NVIDIA: VR: SAUCE: PCI: Add CXL DVSEC reset and capability register definitions
177b561c35aa NVIDIA: VR: SAUCE: PCI: Add HDM decoder state save/restore
dfd1a8f1e61e NVIDIA: VR: SAUCE: PCI: Add cxl DVSEC state save/restore across resets
5be20ab050a3 NVIDIA: VR: SAUCE: PCI: Add virtual extended cap save buffer for CXL state
017c49292ab0 NVIDIA: VR: SAUCE: cxl: Move HDM decoder and register map definitions to include/cxl/cxl.h
743681b4b192 NVIDIA: VR: SAUCE: PCI: Add CXL DVSEC control, lock, and range register definitions
eccf13bac1d5 NVIDIA: SAUCE: Revert "NVIDIA: VR: SAUCE: cxl: add support for cxl reset"