### Bug Description

Repeated declarations of equivalent `@wp.kernel(module="unique")` kernels can retain temporary modules that should be discarded after Warp finds and returns the canonical reused kernel.

When the temporary unique module registers or hashes referenced functions/structs, it can add dependency edges to those referenced modules. If an equivalent unique module already exists, Warp returns the existing canonical kernel object, but the discarded temporary module can remain in referenced modules' `dependents` sets. That keeps the temporary module, kernel, adjoint, AST, and related Python object graph reachable, so repeated factory calls can grow Python memory even though the canonical unique module/kernel count stays stable.

This affects dynamic kernel factory patterns where an application repeatedly recreates semantically identical closure-generated kernels.

Expected behavior:

- Repeated equivalent `module="unique"` declarations reuse one canonical kernel/module.
- Only the canonical reused module remains in referenced modules' `dependents` sets.
- Discarded temporary modules become collectible.

Actual behavior:

- The reused canonical kernel is returned, but discarded temporary modules can remain reachable through dependency edges.
- Referenced modules accumulate extra dependents across repeated factory calls.

Minimal repro:

```python
import warp as wp

@wp.func
def increment(value: int) -> int:
    return value + 1

values = [7, 11]

def make_kernel():
    @wp.kernel(module="unique", enable_backward=False)
    def kernel(out: wp.array(dtype=int)):
        tid = wp.tid()
        for i in range(wp.static(len(values))):
            if wp.static(values[i]) == 7:
                out[tid] = increment(out[tid])

    return kernel

before = set(increment.module.dependents)
kernels = [make_kernel() for _ in range(5)]
after = set(increment.module.dependents)

print(len({id(k) for k in kernels}))
print(len({id(k.module) for k in kernels}))
print(len(after - before))
```

Expected after the fix: all kernels share one object, all modules share one object, and `after - before` contains only the canonical unique module. Before the fix, discarded temporary modules can also appear there.

Related but distinct:

- #995 and #1211: `module="unique"` hashing/static-expression correctness.
- #1231: generic unique kernels with closure-captured functions colliding.

### System Information

Reproduced on the current development branch via a `module="unique"` dependency graph regression test. This is Python-side object retention in module bookkeeping and is not expected to be GPU- or OS-specific.


Warp may retain discarded temporary modules when repeatedly declaring equivalent `@wp.kernel(module="unique")` kernels. This can keep Python object graphs reachable and grow memory over repeated factory calls.

Repeated `module="unique"` kernel declarations retain discarded temporary modules

We need reusable automation for running the release-candidate test matrix. This is meant to support the manual release validation process, not necessarily become a standard CI job.

For prior releases, we tracked and ran the matrix manually, e.g. #2334. The next step is to complete the automation so a release manager can run the same validation consistently for each RC.

Suggested scope:
- Capture the supported test matrix we want for RC validation: OS, Python version, CUDA / driver versions, and any other required environment dimensions.
- Provide a repeatable way to launch or coordinate the matrix for a selected RC branch, tag, or commit.
- Make the output easy to use for release sign-off: clear pass/fail/skipped status per configuration.
- Document where the automation lives and how release managers should run it.

This does not have to live in GitHub Actions or in this repository if another location/tooling is more appropriate. The important part is that the process is repeatable, documented, and usable for each RC.

Acceptance criteria:
- The RC test matrix can be run without manually reconstructing the configuration list each time.
- Results clearly identify the tested RC revision and each tested configuration.
- Failures are attributable to a specific platform / Python / driver configuration.
- Release managers have concise instructions for running the automation and interpreting the results.

Related:
- #2334 tracked a prior manual large-configuration UT run.

Newton wants reusable automation to run a release-candidate validation matrix (OS, Python, CUDA/driver, etc.) for sign-off. Today this is tracked and run manually; they want a consistent launch and reporting workflow for RC branches/tags/commits.

Complete manual RC test matrix automation

### Proposal

Integrate differentiable Reinforcement Learning (DiffRL) support into IsaacLab via the ` isaaclab_diffrl` extension and the [Mineral](https://github.com/etaoxing/mineral.git) algorithm library. This feature provides a dedicated integration layer for the `Direct` workflow and `Newton` (Warp) backend, enabling algorithms like SHAC and APG/BPTT to perform end-to-end gradient backpropagation through physical transitions.

Core capabilities include:
- **Differentiable Direct Environment**: Provides a protocol-compatible environment abstraction for the Mineral algorithm library based on `DirectTaskProtocol`, supporting correct trajectory initialization, as well as checkpoint export and restore functionality.
- **Autograd Bridge**: Inspired by [rewarped](https://github.com/rewarped/rewarped.git), it encapsulates Torch ↔ Warp tensor handoffs and `Warp.Tape()` lifecycle management (automatically managing the clearing of gradient references and physical buffer isolation).
- **Step Semantic Compatibility**: Ensures that rewards and terminal observations are correctly computed in the state right before autoreset occurs (exposed via the `extras["obs_before_reset"]` interface).

### Motivation

While IsaacLab provides the Newton differentiable physics backend powered by NVIDIA Warp, there is currently no native integration solution for DiffRL algorithms that rely on this feature. When attempting to run SHAC or APG on IsaacLab tasks, researchers currently have to "reinvent the wheel" in every project to implement the highly complex differentiable backpropagation paths. This leads to inconsistent implementations of trajectory initialization, reset-detach semantics, and checkpointing, and frequently causes memory issues or unintended gradient truncation due to unmanaged Warp tapes.

This proposal aims to solve these needs once and for all by providing a standard extension (`omni.isaac.lab_diffrl`) that adheres to the IsaacLab 3.0 plugin architecture.

### Alternatives

1. **Manual Implementation**: Users could manually maintain `Warp Tape` and Torch tensor copies within each task. However, this is highly prone to conflicts between the Autograd lifecycle and in-place writes, and it cannot systematically handle global RL semantics like terminal observation reporting.
2. **Vendoring Mineral Source Code**: Copy the algorithm library source code directly into the project. However, we prefer bringing it in as an external dependency to independently track updates and upstream bug fixes.
3. **Alternative Engines**: Using JAX-based simulators (like Brax), but this sacrifices Isaac Sim's high-fidelity rigid-body simulation capabilities, complex contact physics, and its broader ecosystem.

### Build Info

- Isaac Lab Version: [v3.0.0-beta]
- Isaac Sim Version: [6.0.0]

### Additional context

- **MVP Validation**: The first milestone target is to use a `Cartpole`-class task as an MVP to validate the numerical correctness of the end-to-end gradient propagation.
- **Boundary Isolation**: The differentiable path is restricted and isolated within the `AutogradBridge` of the physics step. Observation, reward, and termination condition computations are kept on the Torch side in the MVP version to minimize the risks of cross-boundary Autograd processing.
- **State Decoupling**: The reset logic completely detaches from prior computation graph history, ensuring that gradient calculations for a new rollout are not contaminated by the state of terminated episodes.

### Checklist

- [x] I have checked that there is no similar issue in the repo (**required**)

### Acceptance Criteria

- [ ] `loss.backward()` successfully propagates through the differentiable rollout path with no silent gradient truncation errors.
- [ ] Analytical gradients generated by the Autograd Bridge show numerical agreement with finite-difference estimates on the Cartpole task.
- [ ] The `omni.isaac.lab_diffrl` extension is successfully registered and fully complies with the latest IsaacLab 3.0 plugin file structure specifications.
- [ ] The `DirectTaskProtocol` contract is successfully established, allowing the Mineral algorithm library to invoke the environment protocol without modifying its internal algorithm logic.
- [ ] Checkpoint export and restore work correctly for the Torch and Warp tensor states necessary for forward and backward passes, without involving extraneous environment state buffers.
- [ ] Successful end-to-end algorithm training of APG/BPTT on the reference Cartpole task.

Proposal to integrate DiffRL into IsaacLab via an isaaclab_diffrl extension and the Mineral algorithm library. It targets the Direct workflow and Newton (Warp) backend to enable end-to-end backprop through physics for algorithms like SHAC and APG/BPTT.

[Proposal] Integrate differentiable Reinforcement Learning (DiffRL) support into IsaacLab

PR #2518 added five Kamino examples under `newton/examples/kamino/` but did not include the README entries or screenshots that `AGENTS.md` requires for every example.

**Missing from the release-1.2 branch:**

| Example | README entry | `docs/images/examples/example_<name>.jpg` |
|---|---|---|
| `kamino_basic_fourbar` | missing | missing |
| `kamino_basic_dr_testmech` | missing | missing |
| `kamino_basic_heterogeneous` | missing | missing |
| `kamino_robot_anymal_d` | missing | missing |
| `kamino_robot_dr_legs` | missing | missing |

The examples are discovered and runnable via `python -m newton.examples kamino_basic_fourbar` (etc.) because `get_examples()` walks `newton/examples/` automatically, and they also appear in the GL viewer's example browser. The only gap is README and screenshots.

**`AGENTS.md` policy:**

> ## Examples
> - Register in `README.md` with `python -m newton.examples <name>` command and a 320x320 jpg screenshot.

**Acceptance criteria:**

- [ ] Add a Kamino section (or rows in the existing examples grid) to root `README.md`, one entry per example, each with a `python -m newton.examples <name>` command and a 320×320 jpg link under `docs/images/examples/`.
- [ ] Commit five `docs/images/examples/example_kamino_*.jpg` files (320×320).
- [ ] Cherry-pick the resulting commit onto `release-1.2` before tagging `v1.2.0`.

**References:**

- PR #2518 "Port Kamino-specific examples to outer Newton directory" (introduced the files).
- Tally: `docs/images/examples/` has 65 jpgs vs 70 example files; the 5 missing screenshots are exactly the Kamino set.

Kamino examples were added but are missing required README entries and screenshots on the release branch. This reduces discoverability and fails documented contribution requirements.

Register Kamino examples in README and add screenshots

### Describe the bug
When I try to use multiple GPUs for my IsaacLab project from the IsaacLab 2.3.2 version, I get an error message. 

### Steps to reproduce
I used the command below.
`python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/skrl/train.py --task=Isaac-Lab-Jetbot-Orin-Direct-v0 --enable_cameras --num_envs 2 --video --distributed --headless`
```
2026-05-09T11:08:58Z [48,152ms] [Error] [usdrt.scenegraph.plugin] C++ UsdStage::SelectPrims: GPU 1 requested. GPUs other than cuda:0 are not currently supported
```

After that error below, the process was stuck at the phase below.
`[skrl:INFO] Broadcasting models' parameters`

Besides, I could run the tutorial command below.
`python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 scripts/reinforcement_learning/skrl/train.py --task=Isaac-Cartpole-v0 --headless --distributed`

### System Info
- Commit: 2.3.2
- Isaac Sim Version: 5.1.0
- OS: Ubuntu 22.04
- GPU: Two RTX A6000 under single PC
- CUDA: 12.8
- GPU Driver: 590.48.01

### Additional context

Add any other context about the problem here.

### Checklist

- [x] I have checked that there is no similar issue in the repo (**required**)
- [x] I have checked that the issue is not in running Isaac Sim itself and is related to the repo

### Acceptance Criteria

Add the criteria for which this task is considered **done**. If not known at issue creation time, you can add this once the issue is assigned.

- [ ] Criteria 1
- [ ] Criteria 2


[Bug Report] I can not run Multi-GPU Training from IsaacLab 2.3.2 version

### Description

When loading scene (interactive example) with two robots both using seperate policies, an error occurs in the initialisation, whereby all articulation joints in scene are found, and not per robot.

Two robots have been created with code from examples:
```        
            self.go2 = Go2FlatTerrainPolicy(
            prim_path="/World/Go2",
            name="Go2",
            ##################
            usd_path="https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.1/Isaac/Robots/Unitree/Go2/go2.usd",
            ##################
            position=np.array([0, 0, 0.5]),
        )
        cabinet_prim_path = "/World/cabinet"
        cabinet_usd_path = get_assets_root_path() + "/Isaac/Props/Sektion_Cabinet/sektion_cabinet_instanceable.usd"

        cabinet_name = "cabinet"
        cabinet_position = np.array([0.8, 0.0, 0.4])
        cabinet_orientation = np.array([0.0, 0.0, 0.0, 1.0])

        stage_utils.add_reference_to_stage(cabinet_usd_path, cabinet_prim_path)

        self.cabinet = SingleArticulation(
            prim_path=cabinet_prim_path, name=cabinet_name, position=cabinet_position, orientation=cabinet_orientation
        )

        self.franka = FrankaOpenDrawerPolicy(
            prim_path="/World/franka", name="franka", position=np.array([0, -0, 0]), cabinet=self.cabinet
        )
```

Then performing the initialisation in on_physics_step(self, step_size):

```
            self.go2.initialize()
            self.franka.initialize()
            self.go2.post_reset()
            self.franka.post_reset()
```

The problem occurs inside the self.go2.initialize(), here all joints are found in the scene, not just the for the go2, this is seen in the log:
2026-05-08T22:51:25Z [221,319ms] [Warning] [isaacsim.robot.policy.examples.controllers.config_loader] panda_joint1 default position not found, setting to 0
2026-05-08T22:51:25Z [221,325ms] [Warning] [isaacsim.robot.policy.examples.controllers.config_loader] 
panda_joint2 default position not found, setting to 0
2026-05-08T22:51:25Z [221,327ms] [Warning] [isaacsim.robot.policy.examples.controllers.config_loader] 
panda_joint3 default position not found, setting to 0
2026-05-08T22:51:25Z [221,328ms] [Warning] [isaacsim.robot.policy.examples.controllers.config_loader] 
panda_joint4 default position not found, setting to 0
2026-05-08T22:51:25Z [221,328ms] [Warning] [isaacsim.robot.policy.examples.controllers.config_loader] panda_joint5 default position not found, setting to 0
2026-05-08T22:51:25Z [221,328ms] [Warning] [isaacsim.robot.policy.examples.controllers.config_loader] panda_joint6 default position not found, setting to 0
2026-05-08T22:51:25Z [221,330ms] [Warning] [isaacsim.robot.policy.examples.controllers.config_loader] panda_joint7 default position not found, setting to 0
2026-05-08T22:51:25Z [221,331ms] [Warning] [isaacsim.robot.policy.examples.controllers.config_loader] panda_finger_joint1 default position not found, setting to 0
2026-05-08T22:51:25Z [221,332ms] [Warning] [isaacsim.robot.policy.examples.controllers.config_loader] panda_finger_joint2 default position not found, setting to 0

But then an error occurs, due to the tuples not being of the correct length for the loaded policies.

2026-05-08T22:51:25Z [221,334ms] [Error] [omni.kit.app._impl] [py stderr]: Traceback (most recent call last):
2026-05-08T22:51:25Z [221,335ms] [Error] [omni.kit.app._impl] [py stderr]:   File "c:/isaac-sim/exts/isaacsim.examples.interactive/isaacsim/examples/interactive/quadruped/go2_withArm_example.py", line 187, in on_physics_step
2026-05-08T22:51:25Z [221,337ms] [Error] [omni.kit.app._impl] [py stderr]:     self.go2.initialize()
2026-05-08T22:51:25Z [221,337ms] [Error] [omni.kit.app._impl] [py stderr]:   File "c:/isaac-sim/exts/isaacsim.robot.policy.examples/isaacsim/robot/policy/examples/controllers/policy_controller.py", line 130, in initialize
2026-05-08T22:51:25Z [221,342ms] [Error] [omni.kit.app._impl] [py stderr]:     self.robot._articulation_view.set_gains(stiffness, damping)
2026-05-08T22:51:25Z [221,342ms] [Error] [omni.kit.app._impl] [py stderr]:   File "c:/isaac-sim/exts/isaacsim.core.prims/isaacsim/core/prims/impl/articulation.py", line 3044, in set_gains
2026-05-08T22:51:25Z [221,344ms] [Error] [omni.kit.app._impl] [py stderr]:     stiffnesses = self._backend_utils.assign(
2026-05-08T22:51:25Z [221,344ms] [Error] [omni.kit.app._impl] [py stderr]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2026-05-08T22:51:25Z [221,346ms] [Error] [omni.kit.app._impl] [py stderr]:   File "c:/isaac-sim/exts/isaacsim.core.utils/isaacsim/core/utils/numpy/tensor.py", line 97, in assign
2026-05-08T22:51:25Z [221,347ms] [Error] [omni.kit.app._impl] [py stderr]:     dst[tuple(indices)] = src
2026-05-08T22:51:25Z [221,348ms] [Error] [omni.kit.app._impl] [py stderr]:     ~~~^^^^^^^^^^^^^^^^
2026-05-08T22:51:25Z [221,349ms] [Error] [omni.kit.app._impl] [py stderr]: ValueError: shape mismatch: value array of shape (12,) could not be broadcast to indexing result of shape (1,21)



Problem appears to be inpolicy_controller.py, the function self.robot.initialize(physics_sim_view=physics_sim_view) filing the variable name self.robot.dof_names, with all articulation joints in scene, not just the robot.
This leads to the variables being passed to
get_robot_joint_properties in policyController.py, passing all dof_names to the config_loader.py and single_articulation.py. Where the error occurs

### Isaac Sim version

5.1.0

### Operating System (OS)

Windows WSL Ubuntu 24.04, but also confirmed on normal Ubuntu 24.04.

### GPU Name

RTX 5070

### GPU Driver and CUDA versions

NVIDIA-SMI 591.44                 Driver Version: 591.44         CUDA Version: 13.1

### Logs

_No response_

### Additional information

_No response_

Articulation Initialisation Error when loading scene with two policy controller robots

## Summary
Add diagnostics for the FeatherPGS matrix-free pure Warp solve path so we can distinguish solver convergence, bias-induced correction velocity, and post-step geometry behavior without bringing in large IsaacLab/RL machinery.

## Scope
- Cover only the matrix-free pure Warp path for now.
- Do not instrument dense PGS or generated tiled/CUDA solve kernels in this pass.
- Keep the rollout/data capture lightweight enough to use on single deterministic repros before scaling to IsaacLab tasks.

## Questions to answer
- Given the rows assembled for a frame, how close does the matrix-free solve get to the biased velocity-level constraint problem?
- How much final carried velocity remains in the unbiased velocity-level residual?
- How large is the bias-only term that represents penetration/joint-limit repair pressure?
- Which frames in a rollout are worst offenders by residual, bias magnitude, and post-step penetration?

## Proposed diagnostics
- Per-iteration metrics for selected frames:
  - biased projected residual
  - unbiased projected residual
  - bias-only magnitude
  - impulse delta
  - normal complementarity / friction residual where available
- Per-frame summaries for whole rollouts:
  - contact count
  - pre/post penetration summaries
  - final biased/unbiased projected residuals
  - bias norm/max
  - optional fast-vs-reference deltas later

## Notes
The immediate goal is not a perfect nonlinear contact solver. The goal is to separate: current linearized solver error, bias leakage into carried velocity, and geometry/contact-model issues.

Search — training-infra

Issues