[executorch][cuda] Let the CUDA delegate drive the off-graph KV step

[Gasoonjia]

Stack from ghstack (oldest at bottom):

• [executorch][cuda] Isolate off-graph KV cache by session #22820
• [executorch][cuda] Recapture off-graph KV decode graphs #22819
• [executorch][muse] Wire dynamic KV cache into the runtime #22818
• [executorch][muse] Add dynamic off-graph KV export #22817
• -> [executorch][cuda] Let the CUDA delegate drive the off-graph KV step #22989
• [executorch][cuda] Add dynamic off-graph KV runtime manager #22816
• [executorch][cuda] Lower off-graph KV storage through AOTI #22815

The runner currently calls prepare_step/commit_step on CudaKVCache, which puts
a backend face in the runner's hands. The delegate is the natural owner: it
already runs on the host immediately before and after the compiled program, and
that is the only point where storage can grow and be rebound before the kernels
read it.

The one thing the delegate cannot work out for itself is how many tokens a step
writes. Lowering compiles the cache op away, so at runtime there is only
compiled code over pre-bound memory and nothing calls back out. The export side
knows the width from its own signature, so it declares where to find it as an
"input_index:dim" compile spec, and the delegate reads that input's extent --
shape metadata only, never contents, which are on device.

Committing from here also pairs the two halves on every exit: a run that fails
never advances the cache past tokens the kernels did not write.

Differential Revision: D121131873

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22989

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 2 Pending, 1 Unrelated Failure, 2 Unclassified Failures

As of commit 0340d66 with merge base 57078b4 ():

NEW FAILURE - The following job has failed:

• pull / unittest-nxp-neutron / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Test ROCm AOTI / test-voxtral-realtime-rocm-gfx950-rocm7.2 / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t 9f2272818989a0846a3ddc5818c1882dea130d92e4ae4ed8b92f0bdc3ea6c2fd /exec failed with exit code 2
• Test ROCm AOTI / unittest-rocm-gfx950-rocm7.2 / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t 3bdb71c8736ae9e375d84d18383effa376cdaa2c4bc020cd9b9bb3c46f8eddbb /exec failed with exit code 2

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• pull / unittest-buck / macos / macos-job (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.