Update the TensorRT backend docs page (#24136)

This is a draft because the page describes the intended end state of
Torch-TensorRT 2.15. Some links go to sections of the Torch-TensorRT
guide that exist only after pytorch/TensorRT#4927 merges and its docs
site is rebuilt. And the examples rely on 2.15 changes from
pytorch/TensorRT#4909, such as `torch_tensorrt.save` keeping the
compiled graph by default.

## What is wrong today

The TensorRT backend page has drifted away from Torch-TensorRT, the
project that builds and ships the TensorRT delegate.

- It points readers at a nightly build from early October.
- It says CUDA graph replay never runs for engines on the shared
activation scratch. Newer builds run it.
- It repeats most of the Torch-TensorRT guide, so every delegate change
is made twice.
- The backends table lists TensorRT as Linux only.

The CUDA backend page and the getting started page also disagree with
the TensorRT page and with the code: on outputs kept on the GPU, on what
happens when a planned input gets GPU memory, on CUDA versions, GPUs and
Python versions, and on Windows.

## What this change does

The TensorRT page keeps the usual backend sections, says less in each,
and shows more code:

- Features, target requirements, and development requirements with the
one-line install command.
- Export, including sending the operators TensorRT cannot compile to the
CUDA backend, and keeping inputs and outputs on the GPU.
- Runtime integration from Python, and a complete C++ project with the
commands to build and run it.
- A short list of performance options that link to the Torch-TensorRT
guide.

It describes Torch-TensorRT 2.15 and newer, and marks the two features
that need a newer build.

A second commit fixes the CUDA backend page, the getting started page
and one docstring on the points above. To keep data on the GPU, all
pages now leave inputs and outputs unplanned, which copies the least.

## What was tested

On Linux x86-64 with an H100 GPU, an earlier version of the TensorRT
page was run end to end: every Python block in order with nightly
Torch-TensorRT, and the C++ example built and run against the installed
packages.

With nightly packages, the GPU input and output recipe was run from
Python for both backends, and the CUDA page's C++ lines were built and
run. A profiler showed one GPU copy fewer per Python call than with
planned outputs.

The other fixes were checked only by reading them against the sources.
Windows was not tested.

Author: shoumikhin

docs/source/backends-overview.md

@@ -22,7 +22,7 @@ Backends are the bridge between your exported model and the hardware it runs on.
 |--------------------------------------------------------------|---------------|---------------|---------------------------------|
 | [XNNPACK](backends/xnnpack/xnnpack-overview.md)              | All           | CPU           | General-purpose, fallback       |
 | [CUDA](backends/cuda/cuda-overview.md)                       | Linux/Windows | GPU           | NVIDIA GPU acceleration         |
-| [TensorRT](backends/tensorrt/tensorrt-overview.md)           | Linux         | GPU           | NVIDIA GPU acceleration via TensorRT |
+| [TensorRT](backends/tensorrt/tensorrt-overview.md)           | Linux/Windows | GPU           | NVIDIA GPU acceleration via TensorRT |
 | [Core ML](backends/coreml/coreml-overview.md)                | iOS, macOS    | NPU/GPU/CPU   | Apple devices, high performance |
 | [MLX](/backends/mlx/mlx-overview.md)                         | iOS (experimental), macOS | GPU | Apple Silicon GPU (MLX)         |
 | [Vulkan](backends/vulkan/vulkan-overview.md)                 | Android, Linux, Windows | GPU  | Android devices (mature); Desktops (experimental) |

docs/source/backends/cuda/cuda-overview.md

@@ -7,24 +7,29 @@ The CUDA backend is the ExecuTorch solution for running models on NVIDIA GPUs. I
 - **Optimized GPU Execution**: Uses AOTInductor to generate highly optimized CUDA kernels for model operators
 - **Triton Kernel Support**: Leverages Triton for GEMM (General Matrix Multiply), convolution, and SDPA (Scaled Dot-Product Attention) kernels.
 - **Quantization Support**: INT4 weight quantization with tile-packed format for improved performance and reduced memory footprint
-- **Cross-Platform**: Supports both Linux and Windows platforms
+- **Cross-Platform**: Runs on Linux, and on Windows from C++. Programs for Windows are exported on Linux.
 - **Multiple Model Support**: Works with various models including LLMs, vision-language models, and audio models
 
 ## Target Requirements
 
 Below are the requirements for running a CUDA-delegated ExecuTorch model:
 
-- **Hardware**: NVIDIA GPU with CUDA compute capability
-- **CUDA Toolkit**: CUDA 11.x or later (CUDA 12.x recommended)
-- **Operating System**: Linux or Windows
-- **Drivers**: PyTorch-Compatible NVIDIA GPU drivers installed
+- **Hardware**: NVIDIA GPU with compute capability 8.0 (Ampere) or newer. This is the floor the
+  prebuilt packages are built for.
+- **CUDA**: 13 for the prebuilt packages. On Linux they bring the CUDA runtime with them. On
+  Windows it comes from a CUDA 13 toolkit install. A build from source also supports CUDA 12.6.
+- **Operating System**: Linux, or Windows from C++ for programs exported on Linux (see
+  [CUDA on Windows](../../using-executorch-cpp.md#cuda-on-windows))
+- **Drivers**: An NVIDIA driver that supports the CUDA version you use: CUDA 13 for the prebuilt
+  packages.
 
 ## Development Requirements
 
 To develop and export models using the CUDA backend:
 
-- **Python**: Python 3.8+
+- **Python**: Python 3.11 to 3.15
 - **PyTorch**: PyTorch with CUDA support
+- **CUDA Toolkit**: export compiles kernels with `nvcc`, the CUDA compiler, so it needs a CUDA toolkit
 - **ExecuTorch**: Install ExecuTorch with CUDA backend support
 
 ## Using the CUDA Backend
@@ -83,7 +88,7 @@ For a complete working example, see the [CUDA export script](https://github.com/
 
 ## Runtime Integration
 
-To run the model on device, use the standard ExecuTorch runtime APIs. See [Running on Device](getting-started.md#running-on-device) for more information.
+To run the model on device, use the standard ExecuTorch runtime APIs. See [Running on Device](../../getting-started.md#running-on-device) for more information.
 
 When building from source, pass `-DEXECUTORCH_BUILD_CUDA=ON` when configuring the CMake build to compile the CUDA backend.
 
@@ -118,7 +123,7 @@ exec_program = et_program.to_executorch()
 
 ```python
 # CPU tensors. The runtime copies them to the GPU and copies results back.
-outputs = method([torch.randn(4, 64)])
+outputs = method.execute([torch.randn(4, 64)])
 ```
 
 This is the simplest option and the right default. The cost is a copy in each direction on
@@ -140,24 +145,22 @@ exec_program = et_program.to_executorch(
             skip_d2h_for_method_outputs=True,
         ),
         enable_non_cpu_memory_planning=True,
-        # Required alongside the skips. Memory planning reserves a buffer for graph inputs by
-        # default, and the runtime fills a planned input by copying the caller's memory into that
-        # buffer, which reintroduces the copy you just asked to skip. On device memory that copy is
-        # a host memcpy into a device pointer, which is undefined.
-        # Outputs stay planned, so the runtime allocates them on the device and hands back a
-        # pointer to its own buffer. See the note below for when to unplan them instead.
-        memory_planning_pass=MemoryPlanningPass(alloc_graph_input=False),
+        # Keeps the method from reserving its own input and output memory.
+        memory_planning_pass=MemoryPlanningPass(
+            alloc_graph_input=False, alloc_graph_output=False
+        ),
     )
 )
 ```
 
 ```python
 # Now the tensors must already be on the GPU. Passing CPU tensors is a caller error.
-outputs = method([torch.randn(4, 64, device="cuda")])
+outputs = method.execute([torch.randn(4, 64, device="cuda")])
 ```
 
 Outputs also stay on the device, which is what you want when the next stage of your pipeline is
-also on the GPU.
+also on the GPU. From Python, each output comes back in GPU memory that the next call does not
+overwrite, unless you pass `clone_outputs=False`.
 
 ### Things worth knowing
 
@@ -170,24 +173,17 @@ it to `False` while asking for either skip raises a `ValueError`, because copy i
 during device-aware memory planning.
 
 **Inputs must be unplanned**, via `MemoryPlanningPass(alloc_graph_input=False)`. Without it the
-program reserves its own input buffer and the runtime copies the caller's memory into it, so the
-copy comes back at run time. On device memory that copy is a host memcpy into a device pointer,
-which is undefined, so the program crashes rather than returning a wrong answer.
-
-**Outputs have two valid shapes, and which one you want depends on the language.**
-
-Leave outputs planned, which is the default, when the caller cannot supply device memory itself.
-The planned buffer is allocated on the device because `enable_non_cpu_memory_planning` is on, and
-`skip_d2h_for_method_outputs` means nothing copies it back, so the result stays on the device and
-the caller receives a pointer to the runtime's own buffer. That buffer belongs to the method and is
-reused by the next execution, so read or copy the result before running again. This is the shape
-every Python caller wants, and it is what the exporters in this repository use, for example
-`examples/models/gemma4_31b/export.py`.
-
-Set `alloc_graph_output=False` only when the caller supplies the output memory itself, which today
-means a C++ caller handing the runtime a device tensor through `Module::set_output`. The Python
-bindings allocate host memory for an unplanned output, so a device resident method exported this
-way cannot run from Python.
+program reserves its own input buffer on the device, and the runtime refuses to copy the caller's
+tensor into it. The call fails with error `0x10` (`NotSupported`), and the log says
+`Planned-buffer copy needs host memory on both sides`.
+
+**Leave outputs unplanned too**, via `alloc_graph_output=False`. Then each result lands in memory
+you own, with no extra copy. A C++ caller must give each unplanned output a buffer once with
+`Module::set_output`, before the first run: GPU memory when the output copy is skipped, host
+memory otherwise. Pass each output's index to `set_output`; the C++ example below sets output 0. If
+you keep outputs planned instead, they live in the method's own memory and the next run overwrites
+them. Python's `execute()` then copies each output on every call, unless you pass
+`clone_outputs=False`.
 
 **Per-method selection is on the outer config, not on the flags.** Pass a dict of
 `PropagateDeviceConfig` keyed by method name:
@@ -244,6 +240,11 @@ device right after the scalar type. Nothing is copied:
 
 ```cpp
 auto input = from_blob(device_data, {rows, columns}, ScalarType::Float, DeviceType::CUDA);
+auto output = from_blob(device_result, {rows, out_columns}, ScalarType::Float, DeviceType::CUDA);
+// Once, before the first run. Every run then writes its result to device_result.
+if (module.set_output(output) != Error::Ok) {
+  return 1;
+}
 auto result = module.forward(input);
 ```
 
@@ -258,6 +259,7 @@ across, which costs one transfer:
 ```cpp
 auto host_input = make_tensor_ptr({rows, columns}, std::move(data));
 auto input = clone_tensor_ptr_to(host_input, DeviceType::CUDA);
+// Set the output with module.set_output first, as above.
 auto result = module.forward(input);
 ```