Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file added docs/source/_static/img/native-pipeline.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
13 changes: 13 additions & 0 deletions docs/source/advanced-topics-section.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,18 @@ Key topics:
- PTD file format specification
- Custom file format handling

## ExecuTorch Native

Experimental: export a model to a single `.ptn` file and run it on one of
several engines.

**→ {doc}`native-overview` — ExecuTorch Native**

Key topics:

- Export with `to_native` and the `.ptn` file
- Runtime engines

## Next Steps

After exploring advanced topics:
Expand All @@ -122,4 +134,5 @@ backend-delegate-advanced
runtime-integration-advanced
compiler-ir-advanced
file-formats-advanced
native-overview
```
72 changes: 72 additions & 0 deletions docs/source/native-overview.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
(native-overview)=

# ExecuTorch Native

::::{note}
ExecuTorch Native is experimental. The Python API, the `.ptn` format, and
operator coverage may change.
::::

ExecuTorch Native lowers a PyTorch model to a single `.ptn` file and runs it on
one of several engines. In a `.pte`, delegates lower the graph ahead of time for
specific backends. A `.ptn` keeps the whole graph without backend-specific
information, and each engine lowers it when compiling a method.

```text
.pte: export -> [quantize] -> to_edge -> to_backend -> to_executorch -> .pte (+ .ptd)
.ptn: export -> [quantize] -> to_native -> .ptn
```

<img src="_static/img/native-pipeline.png" alt="ExecuTorch Native pipeline: a PyTorch model goes through torch.export and to_native into a .ptn file, which the Native runtime hands to an engine on the device" style="width:600px;"/>

## Export

```python
import torch
from executorch.exir.native import to_native

ep = torch.export.export(model, example_inputs)
to_native(ep).save("model.ptn")

# Several methods in one file.
to_native({"forward": ep_forward, "prefill": ep_prefill}).save("model.ptn")
```

The partitioner accepts core ATen ops plus `linear`, `matmul`, `addmm`,
`scaled_dot_product_attention`, `rms_norm`, `native::rope`, and TorchAO affine
quantize/dequantize ops. TorchAO int4 group-wise weights of `linear` and
`embedding`, and GGUF k-quant weights from `executorch.extension.llm.export.gguf`,
are stored packed with their scheme in tensor metadata. Each method must lower
wholly to one delegate; `to_native` raises otherwise.

A `.ptn` is an uncompressed zip with fixed entry names:

| Entry | Contents |
| --- | --- |
| `program.ptg` | Methods as graphs; FlatBuffer defined by [`native_graph.fbs`](https://github.com/pytorch/executorch/blob/main/backends/native/serialization/native_graph.fbs). |
| `program.safetensors` | Constants. Byte-identical immutable tensors are stored once. |
| `aliases.json` | Deduplicated key to owner key; present only when needed. |

## Runtime

The C++ runtime reads a `.ptn` into an in-memory graph and hands each method to
an engine through the interfaces in
[`Engine.h`](https://github.com/pytorch/executorch/blob/main/backends/native/runtime/engine/Engine.h).

- `EngineHost`: process-wide device resources.
- `EngineContext`: state for one loaded program, including mutable buffers.
- `EngineExecutable`: one compiled method (`set_input`, `execute`, `get_output`).

An engine rejects a method whose semantics it cannot fully honor.

| Engine | Notes |
| --- | --- |
| Vulkan | Builds on the ET-VK `ComputeGraph` and shaders. |
| ggml | CPU, CUDA, HIP, and Metal variants. |

The runtime currently builds only with Meta's internal Buck setup; there is no
open-source build yet.

The source is in
[`exir/native/`](https://github.com/pytorch/executorch/tree/main/exir/native) and
[`backends/native/`](https://github.com/pytorch/executorch/tree/main/backends/native).