|
| 1 | +(native-overview)= |
| 2 | + |
| 3 | +# ExecuTorch Native |
| 4 | + |
| 5 | +::::{note} |
| 6 | +ExecuTorch Native is experimental. The Python API, the `.ptn` format, and |
| 7 | +operator coverage may change. |
| 8 | +:::: |
| 9 | + |
| 10 | +ExecuTorch Native lowers a PyTorch model to a single `.ptn` file and runs it on |
| 11 | +one of several engines. In a `.pte`, delegates lower the graph ahead of time for |
| 12 | +specific backends. A `.ptn` keeps the whole graph without backend-specific |
| 13 | +information, and each engine lowers it when compiling a method. |
| 14 | + |
| 15 | +```text |
| 16 | +.pte: export -> [quantize] -> to_edge -> to_backend -> to_executorch -> .pte (+ .ptd) |
| 17 | +.ptn: export -> [quantize] -> to_native -> .ptn |
| 18 | +``` |
| 19 | + |
| 20 | +<img src="_static/img/native-pipeline.png" alt="ExecuTorch Native pipeline: a PyTorch model goes through torch.export and to_native into a .ptn file, which the Native runtime hands to an engine on the device" style="width:600px;"/> |
| 21 | + |
| 22 | +## Export |
| 23 | + |
| 24 | +```python |
| 25 | +import torch |
| 26 | +from executorch.exir.native import to_native |
| 27 | + |
| 28 | +ep = torch.export.export(model, example_inputs) |
| 29 | +to_native(ep).save("model.ptn") |
| 30 | + |
| 31 | +# Several methods in one file. |
| 32 | +to_native({"forward": ep_forward, "prefill": ep_prefill}).save("model.ptn") |
| 33 | +``` |
| 34 | + |
| 35 | +The partitioner accepts core ATen ops plus `linear`, `matmul`, `addmm`, |
| 36 | +`scaled_dot_product_attention`, `rms_norm`, `native::rope`, and TorchAO affine |
| 37 | +quantize/dequantize ops. TorchAO int4 group-wise weights of `linear` and |
| 38 | +`embedding`, and GGUF k-quant weights from `executorch.extension.llm.export.gguf`, |
| 39 | +are stored packed with their scheme in tensor metadata. Each method must lower |
| 40 | +wholly to one delegate; `to_native` raises otherwise. |
| 41 | + |
| 42 | +A `.ptn` is an uncompressed zip with fixed entry names: |
| 43 | + |
| 44 | +| Entry | Contents | |
| 45 | +| --- | --- | |
| 46 | +| `program.ptg` | Methods as graphs; FlatBuffer defined by [`native_graph.fbs`](https://github.com/pytorch/executorch/blob/main/backends/native/serialization/native_graph.fbs). | |
| 47 | +| `program.safetensors` | Constants. Byte-identical immutable tensors are stored once. | |
| 48 | +| `aliases.json` | Deduplicated key to owner key; present only when needed. | |
| 49 | + |
| 50 | +## Runtime |
| 51 | + |
| 52 | +The C++ runtime reads a `.ptn` into an in-memory graph and hands each method to |
| 53 | +an engine through the interfaces in |
| 54 | +[`Engine.h`](https://github.com/pytorch/executorch/blob/main/backends/native/runtime/engine/Engine.h). |
| 55 | + |
| 56 | +- `EngineHost`: process-wide device resources. |
| 57 | +- `EngineContext`: state for one loaded program, including mutable buffers. |
| 58 | +- `EngineExecutable`: one compiled method (`set_input`, `execute`, `get_output`). |
| 59 | + |
| 60 | +An engine rejects a method whose semantics it cannot fully honor. |
| 61 | + |
| 62 | +| Engine | Notes | |
| 63 | +| --- | --- | |
| 64 | +| Vulkan | Builds on the ET-VK `ComputeGraph` and shaders. | |
| 65 | +| ggml | CPU, CUDA, HIP, and Metal variants. | |
| 66 | + |
| 67 | +The runtime currently builds only with Meta's internal Buck setup; there is no |
| 68 | +open-source build yet. |
| 69 | + |
| 70 | +The source is in |
| 71 | +[`exir/native/`](https://github.com/pytorch/executorch/tree/main/exir/native) and |
| 72 | +[`backends/native/`](https://github.com/pytorch/executorch/tree/main/backends/native). |
0 commit comments