Skip to content

Commit c9595a6

Browse files
Add documentation and example of arbitrary QAT flow used with recipe-based export. (pytorch#22408)
### Summary Adds a runnable example and documentation showing how to run an arbitrary Quantization-Aware Training flow in the middle of the declarative export recipe, by splitting the recipe around the `QUANTIZE` stage. The declarative `ExportRecipe` / `ExportSession` path runs the whole pipeline as one `export()` call, which is fine for simple calibration but cannot host complex QAT — real training runs involve iterative loops, checkpointing, job restarts, validation-accuracy evaluation, rollback on regression, and possibly separate machines. The recipe's `pipeline_stages` attribute is used to run the pre-quantize stages, hand the graph off to an out-of-band QAT flow as a `.pt2`, and resume the recipe from `TORCH_EXPORT` onward. 1. New example under `examples/export/qat_pipeline_split/`: four independently-invokable stages driven by `run.sh`. `1_prepare.py` captures up to the quantize boundary, `2_qat.py` prepares with `prepare_qat_pt2e` and does a stand-in train + checkpoint round-trip before `convert_pt2e`, `3_lower.py` lowers to `model.pte` via the recipe, and `4_run.py` runs the `.pte` through the runtime. Each stage asserts its own invariants, so the example doubles as a smoke test. 2. The example shows one method with optional simplifications: the full split sets `pipeline_stages` explicitly on both halves, while the simplified path skips the pre-quantize slice and lets the recipe auto-skip `SOURCE_TRANSFORM` / `QUANTIZE` / `TORCH_EXPORT` for `ExportedProgram` input (default pipeline only). 3. New doc `docs/source/using-complex-qat-flows-with-recipe-based-lowering.md`, registered in the `usage.md` toctree and cross-linked from `quantization-overview.md`. `using-executorch-export.md` was left untouched since it documents only the imperative API. Authored with Vero Code. cc @mergennachin @nil-is-all @iseeyuan @lucylq @helunwencser @tarun292 @kimishpatel @jackzhxng @jerryzh168 @metascroy
1 parent fa03587 commit c9595a6

10 files changed

Lines changed: 1054 additions & 0 deletions

File tree

‎docs/source/quantization-overview.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,3 +73,5 @@ Note that numerics on device can differ those in PyTorch even for unquantized mo
7373
## 3. Lower the model
7474

7575
The final step is to lower the quantized_model to the desired backend, as you would an unquantized one. See [backend-specific pages](backends-overview.md) for lowering information.
76+
77+
For complex Quantization-Aware Training flows — where training loops, checkpointing, and job restarts need to live outside a single export call — see [Using Complex QAT Flows with Recipe-Based Lowering](using-complex-qat-flows-with-recipe-based-lowering.md).

‎docs/source/usage.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ troubleshooting, and frequently asked questions.
99
1010
getting-started
1111
using-executorch-export
12+
using-complex-qat-flows-with-recipe-based-lowering
1213
using-executorch-android
1314
using-executorch-ios
1415
using-executorch-cpp
Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
# Using Complex QAT Flows with Recipe-Based Lowering
2+
3+
How to split the declarative export recipe around the quantization boundary so
4+
that an arbitrary Quantization-Aware Training flow can live outside of — and
5+
between the two halves of — the recipe pipeline.
6+
7+
## Motivation
8+
9+
`ExportRecipe` and `ExportSession` run the full export pipeline — source
10+
transforms, quantize, torch.export, lower, to-executorch — as a single
11+
declarative call. That works well for simple calibration flows.
12+
13+
Complex QAT is different. A real training run involves iterative training
14+
loops, periodic checkpointing, job restarts, accuracy evaluation on validation
15+
datasets, rollback when a checkpoint regresses, and potentially separate
16+
machines for training and for compilation. None of that can fit inside a single
17+
`export()` call.
18+
19+
## How It Works
20+
21+
The recipe's `pipeline_stages` attribute lets you restrict an `ExportSession`
22+
to a specific subset of stages. Use it to split the pipeline around the
23+
`QUANTIZE` stage: run everything up to (but not including) quantization in one
24+
recipe invocation, hand the result off to your QAT flow as a `.pt2` file, and
25+
then continue with the post-quantize stages in a second recipe invocation.
26+
27+
```
28+
ExportRecipe (pre-quantize slice)
29+
|
30+
v
31+
.pt2 <-- cross-process / cross-machine handoff
32+
|
33+
v
34+
Your QAT training loop (any framework, any duration)
35+
|
36+
v
37+
.pt2 <-- cross-process / cross-machine handoff
38+
|
39+
v
40+
ExportRecipe (post-quantize slice)
41+
|
42+
v
43+
model.pte
44+
```
45+
46+
The recipe still owns all the lowering and delegation work; you are only
47+
carving out the `QUANTIZE` stage to replace it with your own flow.
48+
49+
## The Method
50+
51+
### Pre-quantize slice
52+
53+
Run only the pre-quantize stages through `ExportSession`, then export the
54+
resulting module to an ATEN-dialect `ExportedProgram` and save it as a `.pt2`
55+
handoff file:
56+
57+
```python
58+
from executorch.export import export as et_export
59+
from executorch.export.types import StageType
60+
61+
recipe.pipeline_stages = [StageType.SOURCE_TRANSFORM]
62+
sess = et_export(model, example_inputs=[inputs], export_recipe=recipe)
63+
64+
transformed = sess.get_stage_artifacts()[StageType.SOURCE_TRANSFORM].data["forward"]
65+
66+
ep = torch.export.export(transformed, inputs, strict=True)
67+
torch.export.save(ep, "pre_qat.pt2")
68+
```
69+
70+
> **Simplification:** if you do not need any pre-quantize recipe passes (such
71+
> as source transforms provided by the recipe), you can skip this half entirely.
72+
> Capture the eager model directly with `torch.export.export` and save the
73+
> result as the handoff `.pt2`. No recipe is involved in that case.
74+
75+
### Your QAT flow
76+
77+
Load the handoff, prepare it for QAT, and train:
78+
79+
```python
80+
# Load the pre-quantize graph.
81+
captured_gm = torch.export.load("pre_qat.pt2").module()
82+
83+
# Prepare for QAT once.
84+
from torchao.quantization.pt2e.quantize_pt2e import prepare_qat_pt2e
85+
from torchao.quantization.pt2e import move_exported_model_to_train
86+
87+
prepared = prepare_qat_pt2e(captured_gm, quantizer)
88+
move_exported_model_to_train(prepared)
89+
90+
# Train -- this is an open-ended process.
91+
# It can be paused and resumed from checkpoints.
92+
# It can run for hours, days, or weeks.
93+
# It can be distributed across separate machines or jobs.
94+
# Evaluate accuracy on a validation dataset periodically and
95+
# roll back to an earlier checkpoint whenever accuracy regresses.
96+
# Save checkpoints as you go:
97+
# torch.save(prepared.state_dict(), "qat_ckpt.pt")
98+
# Resume from a checkpoint:
99+
# prepared.load_state_dict(torch.load("qat_ckpt.pt"))
100+
101+
# Only once you are satisfied with the result:
102+
from torchao.quantization.pt2e import move_exported_model_to_eval
103+
from torchao.quantization.pt2e.quantize_pt2e import convert_pt2e
104+
105+
move_exported_model_to_eval(prepared)
106+
quantized = convert_pt2e(prepared)
107+
108+
ep = torch.export.export(quantized, inputs, strict=True)
109+
torch.export.save(ep, "post_qat.pt2")
110+
```
111+
112+
### Post-quantize slice
113+
114+
Continue the recipe pipeline from `TORCH_EXPORT` onward:
115+
116+
```python
117+
recipe.pipeline_stages = [
118+
StageType.TORCH_EXPORT,
119+
StageType.TO_EDGE_TRANSFORM_AND_LOWER,
120+
StageType.TO_EXECUTORCH,
121+
]
122+
quantized_gm = torch.export.load("post_qat.pt2").module()
123+
sess = et_export(quantized_gm, example_inputs=[inputs], export_recipe=recipe)
124+
sess.save_to_pte("model") # writes model.pte
125+
```
126+
127+
> **Simplification:** if the recipe uses its default pipeline stages, you do
128+
> not need to set `pipeline_stages` at all on the lowering side. Pass the
129+
> `.pt2` file path directly to `export()` and `ExportSession` will
130+
> automatically skip `SOURCE_TRANSFORM`, `QUANTIZE`, and `TORCH_EXPORT` when it
131+
> detects an `ExportedProgram` input, picking up at
132+
> `TO_EDGE_TRANSFORM_AND_LOWER`. This auto-skip only applies to the default
133+
> pipeline; if the recipe defines a custom `pipeline_stages` list, set the
134+
> post-quantize stages explicitly as shown above.
135+
136+
## Example
137+
138+
A runnable end-to-end example demonstrating both the full split and the
139+
simplified path is available at
140+
[`examples/export/qat_pipeline_split/`](https://github.com/pytorch/executorch/tree/main/examples/export/qat_pipeline_split).
141+
142+
See the [example README](https://github.com/pytorch/executorch/blob/main/examples/export/qat_pipeline_split/README.md)
143+
for full setup instructions, expected artifacts, and notes on adapting the
144+
example to a different backend.
Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
# Copyright 2026 NXP
2+
#
3+
# This source code is licensed under the BSD-style license found in the
4+
# LICENSE file in the root directory of this source tree.
5+
6+
# Stage 1: capture the model up to the quantize boundary.
7+
#
8+
# Two example paths (selected with --example):
9+
#
10+
# minimal (default)
11+
# Export the eager nn.Module to an ATEN-dialect ExportedProgram with a
12+
# single torch.export.export call. No recipe, no ExportSession.
13+
#
14+
# sliced
15+
# Drive only the SOURCE_TRANSFORM stage through ExportSession, then
16+
# export to ATEN and save. This shows how to restrict the pipeline to
17+
# the stages that precede quantization.
18+
#
19+
# Both paths write a .pt2 file consumed by 2_qat.py.
20+
21+
import argparse
22+
import os
23+
24+
import torch
25+
26+
from model import get_example_inputs, get_model
27+
28+
29+
def _build_recipe():
30+
from executorch.backends.xnnpack.recipes.xnnpack_recipe_types import (
31+
XNNPackRecipeType,
32+
)
33+
from executorch.export.recipe import ExportRecipe
34+
35+
return ExportRecipe.get_recipe(XNNPackRecipeType.PT2E_INT8_STATIC_PER_TENSOR)
36+
37+
38+
def run_minimal(workdir: str) -> None:
39+
print("[minimal] Capturing ATEN graph with torch.export.export (no recipe).")
40+
41+
model = get_model()
42+
ex = get_example_inputs()
43+
44+
assert isinstance(
45+
model, torch.nn.Module
46+
), f"Expected nn.Module from get_model(), got {type(model)}"
47+
48+
ep = torch.export.export(model, ex, strict=True)
49+
50+
assert ep is not None, "torch.export.export returned None"
51+
print(f"[minimal] Graph nodes: {len(list(ep.graph.nodes))}")
52+
53+
out = os.path.join(workdir, "stage1_minimal.pt2")
54+
torch.export.save(ep, out)
55+
print(f"[minimal] Saved ExportedProgram -> {out}")
56+
57+
reloaded_gm = torch.export.load(out).module()
58+
y = reloaded_gm(*ex)
59+
assert y.shape == (
60+
1,
61+
10,
62+
), f"[minimal] Unexpected output shape after reload: {y.shape}"
63+
print(f"[minimal] Round-trip output shape: {y.shape} (assertion passed)")
64+
65+
66+
def run_sliced(workdir: str) -> None:
67+
print("[sliced] Running SOURCE_TRANSFORM stage only via ExportSession.")
68+
69+
from executorch.export import export as et_export
70+
from executorch.export.types import StageType
71+
72+
recipe = _build_recipe()
73+
74+
# Restrict the pipeline to the pre-quantize stages only.
75+
recipe.pipeline_stages = [StageType.SOURCE_TRANSFORM]
76+
77+
model = get_model()
78+
ex = get_example_inputs()
79+
80+
assert isinstance(
81+
model, torch.nn.Module
82+
), f"Expected nn.Module from get_model(), got {type(model)}"
83+
84+
sess = et_export(model, example_inputs=[ex], export_recipe=recipe)
85+
86+
artifacts = sess.get_stage_artifacts()
87+
assert (
88+
StageType.SOURCE_TRANSFORM in artifacts
89+
), "SOURCE_TRANSFORM artifact not found - did the stage run?"
90+
91+
transformed = artifacts[StageType.SOURCE_TRANSFORM].data
92+
assert isinstance(
93+
transformed, dict
94+
), f"Expected method-keyed dict from SOURCE_TRANSFORM, got {type(transformed)}"
95+
assert (
96+
"forward" in transformed
97+
), "'forward' method missing from SOURCE_TRANSFORM artifact"
98+
99+
transformed_module = transformed["forward"]
100+
assert isinstance(
101+
transformed_module, torch.nn.Module
102+
), f"Expected nn.Module after SOURCE_TRANSFORM, got {type(transformed_module)}"
103+
print(
104+
f"[sliced] SOURCE_TRANSFORM produced: {type(transformed_module).__name__}"
105+
" (pass-through for this PT2E recipe, as expected)"
106+
)
107+
108+
ep = torch.export.export(transformed_module, ex, strict=True)
109+
assert ep is not None, "torch.export.export returned None"
110+
print(f"[sliced] Graph nodes: {len(list(ep.graph.nodes))}")
111+
112+
out = os.path.join(workdir, "stage1_sliced.pt2")
113+
torch.export.save(ep, out)
114+
print(f"[sliced] Saved ExportedProgram -> {out}")
115+
116+
reloaded_gm = torch.export.load(out).module()
117+
y = reloaded_gm(*ex)
118+
assert y.shape == (
119+
1,
120+
10,
121+
), f"[sliced] Unexpected output shape after reload: {y.shape}"
122+
print(f"[sliced] Round-trip output shape: {y.shape} (assertion passed)")
123+
124+
125+
def main() -> None:
126+
parser = argparse.ArgumentParser(
127+
description="Stage 1: capture the model up to the quantize boundary."
128+
)
129+
parser.add_argument(
130+
"--example",
131+
choices=["minimal", "sliced"],
132+
default="minimal",
133+
)
134+
parser.add_argument("--workdir", required=True)
135+
args = parser.parse_args()
136+
137+
os.makedirs(args.workdir, exist_ok=True)
138+
139+
if args.example == "minimal":
140+
run_minimal(args.workdir)
141+
else:
142+
run_sliced(args.workdir)
143+
144+
print(f"\nStage 1 done. Artifact written to: {args.workdir}")
145+
146+
147+
if __name__ == "__main__":
148+
main()

0 commit comments

Comments
 (0)