Repository navigation
Commit a9bf831
authored
Qualcomm AI Engine Direct - [GenAI Pipeline Phase 2] PRB1 - Compilation foundation (#22846)
## Summary
First PR of __Phase 2__ and the start of __Stream B (compilation)__.
Phase 1 added the skeleton, interfaces, stage wrappers, typed configs
with stubbed logic. This PR adds the compilation groundwork the rest of
Stream B builds on and makes `DefaultCompilerAdapter` actually compile a
graph to a `.pte`.
Nothing under `examples/` changes. The legacy `llama.py` flow is
untouched and stays CI's reference; the two coexist until Phase 3, whose
deletion of the legacy path is gated on a parity test added in PR-A2.
### What's included
__Two separate name spaces__: `artifact_keys.py`, `graph_names.py`
An __artifact key__ names a `.pte` file; a __graph name__ names a method
inside one. A hybrid decoder produces one `.pte` (`text_decoder`)
holding two methods (`kv_forward`, `prefill_forward`), so these are
deliberately distinct key spaces.
- Artifact keys are the six the on-device runner already uses for
`pte_paths`, so a compiled bundle reaches inference without translation.
Plus `DECODE_QDQ_FILENAME`, a *filename* (a `.pt2` the SQNR eval reads,
never seen by the runner) and therefore deliberately absent from
`ALL_ARTIFACT_KEYS`.
- `DECODER_GRAPH_NAMES` is __decode first__, and the order matters: `kv`
mode builds no prefill graph, and decode is the authoritative source of
the artifact's constant methods.
- Both sets duplicate strings from `examples/.../decoder_constants.py`
rather than importing them, so this package does not depend on the
example scripts. Agreement tests keep them in step; the legacy copies go
away with the legacy flow.
__`graph_bundle.py`__: `GraphBundle`, the frozen per-graph unit
quantization hands to compilation (`module`, `inputs`, `meta`,
`quant_io_dtypes`, `modality_inputs`, `executorch_config`)
- One decoder is exported several times from the same weights, so the
stages after model preparation work on a *set* of graphs. Bundling them
means the pipeline threads one `{graph_name: GraphBundle}` map rather
than several parallel dicts that can drift apart.
- `frozen=True` stops in-place edits across a stage boundary.
Immutability stops at the field boundary though, `convert_pt2e` mutates
`meta` in place and the docstring says so, so the guarantee is not
overread.
- `quant_io_dtypes` is the one thing compilation cannot derive: the
graph-boundary dtypes follow from the quantization recipe's bit widths.
`None` means quantization was skipped. `__post_init__` requires __both
keys or neither__, because the tagger indexes both unconditionally so a
partial mapping is a `KeyError` at lowering, not a partially-quantized
boundary.
- `CompilationInputConfig.graphs` is the consumer side, added here;
PR-A1 adds the producer side, PR-B2 is the first stage to read it.
Defined here so both streams import one definition.
__`compilation/compile_spec_builder.py`__: `QnnCompileSpecBuilder`,
`resolve_soc_model`, `resolve_backend_type`
- Pairs `generate_htp_compiler_spec` / `generate_gpu_compiler_spec` with
`generate_qnn_executorch_compiler_spec` behind one `build()`; `llama.py`
repeats that branch three times today.
- `resolve_soc_model` converts `str` → `QcomChipset`. `PipelineContext`
carries the SoC as a string so nothing in the pipeline imports the QNN
schema; lowering needs the enum. Per earlier review this belongs in the
adapter layer.
- `resolve_backend_type` uses a lookup map built from the enum itself,
since `QnnExecuTorchBackendType.__str__` already yields the CLI name so
no hardcoded table, and a bad name raises `ValueError` listing the valid
ones.
- `enable_x86_64` is __constructor__ state, not a per-call flag: the
emulator supports neither weight sharing nor shared buffers, which is
exactly why `llama.py` writes `not args.enable_x86_64` at all three call
sites. Hardcoding either on would break the x86 CI run without failing
anything locally.
- __Trap worth knowing:__ the builder's device default for
`shared_buffer` is *on*, while `ControlArgs.shared_buffer` defaults
*off* to mirror `llama.py`'s parser. A caller driving the builder from a
`ControlArgs` must pass it explicitly. Documented on `build()` and
pinned by two tests.
__`control_args.py`__: typed bridge from `PipelineContext` to the
existing LLM components
- Those components read configuration off an `argparse.Namespace`
(`args.max_seq_len`, `args.enable_x86_64`, …), so this is a dataclass
carrying the same 65 attributes, buildable from a `PipelineContext`.
- It __subclasses `argparse.Namespace`__, and that is load-bearing:
`QnnConfig.load_config` dispatches on `isinstance(config,
argparse.Namespace)`, so a plain dataclass would break the device path.
- `build_parser()` derives an `argparse` parser from the same fields, so
a CLI entry point stops maintaining a second default table.
- Defaults are asserted field-for-field against the real `llama.py`
parser by a test, since this is a second configuration surface and drift
would be silent. Two fields are explicitly exempt (`train_config`,
`lr_config` default to `None` rather than to YAML paths under
`examples/`), with a test pinning that the exemption is deliberate.
- Documented as __temporary__: fields leave as their consumers move
behind pipeline interfaces.
__`artifact_paths`: `List[Path]` → `Dict[str, Path]`__
(`CompilationResult`, `CompilationOutputConfig`, `InferenceInputConfig`,
`DeviceRunnerAdapter`, `DefaultDeviceRunnerAdapter`)
Keyed because the runner addresses artifacts individually and which ones
exist varies by model, so position carries no reliable meaning. Absent
artifacts are __omitted__, never mapped to `None` so the runner tests
key membership to decide whether a model is multimodal. This is the
frozen B→A2 interface.
__`DefaultCompilerAdapter` implementation__: replaces PR6's
`NotImplementedError`
- Lowers the graph, converts with the same config the legacy path uses
(`MemoryPlanningPass(alloc_graph_input=False, alloc_graph_output=False)`
+ `BuildQuantIo()`, since with a shared buffer the caller supplies graph
I/O addresses from RPC memory), writes the `.pte`, and returns it under
the `artifact_key` the caller names keyed by the caller because a 1:1
adapter cannot tell a vision encoder from a text decoder when both
arrive as `model`.
- `soc_model` and `backend_type` are __validated against
`compile_specs`__ rather than trusted. Lowering takes its target only
from the specs, so these parameters would otherwise be decorative: a
caller could validate ops for one SoC and compile for another and see
nothing until the artifact ran on device. The target is read back out of
the specs with `flatbuffer_to_option` and a disagreement raises. Specs
with no QNN entry skip the check.
- Two things stay deliberately outside the adapter, stated in its
docstring: __multi-graph grouping__ (fanning out into a multi-method
`.pte` is the *strategy*'s job, so the adapter stays a 1:1 wrapper) and
__spill-fill sizing__ (a property of the group, computed from the
lowered program). Both land in PR-B2.
__Two bugs found along the way__ (worth a look, not specific to this
feature)
- `DefaultDeviceRunnerAdapter.push_artifacts` did `[str(p) for p in
artifact_paths]`. Once that is a dict, iteration yields __keys__, so it
would have pushed the strings `"text_decoder"`, `"tok_embedding"` …
instead of paths. Now `.values()`, matching `EvalBase._get_adb`. This is
why the type change could not be a pure annotation edit.
- `CompilationInputConfig` imported `CompileSpec` from
`executorch.exir.backend.compile_spec`, which does not exist (it is
`compile_spec_schema`). Hidden at runtime by `TYPE_CHECKING`; mypy flags
it.
__Tests__: 6 new files, 8 updated. Beyond the unit tests,
`DefaultCompilerAdapter` was run against the real stack (a small module
compiled through QNN to a non-empty `.pte` on the x86 path); that takes
~15s so the committed adapter tests mock lowering, while the builder
tests keep two real-API cases (specs really are `CompileSpec`s;
`use_multi_contexts` + `online_prepare` is rejected rather than masked).
### PR Review Checklist
- All new classes follow single responsibility (one class per file) -
Yes.
- All dependencies are injected via constructor with sensible defaults -
Yes.
- All external calls are behind injectable interfaces - Yes (adapter
pattern; lowering is imported lazily inside the adapter).
- Unit tests cover every public method - Yes.
- Docstrings on all public classes and methods - Yes.
- Type annotations on all function signatures - Yes.
- Logging follows the strategy in the LLD - Yes (info on entry/exit,
debug per step, warning on degraded paths).
- No existing behaviour changed - Yes (no `examples/` files touched;
`llama.py` and the wrappers import unchanged).
### Related PRs
__Phase 2 flow.__ `PR-A1` and `PR-B1` are independent and can land in
either order. `PR-B2` depends on `PR-B1`. `PR-A2` depends on all three,
and closes Phase 2 by wiring the runner and adding the parity test that
gates the Phase-3 deletion of the legacy flow.
```
PR-A1 ─────────────────────────┐
├─► PR-A2
PR-B1 (this PR) ──► PR-B2 ──────┘
```
Phase 1 (merged):
- PR 1: Core data model, engine routing & exceptions: #20409
- PR 2: Strategy interfaces & stage wrappers: #20795
- PR 3: Pipeline orchestrator: #21149
- PR 4: Adapter interfaces & default implementations: #21751
- PR 5: Model preparation & quantization strategies: #21899
- PR 6: Compilation & inference strategy implementations: #22284
- PR 7: Integration & E2E tests: cancelled, will be raised after Phase 2
is done.
Phase 2:
- PR A1: Model registry, `model_lookup`, CLI, `LLMQuantizerAdapter`,
multi-graph quantization & encoding reconciliation. Consumes
`ControlArgs`, `GraphBundle` and `graph_names` from this PR, but does
not depend on it landing first; pending
- PR B1: Compilation foundation (__this PR__)
- PR B2: Multi-graph lowering, weight sharing, sharding & spill-fill.
Depends on this PR; pending
- PR A2: Device-runner adapter, pipeline-runner wiring, README & the
`llama_stories_260k` E2E parity test. Depends on PR-A1 and PR-B2;
pending
## Test plan
### Run only tests added in this PR:
```
python -m pytest \
backends/qualcomm/genai_pipeline/tests/test_artifact_keys.py \
backends/qualcomm/genai_pipeline/tests/test_control_args.py \
backends/qualcomm/genai_pipeline/tests/compilation/ \
backends/qualcomm/genai_pipeline/tests/strategies/compilation/test_default_compiler_adapter.py \
-v
```
Result:
```
72 passed, 76 subtests passed in 7.55s
```
### Run all `genai_pipeline` tests:
```
python -m pytest backends/qualcomm/genai_pipeline/tests/ -v
```
Result:
```
246 passed, 92 subtests passed in 8.08s
```
### Run all tests with coverage:
```
python -m pytest backends/qualcomm/genai_pipeline/tests/ \
--cov=backends/qualcomm/genai_pipeline \
--cov-config=backends/qualcomm/.coveragerc \
--cov-report=term-missing
```
Result:
```
Name Stmts Miss Branch BrPart Cover Missing
----------------------------------------------------------------------------------------------------------------------------------------------------
backends/qualcomm/genai_pipeline/artifact_keys.py 9 0 0 0 100%
backends/qualcomm/genai_pipeline/compilation/compile_spec_builder.py 55 1 16 0 99% 181
backends/qualcomm/genai_pipeline/configs/compilation_input_config.py 13 0 0 0 100%
backends/qualcomm/genai_pipeline/configs/compilation_output_config.py 8 0 0 0 100%
backends/qualcomm/genai_pipeline/configs/inference_input_config.py 12 0 0 0 100%
backends/qualcomm/genai_pipeline/configs/inference_output_config.py 9 0 0 0 100%
backends/qualcomm/genai_pipeline/configs/model_preparation_input_config.py 7 0 0 0 100%
backends/qualcomm/genai_pipeline/configs/model_preparation_output_config.py 12 0 0 0 100%
backends/qualcomm/genai_pipeline/configs/quantization_input_config.py 13 0 0 0 100%
backends/qualcomm/genai_pipeline/configs/quantization_output_config.py 5 0 0 0 100%
backends/qualcomm/genai_pipeline/control_args.py 154 1 24 1 99% 276
backends/qualcomm/genai_pipeline/datasets/calibration_data_adapter.py 5 0 0 0 100%
backends/qualcomm/genai_pipeline/datasets/default_calibration_data_adapter.py 26 0 6 0 100%
backends/qualcomm/genai_pipeline/datasets/default_training_data_adapter.py 14 0 2 0 100%
backends/qualcomm/genai_pipeline/datasets/training_data_adapter.py 5 0 0 0 100%
backends/qualcomm/genai_pipeline/engine_proxy.py 20 0 4 0 100%
backends/qualcomm/genai_pipeline/exceptions.py 20 0 6 0 100%
backends/qualcomm/genai_pipeline/genai_pipeline.py 99 7 12 1 93% 218-229
backends/qualcomm/genai_pipeline/graph_bundle.py 18 0 4 0 100%
backends/qualcomm/genai_pipeline/graph_names.py 8 0 0 0 100%
backends/qualcomm/genai_pipeline/pipeline_context.py 52 0 14 0 100%
backends/qualcomm/genai_pipeline/pipeline_stage.py 5 0 0 0 100%
backends/qualcomm/genai_pipeline/stages/compilation_stage.py 14 0 0 0 100%
backends/qualcomm/genai_pipeline/stages/inference_stage.py 14 0 0 0 100%
backends/qualcomm/genai_pipeline/stages/model_preparation_stage.py 14 2 0 0 86% 30, 37
backends/qualcomm/genai_pipeline/stages/quantization_stage.py 14 0 0 0 100%
backends/qualcomm/genai_pipeline/strategies/compilation/compilation_strategy.py 7 0 0 0 100%
backends/qualcomm/genai_pipeline/strategies/compilation/compiler_adapter.py 12 0 0 0 100%
backends/qualcomm/genai_pipeline/strategies/compilation/executorch_compilation_strategy.py 45 0 10 0 100%
backends/qualcomm/genai_pipeline/strategies/inference/device_runner_adapter.py 14 0 0 0 100%
backends/qualcomm/genai_pipeline/strategies/inference/executorch_inference_strategy.py 43 0 6 0 100%
backends/qualcomm/genai_pipeline/strategies/inference/inference_strategy.py 7 0 0 0 100%
backends/qualcomm/genai_pipeline/strategies/model_preparation/executorch_model_preparation_strategy.py 68 0 14 0 100%
backends/qualcomm/genai_pipeline/strategies/model_preparation/model_loader_adapter.py 9 0 0 0 100%
backends/qualcomm/genai_pipeline/strategies/model_preparation/model_preparation_strategy.py 7 0 0 0 100%
backends/qualcomm/genai_pipeline/strategies/quantization/executorch_quantization_strategy.py 61 0 18 0 100%
backends/qualcomm/genai_pipeline/strategies/quantization/quantization_strategy.py 7 0 0 0 100%
backends/qualcomm/genai_pipeline/strategies/quantization/quantizer_adapter.py 9 0 0 0 100%
----------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL 914 11 136 2 99%
```
### Confirm the legacy flow is unaffected:
```
python -c "
from executorch.examples.qualcomm.oss_scripts.llama.llama import _build_parser, export_llama
from executorch.examples.qualcomm.oss_scripts.llama.wrappers import MultiModalManager, HybridAttentionSinkEvictor
print('LEGACY IMPORTS OK')
"
```
Result:
```
`LEGACY IMPORTS OK`
```1 parent 12668f8 commit a9bf831
30 files changed
Lines changed: 2725 additions & 100 deletions
File tree
- backends/qualcomm/genai_pipeline
- compilation
- configs
- strategies
- compilation
- inference
- tests
- compilation
- configs
- stages
- strategies
- compilation
- inference
- quantization
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
9 | 24 | | |
10 | 25 | | |
11 | 26 | | |
| |||
16 | 31 | | |
17 | 32 | | |
18 | 33 | | |
| 34 | + | |
19 | 35 | | |
20 | 36 | | |
21 | 37 | | |
| |||
24 | 40 | | |
25 | 41 | | |
26 | 42 | | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
27 | 53 | | |
28 | 54 | | |
29 | 55 | | |
| |||
32 | 58 | | |
33 | 59 | | |
34 | 60 | | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
35 | 68 | | |
36 | 69 | | |
37 | 70 | | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
38 | 74 | | |
39 | 75 | | |
40 | 76 | | |
41 | 77 | | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
42 | 84 | | |
43 | 85 | | |
44 | 86 | | |
| |||
47 | 89 | | |
48 | 90 | | |
49 | 91 | | |
| 92 | + | |
50 | 93 | | |
51 | 94 | | |
| 95 | + | |
| 96 | + | |
52 | 97 | | |
| 98 | + | |
53 | 99 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
0 commit comments