Skip to content

Commit a9bf831

Browse files
authored
Qualcomm AI Engine Direct - [GenAI Pipeline Phase 2] PRB1 - Compilation foundation (#22846)
## Summary First PR of __Phase 2__ and the start of __Stream B (compilation)__. Phase 1 added the skeleton, interfaces, stage wrappers, typed configs with stubbed logic. This PR adds the compilation groundwork the rest of Stream B builds on and makes `DefaultCompilerAdapter` actually compile a graph to a `.pte`. Nothing under `examples/` changes. The legacy `llama.py` flow is untouched and stays CI's reference; the two coexist until Phase 3, whose deletion of the legacy path is gated on a parity test added in PR-A2. ### What's included __Two separate name spaces__: `artifact_keys.py`, `graph_names.py` An __artifact key__ names a `.pte` file; a __graph name__ names a method inside one. A hybrid decoder produces one `.pte` (`text_decoder`) holding two methods (`kv_forward`, `prefill_forward`), so these are deliberately distinct key spaces. - Artifact keys are the six the on-device runner already uses for `pte_paths`, so a compiled bundle reaches inference without translation. Plus `DECODE_QDQ_FILENAME`, a *filename* (a `.pt2` the SQNR eval reads, never seen by the runner) and therefore deliberately absent from `ALL_ARTIFACT_KEYS`. - `DECODER_GRAPH_NAMES` is __decode first__, and the order matters: `kv` mode builds no prefill graph, and decode is the authoritative source of the artifact's constant methods. - Both sets duplicate strings from `examples/.../decoder_constants.py` rather than importing them, so this package does not depend on the example scripts. Agreement tests keep them in step; the legacy copies go away with the legacy flow. __`graph_bundle.py`__: `GraphBundle`, the frozen per-graph unit quantization hands to compilation (`module`, `inputs`, `meta`, `quant_io_dtypes`, `modality_inputs`, `executorch_config`) - One decoder is exported several times from the same weights, so the stages after model preparation work on a *set* of graphs. Bundling them means the pipeline threads one `{graph_name: GraphBundle}` map rather than several parallel dicts that can drift apart. - `frozen=True` stops in-place edits across a stage boundary. Immutability stops at the field boundary though, `convert_pt2e` mutates `meta` in place and the docstring says so, so the guarantee is not overread. - `quant_io_dtypes` is the one thing compilation cannot derive: the graph-boundary dtypes follow from the quantization recipe's bit widths. `None` means quantization was skipped. `__post_init__` requires __both keys or neither__, because the tagger indexes both unconditionally so a partial mapping is a `KeyError` at lowering, not a partially-quantized boundary. - `CompilationInputConfig.graphs` is the consumer side, added here; PR-A1 adds the producer side, PR-B2 is the first stage to read it. Defined here so both streams import one definition. __`compilation/compile_spec_builder.py`__: `QnnCompileSpecBuilder`, `resolve_soc_model`, `resolve_backend_type` - Pairs `generate_htp_compiler_spec` / `generate_gpu_compiler_spec` with `generate_qnn_executorch_compiler_spec` behind one `build()`; `llama.py` repeats that branch three times today. - `resolve_soc_model` converts `str` → `QcomChipset`. `PipelineContext` carries the SoC as a string so nothing in the pipeline imports the QNN schema; lowering needs the enum. Per earlier review this belongs in the adapter layer. - `resolve_backend_type` uses a lookup map built from the enum itself, since `QnnExecuTorchBackendType.__str__` already yields the CLI name so no hardcoded table, and a bad name raises `ValueError` listing the valid ones. - `enable_x86_64` is __constructor__ state, not a per-call flag: the emulator supports neither weight sharing nor shared buffers, which is exactly why `llama.py` writes `not args.enable_x86_64` at all three call sites. Hardcoding either on would break the x86 CI run without failing anything locally. - __Trap worth knowing:__ the builder's device default for `shared_buffer` is *on*, while `ControlArgs.shared_buffer` defaults *off* to mirror `llama.py`'s parser. A caller driving the builder from a `ControlArgs` must pass it explicitly. Documented on `build()` and pinned by two tests. __`control_args.py`__: typed bridge from `PipelineContext` to the existing LLM components - Those components read configuration off an `argparse.Namespace` (`args.max_seq_len`, `args.enable_x86_64`, …), so this is a dataclass carrying the same 65 attributes, buildable from a `PipelineContext`. - It __subclasses `argparse.Namespace`__, and that is load-bearing: `QnnConfig.load_config` dispatches on `isinstance(config, argparse.Namespace)`, so a plain dataclass would break the device path. - `build_parser()` derives an `argparse` parser from the same fields, so a CLI entry point stops maintaining a second default table. - Defaults are asserted field-for-field against the real `llama.py` parser by a test, since this is a second configuration surface and drift would be silent. Two fields are explicitly exempt (`train_config`, `lr_config` default to `None` rather than to YAML paths under `examples/`), with a test pinning that the exemption is deliberate. - Documented as __temporary__: fields leave as their consumers move behind pipeline interfaces. __`artifact_paths`: `List[Path]` → `Dict[str, Path]`__ (`CompilationResult`, `CompilationOutputConfig`, `InferenceInputConfig`, `DeviceRunnerAdapter`, `DefaultDeviceRunnerAdapter`) Keyed because the runner addresses artifacts individually and which ones exist varies by model, so position carries no reliable meaning. Absent artifacts are __omitted__, never mapped to `None` so the runner tests key membership to decide whether a model is multimodal. This is the frozen B→A2 interface. __`DefaultCompilerAdapter` implementation__: replaces PR6's `NotImplementedError` - Lowers the graph, converts with the same config the legacy path uses (`MemoryPlanningPass(alloc_graph_input=False, alloc_graph_output=False)` + `BuildQuantIo()`, since with a shared buffer the caller supplies graph I/O addresses from RPC memory), writes the `.pte`, and returns it under the `artifact_key` the caller names keyed by the caller because a 1:1 adapter cannot tell a vision encoder from a text decoder when both arrive as `model`. - `soc_model` and `backend_type` are __validated against `compile_specs`__ rather than trusted. Lowering takes its target only from the specs, so these parameters would otherwise be decorative: a caller could validate ops for one SoC and compile for another and see nothing until the artifact ran on device. The target is read back out of the specs with `flatbuffer_to_option` and a disagreement raises. Specs with no QNN entry skip the check. - Two things stay deliberately outside the adapter, stated in its docstring: __multi-graph grouping__ (fanning out into a multi-method `.pte` is the *strategy*'s job, so the adapter stays a 1:1 wrapper) and __spill-fill sizing__ (a property of the group, computed from the lowered program). Both land in PR-B2. __Two bugs found along the way__ (worth a look, not specific to this feature) - `DefaultDeviceRunnerAdapter.push_artifacts` did `[str(p) for p in artifact_paths]`. Once that is a dict, iteration yields __keys__, so it would have pushed the strings `"text_decoder"`, `"tok_embedding"` … instead of paths. Now `.values()`, matching `EvalBase._get_adb`. This is why the type change could not be a pure annotation edit. - `CompilationInputConfig` imported `CompileSpec` from `executorch.exir.backend.compile_spec`, which does not exist (it is `compile_spec_schema`). Hidden at runtime by `TYPE_CHECKING`; mypy flags it. __Tests__: 6 new files, 8 updated. Beyond the unit tests, `DefaultCompilerAdapter` was run against the real stack (a small module compiled through QNN to a non-empty `.pte` on the x86 path); that takes ~15s so the committed adapter tests mock lowering, while the builder tests keep two real-API cases (specs really are `CompileSpec`s; `use_multi_contexts` + `online_prepare` is rejected rather than masked). ### PR Review Checklist - All new classes follow single responsibility (one class per file) - Yes. - All dependencies are injected via constructor with sensible defaults - Yes. - All external calls are behind injectable interfaces - Yes (adapter pattern; lowering is imported lazily inside the adapter). - Unit tests cover every public method - Yes. - Docstrings on all public classes and methods - Yes. - Type annotations on all function signatures - Yes. - Logging follows the strategy in the LLD - Yes (info on entry/exit, debug per step, warning on degraded paths). - No existing behaviour changed - Yes (no `examples/` files touched; `llama.py` and the wrappers import unchanged). ### Related PRs __Phase 2 flow.__ `PR-A1` and `PR-B1` are independent and can land in either order. `PR-B2` depends on `PR-B1`. `PR-A2` depends on all three, and closes Phase 2 by wiring the runner and adding the parity test that gates the Phase-3 deletion of the legacy flow. ``` PR-A1 ─────────────────────────┐ ├─► PR-A2 PR-B1 (this PR) ──► PR-B2 ──────┘ ``` Phase 1 (merged): - PR 1: Core data model, engine routing & exceptions: #20409 - PR 2: Strategy interfaces & stage wrappers: #20795 - PR 3: Pipeline orchestrator: #21149 - PR 4: Adapter interfaces & default implementations: #21751 - PR 5: Model preparation & quantization strategies: #21899 - PR 6: Compilation & inference strategy implementations: #22284 - PR 7: Integration & E2E tests: cancelled, will be raised after Phase 2 is done. Phase 2: - PR A1: Model registry, `model_lookup`, CLI, `LLMQuantizerAdapter`, multi-graph quantization & encoding reconciliation. Consumes `ControlArgs`, `GraphBundle` and `graph_names` from this PR, but does not depend on it landing first; pending - PR B1: Compilation foundation (__this PR__) - PR B2: Multi-graph lowering, weight sharing, sharding & spill-fill. Depends on this PR; pending - PR A2: Device-runner adapter, pipeline-runner wiring, README & the `llama_stories_260k` E2E parity test. Depends on PR-A1 and PR-B2; pending ## Test plan ### Run only tests added in this PR: ``` python -m pytest \ backends/qualcomm/genai_pipeline/tests/test_artifact_keys.py \ backends/qualcomm/genai_pipeline/tests/test_control_args.py \ backends/qualcomm/genai_pipeline/tests/compilation/ \ backends/qualcomm/genai_pipeline/tests/strategies/compilation/test_default_compiler_adapter.py \ -v ``` Result: ``` 72 passed, 76 subtests passed in 7.55s ``` ### Run all `genai_pipeline` tests: ``` python -m pytest backends/qualcomm/genai_pipeline/tests/ -v ``` Result: ``` 246 passed, 92 subtests passed in 8.08s ``` ### Run all tests with coverage: ``` python -m pytest backends/qualcomm/genai_pipeline/tests/ \ --cov=backends/qualcomm/genai_pipeline \ --cov-config=backends/qualcomm/.coveragerc \ --cov-report=term-missing ``` Result: ``` Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------------------------------------------------------------------------------------------------- backends/qualcomm/genai_pipeline/artifact_keys.py 9 0 0 0 100% backends/qualcomm/genai_pipeline/compilation/compile_spec_builder.py 55 1 16 0 99% 181 backends/qualcomm/genai_pipeline/configs/compilation_input_config.py 13 0 0 0 100% backends/qualcomm/genai_pipeline/configs/compilation_output_config.py 8 0 0 0 100% backends/qualcomm/genai_pipeline/configs/inference_input_config.py 12 0 0 0 100% backends/qualcomm/genai_pipeline/configs/inference_output_config.py 9 0 0 0 100% backends/qualcomm/genai_pipeline/configs/model_preparation_input_config.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/configs/model_preparation_output_config.py 12 0 0 0 100% backends/qualcomm/genai_pipeline/configs/quantization_input_config.py 13 0 0 0 100% backends/qualcomm/genai_pipeline/configs/quantization_output_config.py 5 0 0 0 100% backends/qualcomm/genai_pipeline/control_args.py 154 1 24 1 99% 276 backends/qualcomm/genai_pipeline/datasets/calibration_data_adapter.py 5 0 0 0 100% backends/qualcomm/genai_pipeline/datasets/default_calibration_data_adapter.py 26 0 6 0 100% backends/qualcomm/genai_pipeline/datasets/default_training_data_adapter.py 14 0 2 0 100% backends/qualcomm/genai_pipeline/datasets/training_data_adapter.py 5 0 0 0 100% backends/qualcomm/genai_pipeline/engine_proxy.py 20 0 4 0 100% backends/qualcomm/genai_pipeline/exceptions.py 20 0 6 0 100% backends/qualcomm/genai_pipeline/genai_pipeline.py 99 7 12 1 93% 218-229 backends/qualcomm/genai_pipeline/graph_bundle.py 18 0 4 0 100% backends/qualcomm/genai_pipeline/graph_names.py 8 0 0 0 100% backends/qualcomm/genai_pipeline/pipeline_context.py 52 0 14 0 100% backends/qualcomm/genai_pipeline/pipeline_stage.py 5 0 0 0 100% backends/qualcomm/genai_pipeline/stages/compilation_stage.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/stages/inference_stage.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/stages/model_preparation_stage.py 14 2 0 0 86% 30, 37 backends/qualcomm/genai_pipeline/stages/quantization_stage.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/compilation/compilation_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/compilation/compiler_adapter.py 12 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/compilation/executorch_compilation_strategy.py 45 0 10 0 100% backends/qualcomm/genai_pipeline/strategies/inference/device_runner_adapter.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/inference/executorch_inference_strategy.py 43 0 6 0 100% backends/qualcomm/genai_pipeline/strategies/inference/inference_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/model_preparation/executorch_model_preparation_strategy.py 68 0 14 0 100% backends/qualcomm/genai_pipeline/strategies/model_preparation/model_loader_adapter.py 9 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/model_preparation/model_preparation_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/quantization/executorch_quantization_strategy.py 61 0 18 0 100% backends/qualcomm/genai_pipeline/strategies/quantization/quantization_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/quantization/quantizer_adapter.py 9 0 0 0 100% ---------------------------------------------------------------------------------------------------------------------------------------------------- TOTAL 914 11 136 2 99% ``` ### Confirm the legacy flow is unaffected: ``` python -c " from executorch.examples.qualcomm.oss_scripts.llama.llama import _build_parser, export_llama from executorch.examples.qualcomm.oss_scripts.llama.wrappers import MultiModalManager, HybridAttentionSinkEvictor print('LEGACY IMPORTS OK') " ``` Result: ``` `LEGACY IMPORTS OK` ```
1 parent 12668f8 commit a9bf831

30 files changed

Lines changed: 2725 additions & 100 deletions

‎backends/qualcomm/genai_pipeline/__init__.py‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,21 @@
66

77
__version__ = "1.0.0"
88

9+
from executorch.backends.qualcomm.genai_pipeline.artifact_keys import (
10+
ALL_ARTIFACT_KEYS,
11+
ARTIFACT_ATTENTION_SINK_EVICTOR,
12+
ARTIFACT_AUDIO_ENCODER,
13+
ARTIFACT_TEXT_DECODER,
14+
ARTIFACT_TEXT_ENCODER,
15+
ARTIFACT_TOK_EMBEDDING,
16+
ARTIFACT_VISION_ENCODER,
17+
DECODE_QDQ_FILENAME,
18+
)
19+
from executorch.backends.qualcomm.genai_pipeline.compilation import (
20+
QnnCompileSpecBuilder,
21+
resolve_backend_type,
22+
resolve_soc_model,
23+
)
924
from executorch.backends.qualcomm.genai_pipeline.configs import (
1025
CompilationInputConfig,
1126
CompilationOutputConfig,
@@ -16,6 +31,7 @@
1631
QuantizationInputConfig,
1732
QuantizationOutputConfig,
1833
)
34+
from executorch.backends.qualcomm.genai_pipeline.control_args import ControlArgs
1935
from executorch.backends.qualcomm.genai_pipeline.engine_proxy import EngineProxy
2036
from executorch.backends.qualcomm.genai_pipeline.exceptions import (
2137
ConfigValidationError,
@@ -24,6 +40,16 @@
2440
StageError,
2541
)
2642
from executorch.backends.qualcomm.genai_pipeline.genai_pipeline import GenAIPipeline
43+
from executorch.backends.qualcomm.genai_pipeline.graph_bundle import GraphBundle
44+
from executorch.backends.qualcomm.genai_pipeline.graph_names import (
45+
DECODER_GRAPH_NAMES,
46+
GRAPH_FORWARD,
47+
GRAPH_KV_FORWARD,
48+
GRAPH_PREFILL_FORWARD,
49+
GRAPH_TOK_EMBEDDING_KV_FORWARD,
50+
GRAPH_TOK_EMBEDDING_PREFILL_FORWARD,
51+
TOK_EMBEDDING_GRAPH_NAMES,
52+
)
2753
from executorch.backends.qualcomm.genai_pipeline.pipeline_context import (
2854
PipelineContext,
2955
PipelineContextBuilder,
@@ -32,13 +58,29 @@
3258
from executorch.backends.qualcomm.genai_pipeline.pipeline_types import EngineType
3359

3460
__all__ = [
61+
"ALL_ARTIFACT_KEYS",
62+
"ARTIFACT_ATTENTION_SINK_EVICTOR",
63+
"ARTIFACT_AUDIO_ENCODER",
64+
"ARTIFACT_TEXT_DECODER",
65+
"ARTIFACT_TEXT_ENCODER",
66+
"ARTIFACT_TOK_EMBEDDING",
67+
"ARTIFACT_VISION_ENCODER",
3568
"CompilationInputConfig",
3669
"CompilationOutputConfig",
3770
"ConfigValidationError",
71+
"ControlArgs",
72+
"DECODER_GRAPH_NAMES",
73+
"DECODE_QDQ_FILENAME",
3874
"EngineNotAvailableError",
3975
"EngineProxy",
4076
"EngineType",
4177
"GenAIPipeline",
78+
"GRAPH_FORWARD",
79+
"GRAPH_KV_FORWARD",
80+
"GRAPH_PREFILL_FORWARD",
81+
"GRAPH_TOK_EMBEDDING_KV_FORWARD",
82+
"GRAPH_TOK_EMBEDDING_PREFILL_FORWARD",
83+
"GraphBundle",
4284
"InferenceInputConfig",
4385
"InferenceOutputConfig",
4486
"ModelPreparationInputConfig",
@@ -47,7 +89,11 @@
4789
"PipelineContextBuilder",
4890
"PipelineError",
4991
"PipelineStage",
92+
"QnnCompileSpecBuilder",
5093
"QuantizationInputConfig",
5194
"QuantizationOutputConfig",
95+
"resolve_backend_type",
96+
"resolve_soc_model",
5297
"StageError",
98+
"TOK_EMBEDDING_GRAPH_NAMES",
5399
]
Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Copyright (c) Qualcomm Innovation Center, Inc.
2+
# All rights reserved
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
6+
7+
"""Keys identifying the compiled artifacts a GenAI model produces.
8+
9+
``CompilationOutputConfig.artifact_paths`` is keyed by these names, one entry
10+
per ``.pte`` file. The values match the on-device runner's ``pte_paths`` keys
11+
exactly, so a compiled bundle can be handed to the runner without translation.
12+
13+
.. note::
14+
These are **artifact** keys, not **graph** names. One artifact may hold
15+
several methods: a hybrid text decoder lowers its AR-N prefill and AR-1
16+
decode graphs into a single multi-method ``.pte`` so they can share
17+
weights, and that one file is reached through
18+
``ARTIFACT_TEXT_DECODER``. Graph names (``kv_forward`` /
19+
``prefill_forward``) live in a separate key space used *inside* a lowering
20+
call.
21+
22+
The same string constants exist in the legacy
23+
``examples/qualcomm/oss_scripts/llama/decoder_constants.py``. They are
24+
duplicated here rather than imported so that this package does not depend
25+
on the example scripts; ``tests/test_artifact_keys.py`` asserts the two
26+
stay in agreement, and the legacy copy goes away with the legacy flow.
27+
"""
28+
29+
from __future__ import annotations
30+
31+
# Text decoder: the language model itself. Always present.
32+
ARTIFACT_TEXT_DECODER = "text_decoder"
33+
34+
# Token embedding, lowered separately for multimodal models so that the
35+
# embedding lookup can run while modality features are being inserted.
36+
ARTIFACT_TOK_EMBEDDING = "tok_embedding"
37+
38+
# Attention sink evictor, compiled only when the attention sink feature is on.
39+
ARTIFACT_ATTENTION_SINK_EVICTOR = "attention_sink_evictor"
40+
41+
# Modality encoders. Present only for the corresponding multimodal model.
42+
ARTIFACT_AUDIO_ENCODER = "audio_encoder"
43+
ARTIFACT_TEXT_ENCODER = "text_encoder"
44+
ARTIFACT_VISION_ENCODER = "vision_encoder"
45+
46+
# Every key this package may emit.
47+
#
48+
# Membership in ``artifact_paths`` is meaningful: the runner decides whether a
49+
# model is multimodal by testing for the encoder keys, so an artifact that was
50+
# not compiled must be **absent** rather than mapped to ``None``.
51+
ALL_ARTIFACT_KEYS = frozenset(
52+
{
53+
ARTIFACT_ATTENTION_SINK_EVICTOR,
54+
ARTIFACT_AUDIO_ENCODER,
55+
ARTIFACT_TEXT_DECODER,
56+
ARTIFACT_TEXT_ENCODER,
57+
ARTIFACT_TOK_EMBEDDING,
58+
ARTIFACT_VISION_ENCODER,
59+
}
60+
)
61+
62+
# The decode graph's QDQ exported program, written by quantization before the
63+
# calibration graph is released and read back by the SQNR evaluation.
64+
#
65+
# A **filename**, not an artifact key: it is a ``.pt2`` exported program rather
66+
# than a lowered ``.pte``, the runner never sees it, and only the A-side stages
67+
# touch it. Hence deliberately absent from ``ALL_ARTIFACT_KEYS``. It lives here
68+
# because this module already owns the names the pipeline writes to disk.
69+
DECODE_QDQ_FILENAME = "decode_qdq.pt2"
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Copyright (c) Qualcomm Innovation Center, Inc.
2+
# All rights reserved
3+
#
4+
# This source code is licensed under the BSD-style license found in the
5+
# LICENSE file in the root directory of this source tree.
6+
7+
from executorch.backends.qualcomm.genai_pipeline.compilation.compile_spec_builder import (
8+
QnnCompileSpecBuilder,
9+
resolve_backend_type,
10+
resolve_soc_model,
11+
)
12+
13+
__all__ = [
14+
"QnnCompileSpecBuilder",
15+
"resolve_backend_type",
16+
"resolve_soc_model",
17+
]

0 commit comments

Comments
 (0)