Skip to content

Commit c27baa8

Browse files
authored
Qualcomm AI Engine Direct - [GenAI Pipeline] PR6: Compilation & inference strategy implementations (pytorch#22284)
## Summary This PR implements the __compilation__ and __inference__ strategy implementations, completing the strategy layer. Each strategy delegates to injectable adapter interfaces for testability. ### What's included #### Strategy implementations: - `ExecuTorchCompilationStrategy`: Compiles model to .pte artifacts via `CompilerAdapter` - Validates model, example_inputs, soc_model, and backend_type are present - Passes `example_inputs` explicitly to the adapter (not via `extra_options`) — mirrors the PR5 fix for the quantization stage - Delegates to adapter with example_inputs, compile specs, artifact dir, soc_model, backend_type - Filters `context.extra_options` to a compilation-relevant allow-list - Returns artifact paths and optional `ETRecord` - `ExecuTorchInferenceStrategy`: Runs on-device inference via `DeviceRunnerAdapter` - Validates artifact_paths and adapter are present (no default adapter — device config is required) - Push → execute → pull results flow - Two-step protocol: uses `output_data` from execute if present, falls back to pulled file paths - Returns inference results, performance metrics, and optional `ETDump` #### `DefaultCompilerAdapter`: - `compile_model` signature finalised — mirrors `to_edge_transform_and_lower_to_qnn` argument-for-argument, so per-graph lowering inputs (`compile_specs`, `dep_table`, `passes_job`, `constant_methods`) are explicit parameters rather than `extra_options` keys - Body deliberately raises `NotImplementedError`: the version this package needs is the multi-graph one (graph-name-keyed dicts, single multi-method `.pte` for weight sharing), so it lands with the strategy-level fan-out that calls it rather than being written single-graph and then replaced. Inject a custom `CompilerAdapter` for now. #### Config addition: - `CompilationInputConfig.example_inputs: Optional[Tuple[Any, ...]]` — sourced from the model via `ModelLoaderAdapter.get_example_inputs` #### Orchestrator wiring: - `genai_pipeline.py` `_run_compilation`: passes `example_inputs=model_prep_output.example_inputs` #### Unit tests: - `test_executorch_compilation_strategy.py` - `test_executorch_inference_strategy.py` - `test_compilation_input_config.py` ### PR Review Checklist - All new classes follow single responsibility (one class per file) - Yes. - All dependencies are injected via constructor with sensible defaults - Yes. - All external calls are behind injectable interfaces - Yes (adapter pattern). - Unit tests cover every public method - Yes (100% coverage on strategy impls). - Docstrings on all public classes and methods - Yes. - Type annotations on all function signatures - Yes. - Logging follows the strategy in the LLD - Yes (info on entry/exit, debug per step). ### Related PRs - PR 1: Core data model, engine routing & exceptions: pytorch#20409 - PR 2: Strategy interfaces & stage wrappers: pytorch#20795 - PR 3: Pipeline orchestrator: pytorch#21149 - PR 4: Adapter interfaces, default implementations & dataset providers: pytorch#21751 - PR 5: Model preparation & quantization strategies: pytorch#21899 - PR 6: Compilation & inference strategy implementations: this pr. - PR 7: Integration & E2E tests: pending. ## Test plan ### Run only tests added in this PR: ``` python -m pytest \ backends/qualcomm/genai_pipeline/tests/strategies/compilation/ \ backends/qualcomm/genai_pipeline/tests/strategies/inference/ \ backends/qualcomm/genai_pipeline/tests/configs/test_compilation_input_config.py -v ``` ### Run only this PR's tests with coverage: ``` python -m pytest \ backends/qualcomm/genai_pipeline/tests/strategies/compilation/ \ backends/qualcomm/genai_pipeline/tests/strategies/inference/ \ backends/qualcomm/genai_pipeline/tests/configs/test_compilation_input_config.py \ --cov=backends/qualcomm/genai_pipeline/strategies/compilation \ --cov=backends/qualcomm/genai_pipeline/strategies/inference \ --cov=backends/qualcomm/genai_pipeline/configs/compilation_input_config \ --cov-report=term-missing ``` Result: ``` Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------------------------------------------------------------------------------------- backends/qualcomm/genai_pipeline/strategies/compilation/compilation_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/compilation/compiler_adapter.py 11 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/compilation/executorch_compilation_strategy.py 45 0 10 0 100% backends/qualcomm/genai_pipeline/strategies/inference/device_runner_adapter.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/inference/executorch_inference_strategy.py 43 0 6 0 100% backends/qualcomm/genai_pipeline/strategies/inference/inference_strategy.py 7 0 0 0 100% ---------------------------------------------------------------------------------------------------------------------------------------- TOTAL 127 0 16 0 100% ``` ### Run all `genai_pipeline` tests: ``` python -m pytest backends/qualcomm/genai_pipeline/tests/ -v ``` ### Run all tests with coverage: ``` python -m pytest backends/qualcomm/genai_pipeline/tests/ \ --cov=backends/qualcomm/genai_pipeline \ --cov-config=backends/qualcomm/.coveragerc \ --cov-report=term-missing ``` Result: ``` Name Stmts Miss Branch BrPart Cover Missing ---------------------------------------------------------------------------------------------------------------------------------------------------- backends/qualcomm/genai_pipeline/configs/compilation_input_config.py 12 0 0 0 100% backends/qualcomm/genai_pipeline/configs/compilation_output_config.py 8 0 0 0 100% backends/qualcomm/genai_pipeline/configs/inference_input_config.py 12 0 0 0 100% backends/qualcomm/genai_pipeline/configs/inference_output_config.py 9 0 0 0 100% backends/qualcomm/genai_pipeline/configs/model_preparation_input_config.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/configs/model_preparation_output_config.py 12 0 0 0 100% backends/qualcomm/genai_pipeline/configs/quantization_input_config.py 13 0 0 0 100% backends/qualcomm/genai_pipeline/configs/quantization_output_config.py 5 0 0 0 100% backends/qualcomm/genai_pipeline/datasets/calibration_data_adapter.py 5 0 0 0 100% backends/qualcomm/genai_pipeline/datasets/default_calibration_data_adapter.py 26 0 6 0 100% backends/qualcomm/genai_pipeline/datasets/default_training_data_adapter.py 14 0 2 0 100% backends/qualcomm/genai_pipeline/datasets/training_data_adapter.py 5 0 0 0 100% backends/qualcomm/genai_pipeline/engine_proxy.py 20 0 4 0 100% backends/qualcomm/genai_pipeline/exceptions.py 20 0 6 0 100% backends/qualcomm/genai_pipeline/genai_pipeline.py 99 7 12 1 93% 190-201 backends/qualcomm/genai_pipeline/pipeline_context.py 52 0 14 0 100% backends/qualcomm/genai_pipeline/pipeline_stage.py 5 0 0 0 100% backends/qualcomm/genai_pipeline/stages/compilation_stage.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/stages/inference_stage.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/stages/model_preparation_stage.py 14 2 0 0 86% 30, 37 backends/qualcomm/genai_pipeline/stages/quantization_stage.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/compilation/compilation_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/compilation/compiler_adapter.py 11 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/compilation/executorch_compilation_strategy.py 45 0 10 0 100% backends/qualcomm/genai_pipeline/strategies/inference/device_runner_adapter.py 14 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/inference/executorch_inference_strategy.py 43 0 6 0 100% backends/qualcomm/genai_pipeline/strategies/inference/inference_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/model_preparation/executorch_model_preparation_strategy.py 68 0 14 0 100% backends/qualcomm/genai_pipeline/strategies/model_preparation/model_loader_adapter.py 9 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/model_preparation/model_preparation_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/quantization/executorch_quantization_strategy.py 61 0 18 0 100% backends/qualcomm/genai_pipeline/strategies/quantization/quantization_strategy.py 7 0 0 0 100% backends/qualcomm/genai_pipeline/strategies/quantization/quantizer_adapter.py 9 0 0 0 100% ---------------------------------------------------------------------------------------------------------------------------------------------------- TOTAL 668 9 92 1 99% ```
1 parent 6e8f02a commit c27baa8

11 files changed

Lines changed: 929 additions & 59 deletions

‎backends/qualcomm/genai_pipeline/configs/compilation_input_config.py‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88

99
from dataclasses import dataclass, field
1010
from pathlib import Path
11-
from typing import List, Optional, TYPE_CHECKING
11+
from typing import Any, List, Optional, Tuple, TYPE_CHECKING
1212

1313
if TYPE_CHECKING:
1414
from executorch.backends.qualcomm.serialization.qc_schema import (
@@ -23,16 +23,29 @@
2323
class CompilationInputConfig:
2424
"""Input configuration for the compilation stage.
2525
26+
``model`` and ``example_inputs`` are ``Optional`` only because the
27+
orchestrator builds this from the previous stages' output, which is empty
28+
when those stages are skipped. **Both are required once the compilation
29+
stage executes**, and strategies should validate their presence.
30+
2631
Attributes:
2732
soc_model: The target SoC (e.g., QcomChipset.SM8750). Required.
2833
backend_type: QNN backend type (HTP, GPU, LPAI, etc.). Required.
2934
model: The nn.Module to compile (quantized or original for FP16 mode).
35+
Required when the stage runs.
36+
example_inputs: Positional example inputs for ``torch.export``. Required
37+
when the stage runs. Sourced from the **model** via
38+
``ModelLoaderAdapter.get_example_inputs``, never from calibration
39+
data: this tuple defines the exported graph's positional signature,
40+
supplies the zero-initialized KV caches a dataset sample does not
41+
carry, and fixes the AR length because HTP has no dynamic shapes.
3042
artifact_dir: Directory to store compiled artifacts.
3143
compile_specs: QNN compiler specifications for backend delegation.
3244
"""
3345

3446
soc_model: "QcomChipset"
3547
backend_type: "QnnExecuTorchBackendType"
3648
model: Optional["nn.Module"] = None
49+
example_inputs: Optional[Tuple[Any, ...]] = None
3750
artifact_dir: Path = field(default_factory=lambda: Path("."))
3851
compile_specs: Optional[List["CompileSpec"]] = None

‎backends/qualcomm/genai_pipeline/configs/inference_output_config.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,12 @@ class InferenceOutputConfig:
1818
"""Output produced by the inference stage.
1919
2020
Attributes:
21-
inference_results: Generated text output(s) from the model.
21+
inference_results: Generated text output(s) from the model, or the paths
22+
of the pulled result files when the adapter writes results to disk
23+
rather than returning them. Stays ``List[str]`` rather than widening
24+
to ``List[Any]``: ``InferenceResult.output_data`` is already decoded
25+
text by contract, so it needs no conversion, and coercing it in the
26+
strategy would risk stringifying raw token ids.
2227
performance_metrics: Performance data (e.g., TTFT, tokens/sec).
2328
eval_results: Evaluation metric results (e.g., SQNR, perplexity).
2429
etdump: Optional ETDump for debugging. ExecuTorch engine only.

‎backends/qualcomm/genai_pipeline/genai_pipeline.py‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,6 +150,34 @@ def _resolve_stage(
150150
stage_name: str,
151151
skip: set,
152152
):
153+
"""Build one stage from the registry, or ``None`` if it is skipped.
154+
155+
Args:
156+
engine_proxy: Routes each stage to its engine.
157+
stage_name: The stage to resolve.
158+
skip: Stage names to skip entirely.
159+
160+
Returns:
161+
The constructed stage, or ``None`` when the stage is skipped.
162+
163+
Raises:
164+
ValueError: If no strategy is registered for the stage's engine.
165+
166+
.. note::
167+
Strategies are constructed with **no arguments**, so each gets its
168+
default adapter. That suffices for model preparation and
169+
quantization, but not for compilation or inference: the default
170+
compiler adapter has no body yet, and the inference strategy
171+
requires an injected device runner. A pipeline built purely from the
172+
registry therefore cannot reach a working adapter for those two
173+
stages.
174+
175+
Injecting adapters today means constructing ``GenAIPipeline`` and
176+
the stages directly rather than going through ``from_proxy``.
177+
Threading adapters through this path lands together with the adapter
178+
bodies, so the hook is shaped by what its callers actually need
179+
rather than guessed at now.
180+
"""
153181
if stage_name in skip:
154182
return None
155183

@@ -241,6 +269,8 @@ def _run_compilation(
241269
soc_model=context.soc_model,
242270
backend_type=self._engine_proxy.backend_type,
243271
model=model,
272+
# Export inputs come from the model, not from calibration_data.
273+
example_inputs=model_prep_output.example_inputs,
244274
artifact_dir=Path(context.artifact_dir),
245275
)
246276
output = self._compilation_stage.invoke(context, input_config)

‎backends/qualcomm/genai_pipeline/strategies/compilation/compiler_adapter.py‎

Lines changed: 26 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88

99
from dataclasses import dataclass, field
1010
from pathlib import Path
11-
from typing import Any, Dict, List, Optional, Protocol, runtime_checkable
11+
from typing import Any, Dict, List, Optional, Protocol, runtime_checkable, Tuple
1212

1313

1414
@dataclass
@@ -28,8 +28,10 @@ class CompilationResult:
2828
class CompilerAdapter(Protocol):
2929
"""Protocol for compilation operations.
3030
31-
Wraps external compilation APIs (ExportSession, to_edge_transform_and_lower_to_qnn)
32-
behind an injectable interface for testability.
31+
Wraps ``to_edge_transform_and_lower_to_qnn`` behind an injectable interface
32+
for testability. The parameter list of ``compile_model`` deliberately
33+
mirrors that function so that no required lowering input has to travel as a
34+
string-keyed option.
3335
3436
.. note::
3537
``compile_model`` lowers a **single graph**, mirroring the underlying
@@ -47,23 +49,43 @@ class CompilerAdapter(Protocol):
4749
def compile_model(
4850
self,
4951
model: Any,
52+
example_inputs: Tuple[Any, ...],
5053
compile_specs: Any,
5154
artifact_dir: Path,
5255
file_name: str,
5356
soc_model: Any,
5457
backend_type: Any,
58+
constant_methods: Optional[Dict[str, Any]] = None,
59+
dep_table: Optional[Dict] = None,
60+
passes_job: Optional[Any] = None,
5561
extra_options: Optional[Dict[str, Any]] = None,
5662
) -> CompilationResult:
5763
"""Compile the model to on-device .pte artifacts.
5864
65+
The parameter list mirrors ``to_edge_transform_and_lower_to_qnn``: every
66+
argument that lowering genuinely needs is explicit, and
67+
``extra_options`` is reserved for optional tuning knobs. Passing a
68+
required input as a string-keyed option is deliberately avoided -- it
69+
hides the contract and fails at runtime rather than at the call site.
70+
5971
Args:
6072
model: The model to compile (nn.Module or quantized model).
73+
example_inputs: Positional example inputs for ``torch.export``,
74+
sourced from the model itself.
6175
compile_specs: QNN compiler specifications for backend delegation.
6276
artifact_dir: Directory to store compiled artifacts.
6377
file_name: Base name for the output .pte file.
6478
soc_model: Target SoC chipset.
6579
backend_type: QNN backend type (HTP, GPU, LPAI).
66-
extra_options: Additional compilation options.
80+
constant_methods: Methods returning constants in eager mode. For a
81+
decoder this carries the quantization attributes written during
82+
quantization, so it is only complete after that stage.
83+
dep_table: Per-graph pass dependency table.
84+
passes_job: Per-graph pass configuration.
85+
extra_options: Optional tuning knobs (``skip_node_id_set``,
86+
``skip_node_op_set``, ``skip_mutable_buffer``,
87+
``convert_linear_to_conv2d``, ``generate_etrecord``,
88+
``executorch_backend_config``).
6789
6890
Returns:
6991
CompilationResult with artifact paths and optional etrecord. May hold

‎backends/qualcomm/genai_pipeline/strategies/compilation/default_compiler_adapter.py‎

Lines changed: 54 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -6,68 +6,95 @@
66

77
from __future__ import annotations
88

9+
import logging
910
from pathlib import Path
10-
from typing import Any, Dict, Optional
11+
from typing import Any, Dict, Optional, Tuple
1112

1213
from executorch.backends.qualcomm.genai_pipeline.strategies.compilation.compiler_adapter import (
1314
CompilationResult,
1415
)
1516

17+
logger = logging.getLogger(__name__)
18+
1619

1720
class DefaultCompilerAdapter:
18-
"""Default adapter delegating to the ExportSession recipe-based pipeline.
21+
"""Default adapter delegating to ``to_edge_transform_and_lower_to_qnn``.
1922
2023
.. note::
21-
The body is deliberately unimplemented **in this PR only**: this PR
22-
establishes the adapter interfaces, and the compilation strategy that
23-
drives this adapter lands in the following PR, so the implementation
24-
ships alongside its caller rather than ahead of it.
24+
The signature below is the final one -- it mirrors
25+
``to_edge_transform_and_lower_to_qnn`` argument-for-argument, so the
26+
per-graph inputs (``compile_specs``, ``dep_table``, ``passes_job``,
27+
``constant_methods``) are explicit parameters rather than
28+
``extra_options`` keys.
29+
30+
The **body** is not implemented in this PR. Lowering is implementation
31+
work rather than interface work, and the version this package needs is
32+
the multi-graph one: ``to_edge_transform_and_lower_to_qnn`` accepts
33+
graph-name-keyed dicts for ``module`` / ``inputs`` / ``compiler_specs``
34+
/ ``dep_table`` / ``passes_job``, and a hybrid decoder groups its graphs
35+
into a single multi-method ``.pte`` for weight sharing. Writing a
36+
single-graph body here and then replacing it would mean implementing the
37+
lowering twice, so it lands with the strategy-level fan-out that calls
38+
it.
2539
26-
The APIs it will delegate to -- ``ExportRecipe.get_recipe`` with
27-
``QNNRecipeType.FP16`` and ``ExportSession`` -- are already available
28-
in-tree; nothing external is blocking it. Until the follow-up lands,
29-
inject a custom ``CompilerAdapter`` implementation.
40+
A recipe-based body (``ExportRecipe.get_recipe(QNNRecipeType.FP16)`` +
41+
``ExportSession``) was considered and rejected: ``QNNRecipeProvider``
42+
accepts only ``soc_model`` and the three ``skip_*`` keys and silently
43+
ignores the rest, so ``dep_table``, ``passes_job``, ``constant_methods``
44+
and ``convert_linear_to_conv2d`` are unreachable through it, and it
45+
hardcodes ``use_fp16=True``.
46+
47+
Until the body lands, inject a custom ``CompilerAdapter``.
3048
"""
3149

3250
def compile_model(
3351
self,
3452
model: Any,
53+
example_inputs: Tuple[Any, ...],
3554
compile_specs: Any,
3655
artifact_dir: Path,
3756
file_name: str,
3857
soc_model: Any,
3958
backend_type: Any,
59+
constant_methods: Optional[Dict[str, Any]] = None,
60+
dep_table: Optional[Dict] = None,
61+
passes_job: Optional[Any] = None,
4062
extra_options: Optional[Dict[str, Any]] = None,
4163
) -> CompilationResult:
42-
"""Compile the model using ExportSession.
43-
44-
Placeholder for the recipe-based compilation flow, which uses
45-
``ExportRecipe.get_recipe(QNNRecipeType.FP16, ...)`` combined with
46-
``ExportSession``. Implemented in the compilation-strategy PR.
64+
"""Compile the model via ``to_edge_transform_and_lower_to_qnn``.
4765
4866
Args:
4967
model: The model to compile (nn.Module or quantized model).
50-
compile_specs: QNN compiler specifications (currently unused when
51-
using recipe-based flow; kept for future custom compile spec support).
68+
example_inputs: Positional example inputs for ``torch.export``,
69+
sourced from the model itself.
70+
compile_specs: QNN compiler specifications for backend delegation.
5271
artifact_dir: Directory to store compiled .pte artifacts.
5372
file_name: Base name for the output .pte file.
5473
soc_model: Target SoC chipset enum value.
5574
backend_type: QNN backend type (HTP, GPU, LPAI).
56-
extra_options: Additional compilation options. Supported keys:
57-
- ``example_inputs``: Sample inputs for torch.export.
58-
- ``generate_etrecord``: Whether to generate ETRecord.
59-
- ``constant_methods``: Dict of constant methods.
75+
constant_methods: Methods returning constants in eager mode. For a
76+
decoder this carries the quantization attributes written into
77+
``meta`` during quantization, so it is only complete once that
78+
stage has run.
79+
dep_table: Per-graph pass dependency table.
80+
passes_job: Per-graph pass configuration.
81+
extra_options: Optional tuning knobs forwarded to lowering:
82+
``skip_node_id_set``, ``skip_node_op_set``,
83+
``skip_mutable_buffer``, ``convert_linear_to_conv2d``,
84+
``generate_etrecord``, and ``executorch_backend_config`` to
85+
override the ``to_executorch`` configuration.
6086
6187
Returns:
6288
CompilationResult with artifact paths and optional etrecord.
6389
6490
Raises:
65-
NotImplementedError: Always raised in this PR. The body lands with
66-
the compilation strategy that drives it; inject a custom
67-
CompilerAdapter until then.
91+
NotImplementedError: Always raised in this PR; the body lands with
92+
the multi-graph lowering that calls it.
6893
"""
6994
raise NotImplementedError(
70-
"DefaultCompilerAdapter is not implemented in this PR: the body "
71-
"lands together with the compilation strategy that drives it. "
72-
"Inject a custom CompilerAdapter implementation until then."
95+
"DefaultCompilerAdapter has no body yet: lowering is implemented "
96+
"together with the strategy-level multi-graph fan-out that calls "
97+
"it, so that to_edge_transform_and_lower_to_qnn is wired up once "
98+
"in its graph-keyed form. Inject a custom CompilerAdapter until "
99+
"then."
73100
)

0 commit comments

Comments
 (0)