Skip to content

Commit 52bbb30

Browse files
authored
Fix export_llama never passing enable_bf16 to the XNNPACK partitioner (pytorch#21864)
Fixes pytorch#21859 ## Problem pytorch#21784 added `enable_bf16` and threaded it through `_to_edge_and_lower_llama_xnnpack` and `get_xnnpack_partitioner`, defaulting `False` at each layer, but the sole call site never set it: ```python builder = _to_edge_and_lower_llama_xnnpack( ... xnnpack_extended_ops=llm_config.backend.xnnpack.extended_ops, generate_etrecord=llm_config.debug.generate_etrecord, verbose=llm_config.debug.verbose, gen_tag_fn=gen_tag_fn, ) ``` So the parameter stayed `False` on every path, and `--dtype-override bf16 -X` exports silently lost all XNNPACK delegation of their linears: exit 0, valid `.pte`, `Total delegated subgraphs: 0` visible only under `-v`. Full measurements in pytorch#21859. ## Fix Derive the flag from the dtype override at the call site: ```python enable_bf16=llm_config.model.dtype_override.value == "bf16", ``` This is the condition the parameter was added for, and it cannot change behavior for fp32 or fp16 exports. With it, the bf16 demo export delegates the same 21 subgraphs as the fp32 control (per-op table in pytorch#21859). ## Testing New `test_bf16_xnnpack_delegates_linears` in test_export_llama_lib.py exports the no-checkpoint demo model with `dtype_override="bf16"` and XNNPACK on, and asserts every `aten_linear_default` is delegated. The assertion is on linears specifically, not a bare delegation count: measuring both states showed 12 non-gemm subgraphs delegate even without the flag, so a count-only assertion passes either way and cannot guard this wiring. With only the source change reverted and the test kept, the test fails (`linear.delegated=0, non_delegated=8`); with the fix it passes (`8, 0`). lintrunner reports no issues on both changed files. cc @JakeStevens as requested in pytorch#21859
1 parent 52f9431 commit 52bbb30

2 files changed

Lines changed: 24 additions & 0 deletions

File tree

‎examples/models/llama/export_llama_lib.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1696,6 +1696,7 @@ def _export_llama(llm_config: LlmConfig) -> LLMEdgeManager: # noqa: C901
16961696
generate_etrecord=llm_config.debug.generate_etrecord,
16971697
verbose=llm_config.debug.verbose,
16981698
gen_tag_fn=gen_tag_fn,
1699+
enable_bf16=llm_config.model.dtype_override.value == "bf16",
16991700
)
17001701
elif llm_config.backend.openvino.enabled:
17011702
builder = _to_edge_and_lower_llama_openvino(

‎examples/models/llama/tests/test_export_llama_lib.py‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,29 @@ def test_has_expected_ops_and_op_counts(self):
7070
for op, _op_info in delegation_info.delegation_by_operator.items():
7171
self.assertTrue(op not in UNWANTED_OPS)
7272

73+
def test_bf16_xnnpack_delegates_linears(self):
74+
"""
75+
bf16 linears only delegate when enable_bf16 reaches the partitioner,
76+
so this guards the wiring from dtype_override to
77+
get_xnnpack_partitioner. Some non-gemm subgraphs delegate either way,
78+
so the assertion is on linears specifically.
79+
"""
80+
parser = build_args_parser()
81+
args = parser.parse_args([])
82+
args.use_kv_cache = True
83+
args.xnnpack = True
84+
args.xnnpack_extended_ops = True
85+
args.dtype_override = "bf16"
86+
87+
llm_config = LlmConfig.from_args(args)
88+
builder = _export_llama(llm_config)
89+
graph_module = builder.edge_manager.exported_program().graph_module
90+
delegation_info = get_delegation_info(graph_module)
91+
92+
linear = delegation_info.delegation_by_operator["aten_linear_default"]
93+
self.assertGreater(linear.delegated, 0)
94+
self.assertEqual(linear.non_delegated, 0)
95+
7396
@unittest.skipUnless(HAS_ARM_BACKEND, "ARM backend not available")
7497
def test_get_quantizer_and_quant_params_returns_tosa_quantizer(self):
7598
llm_config = LlmConfig()

0 commit comments

Comments
 (0)