Skip to content

Commit 422e07c

Browse files
authored
examples/llama: lower to Core ML with to_edge_transform_and_lower (pytorch#22052)
### Summary Fixes pytorch#19634. The Core ML branch of `_to_edge_and_lower_llama()` still used the deprecated `export_to_edge()` + `to_backend()` split, and `CoreMLPartitioner` logs a deprecation warning about it on every invocation. The split matters beyond the warning. It decomposes the graph before the partitioner runs, so the ops Core ML has its own implementations for are already broken into primitives by the time it sees them. `CoreMLPartitioner.ops_to_not_decompose()` asks to keep every op Core ML supports, and only `to_edge_transform_and_lower` honours that request. This adds `_to_edge_and_lower_llama_coreml()`, matching the existing xnnpack and mlx helpers, and routes to it when Core ML is the only backend enabled. ### Scope Enabling Core ML together with Vulkan, MPS or QNN still takes the old combined path, which builds a single partitioner list. QNN in particular needs the edge manager for its pass pipeline and model sharding, so it is left alone; migrating it is a separate piece of work. Etrecord generation comes along for free: `to_edge_transform_and_lower` takes `generate_etrecord` directly, so the helper does not need the deepcopy of the edge manager the old path used. ### Results LFM2.5 350M, fp32, `max_seq_length=512`, `ios: 18`, `compute_units: cpu_and_ne`: | | subgraphs | delegated nodes | non-delegated | deprecation warnings | | --- | --- | --- | --- | --- | | before | 1 | 1156 | 13 | 1 | | after | 1 | 893 | 13 | 0 | Same partitioning, 263 fewer nodes for Core ML to reassemble. `expand_copy` drops from 133 to 49 and `unsqueeze_copy` from 375 to 99. ### A note on the original report The issue also reported a `SpecViolationError` ("Mutation node aten_index_put_default_N is neither a buffer nor a user input") from the split path on LFM2 hybrid models, where the short-conv `conv_state.copy_()` decomposes to `slice_copy + index_put` and only one of the two is recorded as the mutation source. That no longer reproduces on `main`: the same export now completes on both the old and the new path. Only the deprecation and the extra decomposition remain, which is what this PR addresses. ### Test plan Adds `examples/models/lfm2/config/lfm2_coreml_fp32.yaml` alongside the existing `lfm2_xnnpack_fp32.yaml` and `lfm2_mlx_4w.yaml`, so the Core ML path has a config to run: ```bash python -m extension.llm.export.export_llm \ --config examples/models/lfm2/config/lfm2_coreml_fp32.yaml \ +base.model_class=lfm2_5_350m \ +base.params=examples/models/lfm2/config/lfm2_5_350m_config.json \ +export.max_seq_length=512 \ +export.max_context_length=512 \ +export.output_name=lfm2_coreml.pte ```
1 parent 096a20a commit 422e07c

2 files changed

Lines changed: 87 additions & 0 deletions

File tree

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
base:
2+
metadata: '{"get_bos_id": 1, "get_eos_ids":[7]}'
3+
4+
model:
5+
use_kv_cache: True
6+
enable_dynamic_shape: False
7+
dtype_override: fp32
8+
9+
backend:
10+
coreml:
11+
enabled: True
12+
ios: 18
13+
enable_state: True
14+
preserve_sdpa: True
15+
compute_units: cpu_and_ne

‎examples/models/llama/export_llama_lib.py‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1344,6 +1344,52 @@ def _to_edge_and_lower_llama_mlx(
13441344
return builder.to_executorch(passes=additional_passes)
13451345

13461346

1347+
def _to_edge_and_lower_llama_coreml(
1348+
builder_exported,
1349+
modelname,
1350+
quantizers,
1351+
additional_passes,
1352+
embedding_quantize: Optional[str] = None,
1353+
pt2e_quantize: Optional[str] = None,
1354+
coreml_ios: int = 15,
1355+
coreml_quantize: Optional[str] = None,
1356+
coreml_compute_units: str = "cpu_only",
1357+
generate_etrecord: bool = False,
1358+
verbose: bool = False,
1359+
) -> LLMEdgeManager:
1360+
"""
1361+
Lower Llama model to Core ML using to_edge_transform_and_lower.
1362+
1363+
The deprecated export_to_edge() + to_backend() split decomposes the graph
1364+
before the partitioner runs, so the ops Core ML has its own implementations
1365+
for are already broken into primitives by the time it sees them.
1366+
CoreMLPartitioner.ops_to_not_decompose() asks to keep every op Core ML
1367+
supports, and only to_edge_transform_and_lower honours that request.
1368+
"""
1369+
logging.info("Lowering model using Core ML partitioner")
1370+
1371+
partitioners = [
1372+
get_coreml_partitioner(
1373+
coreml_ios,
1374+
embedding_quantize,
1375+
pt2e_quantize,
1376+
coreml_quantize,
1377+
coreml_compute_units,
1378+
)
1379+
]
1380+
1381+
builder_exported.generate_etrecord = generate_etrecord
1382+
1383+
builder = builder_exported.pt2e_quantize(quantizers).to_edge_transform_and_lower(
1384+
partitioners
1385+
)
1386+
1387+
if verbose:
1388+
print_delegation_info(builder.edge_manager.exported_program().graph_module)
1389+
1390+
return builder.to_executorch(passes=additional_passes)
1391+
1392+
13471393
def _to_edge_and_lower_llama( # noqa: C901
13481394
builder_exported,
13491395
modelname,
@@ -1746,6 +1792,32 @@ def _export_llama(llm_config: LlmConfig) -> LLMEdgeManager: # noqa: C901
17461792
additional_passes,
17471793
verbose=llm_config.debug.verbose,
17481794
)
1795+
elif llm_config.backend.coreml.enabled and not (
1796+
llm_config.backend.vulkan.enabled
1797+
or llm_config.backend.mps.enabled
1798+
or llm_config.backend.qnn.enabled
1799+
):
1800+
builder = _to_edge_and_lower_llama_coreml(
1801+
builder_exported,
1802+
modelname,
1803+
quantizers,
1804+
additional_passes,
1805+
embedding_quantize=llm_config.quantization.embedding_quantize,
1806+
pt2e_quantize=(
1807+
llm_config.quantization.pt2e_quantize.value
1808+
if llm_config.quantization.pt2e_quantize
1809+
else None
1810+
),
1811+
coreml_ios=llm_config.backend.coreml.ios,
1812+
coreml_quantize=(
1813+
llm_config.backend.coreml.quantize.value
1814+
if llm_config.backend.coreml.quantize
1815+
else None
1816+
),
1817+
coreml_compute_units=llm_config.backend.coreml.compute_units.value,
1818+
generate_etrecord=llm_config.debug.generate_etrecord,
1819+
verbose=llm_config.debug.verbose,
1820+
)
17491821
else:
17501822
builder = _to_edge_and_lower_llama(
17511823
builder_exported,

0 commit comments

Comments
 (0)