examples/llama: lower to Core ML with to_edge_transform_and_lower - #22052
Open
msluszniak wants to merge 1 commit into
Open
examples/llama: lower to Core ML with to_edge_transform_and_lower#22052msluszniak wants to merge 1 commit into
msluszniak wants to merge 1 commit into
Conversation
The Core ML branch of _to_edge_and_lower_llama() still used the deprecated export_to_edge() + to_backend() split, and CoreMLPartitioner logs a deprecation warning about it on every invocation. The split matters beyond the warning. It decomposes the graph before the partitioner runs, so the ops Core ML has its own implementations for are already broken into primitives by the time it sees them. CoreMLPartitioner.ops_to_not_decompose() asks to keep every op Core ML supports, and only to_edge_transform_and_lower honours that request. Adds _to_edge_and_lower_llama_coreml(), matching the existing xnnpack and mlx helpers, and routes to it when Core ML is the only backend enabled. Enabling Core ML together with Vulkan, MPS or QNN still takes the old combined path, which builds one partitioner list; QNN in particular needs the edge manager for its pass pipeline and model sharding, so it is left alone. Etrecord generation comes along for free: to_edge_transform_and_lower takes generate_etrecord directly, so the helper does not need the deepcopy of the edge manager the old path used. Measured on LFM2.5 350M, fp32, seq 512, ios 18, cpu_and_ne: before: 1 subgraph, 1156 delegated nodes, 13 non-delegated, 1 warning after: 1 subgraph, 893 delegated nodes, 13 non-delegated, 0 warnings Same partitioning, 263 fewer nodes for Core ML to reassemble; expand_copy drops from 133 to 49 and unsqueeze_copy from 375 to 99. Also adds examples/models/lfm2/config/lfm2_coreml_fp32.yaml, alongside the existing lfm2_xnnpack_fp32.yaml and lfm2_mlx_4w.yaml, so the Core ML path has a config to run. Fixes pytorch#19634
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22052
Note: Links to docs will display an error until the docs builds have been completed.
|
This PR needs a
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #19634.
The Core ML branch of
_to_edge_and_lower_llama()still used the deprecatedexport_to_edge()+to_backend()split, andCoreMLPartitionerlogs a deprecation warning about it on every invocation.The split matters beyond the warning. It decomposes the graph before the partitioner runs, so the ops Core ML has its own implementations for are already broken into primitives by the time it sees them.
CoreMLPartitioner.ops_to_not_decompose()asks to keep every op Core ML supports, and onlyto_edge_transform_and_lowerhonours that request.This adds
_to_edge_and_lower_llama_coreml(), matching the existing xnnpack and mlx helpers, and routes to it when Core ML is the only backend enabled.Scope
Enabling Core ML together with Vulkan, MPS or QNN still takes the old combined path, which builds a single partitioner list. QNN in particular needs the edge manager for its pass pipeline and model sharding, so it is left alone; migrating it is a separate piece of work.
Etrecord generation comes along for free:
to_edge_transform_and_lowertakesgenerate_etrecorddirectly, so the helper does not need the deepcopy of the edge manager the old path used.Results
LFM2.5 350M, fp32,
max_seq_length=512,ios: 18,compute_units: cpu_and_ne:Same partitioning, 263 fewer nodes for Core ML to reassemble.
expand_copydrops from 133 to 49 andunsqueeze_copyfrom 375 to 99.A note on the original report
The issue also reported a
SpecViolationError("Mutation node aten_index_put_default_N is neither a buffer nor a user input") from the split path on LFM2 hybrid models, where the short-convconv_state.copy_()decomposes toslice_copy + index_putand only one of the two is recorded as the mutation source. That no longer reproduces onmain: the same export now completes on both the old and the new path. Only the deprecation and the extra decomposition remain, which is what this PR addresses.Test plan
Adds
examples/models/lfm2/config/lfm2_coreml_fp32.yamlalongside the existinglfm2_xnnpack_fp32.yamlandlfm2_mlx_4w.yaml, so the Core ML path has a config to run: