Skip to content

examples/llama: lower to Core ML with to_edge_transform_and_lower - #22052

Open
msluszniak wants to merge 1 commit into
pytorch:mainfrom
msluszniak:ms/llama-coreml-transform-and-lower
Open

examples/llama: lower to Core ML with to_edge_transform_and_lower#22052
msluszniak wants to merge 1 commit into
pytorch:mainfrom
msluszniak:ms/llama-coreml-transform-and-lower

Conversation

@msluszniak

Copy link
Copy Markdown
Contributor

Summary

Fixes #19634.

The Core ML branch of _to_edge_and_lower_llama() still used the deprecated export_to_edge() + to_backend() split, and CoreMLPartitioner logs a deprecation warning about it on every invocation.

The split matters beyond the warning. It decomposes the graph before the partitioner runs, so the ops Core ML has its own implementations for are already broken into primitives by the time it sees them. CoreMLPartitioner.ops_to_not_decompose() asks to keep every op Core ML supports, and only to_edge_transform_and_lower honours that request.

This adds _to_edge_and_lower_llama_coreml(), matching the existing xnnpack and mlx helpers, and routes to it when Core ML is the only backend enabled.

Scope

Enabling Core ML together with Vulkan, MPS or QNN still takes the old combined path, which builds a single partitioner list. QNN in particular needs the edge manager for its pass pipeline and model sharding, so it is left alone; migrating it is a separate piece of work.

Etrecord generation comes along for free: to_edge_transform_and_lower takes generate_etrecord directly, so the helper does not need the deepcopy of the edge manager the old path used.

Results

LFM2.5 350M, fp32, max_seq_length=512, ios: 18, compute_units: cpu_and_ne:

subgraphs delegated nodes non-delegated deprecation warnings
before 1 1156 13 1
after 1 893 13 0

Same partitioning, 263 fewer nodes for Core ML to reassemble. expand_copy drops from 133 to 49 and unsqueeze_copy from 375 to 99.

A note on the original report

The issue also reported a SpecViolationError ("Mutation node aten_index_put_default_N is neither a buffer nor a user input") from the split path on LFM2 hybrid models, where the short-conv conv_state.copy_() decomposes to slice_copy + index_put and only one of the two is recorded as the mutation source. That no longer reproduces on main: the same export now completes on both the old and the new path. Only the deprecation and the extra decomposition remain, which is what this PR addresses.

Test plan

Adds examples/models/lfm2/config/lfm2_coreml_fp32.yaml alongside the existing lfm2_xnnpack_fp32.yaml and lfm2_mlx_4w.yaml, so the Core ML path has a config to run:

python -m extension.llm.export.export_llm \
  --config examples/models/lfm2/config/lfm2_coreml_fp32.yaml \
  +base.model_class=lfm2_5_350m \
  +base.params=examples/models/lfm2/config/lfm2_5_350m_config.json \
  +export.max_seq_length=512 \
  +export.max_context_length=512 \
  +export.output_name=lfm2_coreml.pte

The Core ML branch of _to_edge_and_lower_llama() still used the
deprecated export_to_edge() + to_backend() split, and CoreMLPartitioner
logs a deprecation warning about it on every invocation.

The split matters beyond the warning. It decomposes the graph before the
partitioner runs, so the ops Core ML has its own implementations for are
already broken into primitives by the time it sees them.
CoreMLPartitioner.ops_to_not_decompose() asks to keep every op Core ML
supports, and only to_edge_transform_and_lower honours that request.

Adds _to_edge_and_lower_llama_coreml(), matching the existing xnnpack and
mlx helpers, and routes to it when Core ML is the only backend enabled.
Enabling Core ML together with Vulkan, MPS or QNN still takes the old
combined path, which builds one partitioner list; QNN in particular needs
the edge manager for its pass pipeline and model sharding, so it is left
alone.

Etrecord generation comes along for free: to_edge_transform_and_lower
takes generate_etrecord directly, so the helper does not need the
deepcopy of the edge manager the old path used.

Measured on LFM2.5 350M, fp32, seq 512, ios 18, cpu_and_ne:

  before: 1 subgraph, 1156 delegated nodes, 13 non-delegated, 1 warning
  after:  1 subgraph,  893 delegated nodes, 13 non-delegated, 0 warnings

Same partitioning, 263 fewer nodes for Core ML to reassemble; expand_copy
drops from 133 to 49 and unsqueeze_copy from 375 to 99.

Also adds examples/models/lfm2/config/lfm2_coreml_fp32.yaml, alongside
the existing lfm2_xnnpack_fp32.yaml and lfm2_mlx_4w.yaml, so the Core ML
path has a config to run.

Fixes pytorch#19634
@pytorch-bot

pytorch-bot Bot commented Aug 22, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22052

Note: Links to docs will display an error until the docs builds have been completed.

⚠️ 20 Awaiting Approval

As of commit de8e0bc with merge base fbd4bbf (image):

AWAITING APPROVAL - The following workflows need approval before CI can run:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 22, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

examples/llama: CoreML/MPS/QNN export still uses deprecated to_edge() + to_backend() split

2 participants