Skip to content

Use Phi-4-mini's own BOS and EOS ids in its example config - #22042

Open
john-rocky wants to merge 1 commit into
pytorch:mainfrom
john-rocky:fix-phi4-token-ids
Open

Use Phi-4-mini's own BOS and EOS ids in its example config#22042
john-rocky wants to merge 1 commit into
pytorch:mainfrom
john-rocky:fix-phi4-token-ids

Conversation

@john-rocky

Copy link
Copy Markdown
Contributor

examples/models/phi_4_mini/config/phi_4_mini_xnnpack.yaml carries

metadata: '{"get_bos_id":151643, "get_eos_ids":[151643]}'

151643 is Qwen's <|endoftext|>. Phi-4-mini has a different vocabulary, and in it that id is a subword:

>>> from transformers import AutoTokenizer
>>> AutoTokenizer.from_pretrained("microsoft/Phi-4-mini-instruct").convert_ids_to_tokens(151643)
"Ġprzyp"
>>> AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B").convert_ids_to_tokens(151643)
"<|endoftext|>"

So the runner is told to begin on a fragment of a Polish word and to stop on one, and it never sees the token the model actually ends turns with.

microsoft/Phi-4-mini-instruct/generation_config.json gives:

{"bos_token_id": 199999, "eos_token_id": [200020, 199999]}

which are <|endoftext|> and <|end|>. This patch uses those.

Exported with the corrected ids (executorch 1.4.0, XNNPACK, 8da4w + 8-bit embedding, 2597.9 MB), the model answers and stops where it should:

prompt: <|user|>What is 17 times 4?<|end|><|assistant|>
output: 17 times 4 is equal to 68.
        decode: 10 tok in 0.20s (49.73 tok/s)

The generation ends after 10 tokens because <|end|> is reached, which is the part that does not happen with 151643.

The config carries 151643 for both, which is Qwen's <|endoftext|>. In Phi-4-mini's
vocabulary that id is the subword 'Ġprzyp', so the runner is told to start on a
fragment of a Polish word and to stop on one.

Phi-4-mini's generation_config.json gives bos_token_id 199999 (<|endoftext|>) and
eos_token_id [200020, 199999] (<|end|>, <|endoftext|>).
@pytorch-bot

pytorch-bot Bot commented Aug 22, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22042

Note: Links to docs will display an error until the docs builds have been completed.

⚠️ 20 Awaiting Approval

As of commit 3e14c95 with merge base e4576d0 (image):

AWAITING APPROVAL - The following workflows need approval before CI can run:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@linux-foundation-easycla

Copy link
Copy Markdown

CLA Missing ID

  • ❌ The email address for the commit (3e14c95) is not linked to the GitHub account, preventing the EasyCLA check. Consult this Help Article and GitHub Help to resolve. (To view the commit's email address, add .patch at the end of this PR page's URL.) For further assistance with EasyCLA, please visit our EasyCLA portal and chat with our support bot.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 22, 2026
@github-actions

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants