fix(tokenizer): treat special-token strings as text to prevent batch crashes (#3110)

[14790897]

Summary

When transcribing long audio in a batch with FunAudioLLM/Fun-ASR-Nano-2512 (+ vad_model=fsmn-vad, spk_model=campplus), a single sample can crash with Encountered text corresponding to disallowed special token '<|no|>'. and abort the entire batch, even though every other sample transcribes fine (stable RTF). Closes #3110.

Root cause. The model occasionally emits a special-token string (e.g. <|no|>, the Norwegian language tag) as part of its transcription text. That text is then re-encoded for forced alignment / loss computation:

# funasr/models/fun_asr_nano/model.py:928
target_ids = torch.tensor(self.ctc_tokenizer.encode(result["text"]), dtype=torch.int64)

self.ctc_tokenizer resolves to SenseVoiceTokenizer (funasr/tokenizer/whisper_tokenizer.py:30) → funasr/models/sense_voice/whisper_lib/tokenizer.py::Tokenizer, whose encode forwards straight to tiktoken.Encoding.encode:

def encode(self, text, **kwargs):
    return self.encoding.encode(text, **kwargs)

tiktoken defaults to disallowed_special="all", so any text containing a special-token string raises ValueError, taking down the whole batch. model.py:921 already strips <|nospeech|> from the CTC text — which confirms ASR output can leak special-token strings — but the allow-list isn't exhaustive (language tags like <|no|> slip through).

Fix. Default disallowed_special=() in Tokenizer.encode so special-token strings are treated as ordinary text unless a caller explicitly opts into stricter behaviour:

def encode(self, text, **kwargs):
    kwargs.setdefault("disallowed_special", ())
    return self.encoding.encode(text, **kwargs)

Type of change

• Bug fix
• Documentation
• Example or demo
• Runtime or deployment
• Benchmark or evaluation
• Model/training change

Validation

• python -m compileall funasr examples tests
• Docs or links checked
• Runtime/deployment command tested

python -m compileall -q funasr examples tests exits 0 (only two pre-existing SyntaxWarnings in examples/industrial_data_pretraining/fun_asr_nano/tools/format5res.py, untouched by this PR).

Adds tests/test_sensevoice_tokenizer_special_tokens.py. It builds a minimal in-memory tiktoken.Encoding (byte-level BPE + the specials Tokenizer.__post_init__ needs), so it runs without the multilingual.tiktoken vocab (ships with the model, not this repo), without a GPU and without a model download:

• plain text encodes
• text containing <|no|> / <|nospeech|> no longer raises (regression for ASR Inference Failure: Disallowed special<|no|>` causes single-sample transcription crash in batch processing #3110)
• explicit disallowed_special="all" still raises (back-compat)
• allowed_special="all" still returns the special-token id (back-compat for decoding.py)

Local exercise of the fix against the minimal encoding (importlib-loaded tokenizer.py to bypass the heavy funasr package import chain on a machine without torch/kaldiio):

1. raw tiktoken encode("<|no|>")                        -> raises (reproduces #3110)
2. Tokenizer.encode("hello <|no|> world")               -> OK, 18 ids (fixed)
3. Tokenizer.encode("<|no|>", disallowed_special="all") -> still raises (back-compat)
4. Tokenizer.encode("<|no|>", allowed_special="all")   -> [257] == <|no|> id (back-compat)

User impact

Production deployers running Fun-ASR-Nano (and any model reusing the SenseVoice Tokenizer) on batch / streaming ASR pipelines. Today a single bad sample aborts the whole batch; after this PR that sample is transcribed with the stray token treated as text and the batch completes. Researchers using ctc_tokenizer.encode for forced alignment / loss hit the same crash.

Notes for reviewers

Why the tokenizer, not the call sites. There are multiple call sites that re-encode ASR output — fun_asr_nano/model.py:443, 495, 923, 928, inference_vllm.py:668, inference_vllm_pipeline.py:348. Patching the tokenizer covers all of them in one place, and also protects SenseVoice and any future model that reuses this Tokenizer.

Backward compatibility. setdefault keeps the change fully backward compatible:

• Callers that pass disallowed_special=... are untouched.
• decoding.py's tokenizer.encode(prompt, allowed_special="all") still encodes special tokens as their token ids (verified in case 4 above).
• The internal non_speech_tokens cached_property calls self.encoding.encode(...) directly (not self.encode), so it's unaffected.

Scope. One-line behavioural default in Tokenizer.encode + regression tests. No model weights, no config, no public API change.

[gemini-code-assist]

Code Review

This pull request addresses issue #3110 by updating the encode method in the SenseVoice tokenizer to treat special-token strings as ordinary text by default. This is achieved by setting disallowed_special to an empty tuple in kwargs if not already specified, preventing crashes when tiktoken encounters special tokens in transcription text. Additionally, a comprehensive suite of unit tests has been added to verify this behavior and ensure regression protection. I have no further feedback to provide as there are no review comments to assess.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.