fix(tokenizer): treat special-token strings as text to prevent batch crashes (#3110)#3111

LauraGPT merged 1 commit into

modelscope:mainmodelscope/FunASR:mainfrom

14790897:fix/tokenizer-special-tokens-311014790897/FunASR:fix/tokenizer-special-tokens-3110Copy head branch name to clipboard