Skip to content

fix(tokenizer): do not strip a BOS token the tokenizer does not have - #1629

Merged
jlarson4 merged 1 commit into
TransformerLensOrg:dev-4.xfrom
sohv:fix/tokenizer-bos-removal
Aug 10, 2026
Merged

fix(tokenizer): do not strip a BOS token the tokenizer does not have#1629
jlarson4 merged 1 commit into
TransformerLensOrg:dev-4.xfrom
sohv:fix/tokenizer-bos-removal