Skip to content

fix(tokenizer): do not prepend a BOS token the tokenizer does not have - #1634

Open
Chinmayrawat15 wants to merge 1 commit into
TransformerLensOrg:dev-4.xfrom
Chinmayrawat15:fix/tokenizer-bos-prepend
Open

fix(tokenizer): do not prepend a BOS token the tokenizer does not have#1634
Chinmayrawat15 wants to merge 1 commit into
TransformerLensOrg:dev-4.xfrom
Chinmayrawat15:fix/tokenizer-bos-prepend