A tokenizer divides input into units such as whole words, word fragments, punctuation, bytes, or special symbols and maps them to numeric identifiers. The model is trained and executed using those identifiers rather than raw text.
Tokenizer behavior affects context usage, multilingual efficiency, code handling, and compatibility with a model checkpoint. Distinctive tokenization can also provide evidence about a model’s likely family, though it is not conclusive by itself.
