What is a tokenizer?

Definition

A tokenizer divides input into units such as whole words, word fragments, punctuation, bytes, or special symbols and maps them to numeric identifiers. The model is trained and executed using those identifiers rather than raw text.

Tokenizer behavior affects context usage, multilingual efficiency, code handling, and compatibility with a model checkpoint. Distinctive tokenization can also provide evidence about a model’s likely family, though it is not conclusive by itself.

ELI5

A tokenizer is like a text-cutting machine for an AI model. It breaks a sentence into smaller pieces called tokens, and each piece gets a number that the model can work with.

For example, the word "unhelpful" might become one piece or several pieces such as "un" and "helpful." Different tokenizers divide text differently, so the same sentence can take more or less space in different models.

Why do AI models use tokenizers?

Models operate on numeric representations, so tokenizers provide a consistent mapping between human-readable input and the identifiers used during training and inference.

Can tokenizer behavior identify a model?

It can provide a useful clue because model families often use characteristic vocabularies and rules, but implementations can be copied or modified.

Definition card for tokenizer: What is a tokenizer?

A tokenizer converts text or other model input into token identifiers that an AI model can process.