A tokenizer may split text into words or word fragments. For other data, such as transaction sequences, a system can instead encode categories or ranges so events become a sequence of tokens.
The choice affects what distinctions the model can learn. Coarse categories can hide differences, while a very large vocabulary can make rare units harder to learn.
ELI5
Imagine cutting text into pieces from a vocabulary before giving it to a model.
For example, one familiar word may be one token while a less familiar word is represented by several smaller pieces.
Are tokens always words?
No. They can represent word fragments, symbols or units designed for other kinds of data.
Why does the choice matter?
It affects which patterns the model can see and how much detail it can retain.
