Language model token entropy is high when probability is spread across many plausible next tokens and low when one or a few continuations dominate. It is calculated from the next-token probability distribution and changes with context, decoding settings, and model behavior.
Watermarking has more flexibility at high-entropy steps because several substitutions can remain natural. Low-entropy tasks provide less room to encode a signal without changing meaning or quality, which makes short factual answers difficult to watermark robustly.
ELI5
Language model token entropy measures how uncertain the model is about its next token. Entropy is high when many choices are plausible and low when one or two choices dominate.
For example, creative prose may offer many natural next words, while a short factual answer may have only one correct continuation. Watermarks have more room to choose among tokens in the high-entropy case without hurting meaning.
