Language model token entropy is high when probability is spread across many plausible next tokens and low when one or a few continuations dominate. It is calculated from the next-token probability distribution and changes with context, decoding settings, and model behavior.
Watermarking has more flexibility at high-entropy steps because several substitutions can remain natural. Low-entropy tasks provide less room to encode a signal without changing meaning or quality, which makes short factual answers difficult to watermark robustly.
