Sparse attention is an attention pattern that limits which token pairs interact. Instead of computing every possible relationship in a sequence, the model uses local windows, selected global tokens, learned routing or another structured subset.
Reducing the number of comparisons can lower memory use and serving cost, especially for long contexts or media generation. The tradeoff is that a poorly chosen sparse pattern may miss important distant information, so designs must preserve the connections needed by the task.
ELI5
Sparse attention lets a model focus on selected parts of a long input instead of comparing every part with every other part. This reduces the amount of work and memory the model needs.
For example, when reading a long book, a model might focus mainly on nearby sentences plus a few important chapter markers. That can be faster, but a poor selection could cause it to miss a clue from a distant chapter.
How does sparse attention improve efficiency?
It computes fewer token-to-token relationships than full attention. This can reduce memory and processing requirements as the input sequence grows.
Does sparse attention reduce model quality?
It can if the pattern excludes useful information. Well-designed methods preserve important local and global connections while avoiding many low-value comparisons.
