Sparse attention is an attention pattern that limits which token pairs interact. Instead of computing every possible relationship in a sequence, the model uses local windows, selected global tokens, learned routing or another structured subset.
Reducing the number of comparisons can lower memory use and serving cost, especially for long contexts or media generation. The tradeoff is that a poorly chosen sparse pattern may miss important distant information, so designs must preserve the connections needed by the task.
