Block-sparse attention groups tokens into contiguous blocks and selects which blocks should interact. Computation is performed only for the selected block pairs, avoiding a large number of individual token comparisons. Block-level operations also map efficiently to matrix hardware, which can make the theoretical reduction useful in real deployments.
Different implementations use fixed local blocks, global blocks, learned relevance scores, or a hybrid pattern. The method is well suited to long sequences, but its usefulness depends on selecting blocks that contain the necessary context. Coarse blocks may include irrelevant tokens, while aggressive pruning may exclude an important passage.
