In full self-attention, each token is compared with every token in the sequence. If the sequence length doubles, the number of pairwise attention relationships grows by about four times. This quadratic scaling can sharply increase computation and memory use for long prompts, documents, or conversations.
Architectures mitigate the cost through sparse or local attention, block selection, compressed memory, linear-attention approximations, or hybrids that retain occasional global access. The practical bottleneck depends on implementation and hardware, but quadratic growth remains a central reason that long-context inference needs specialized techniques.
Acronyms and aliases
quadratic attention cost synonym
Related terms
Frequently asked questions
Why is full attention described as quadratic?
A sequence of n tokens produces roughly n multiplied by n token-pair relationships, so the attention work grows with the square of sequence length.
How can models reduce quadratic attention costs?
They can restrict attention to local or selected tokens, process blocks, compress older context, or use approximate attention mechanisms with better scaling.
