In full self-attention, each token is compared with every token in the sequence. If the sequence length doubles, the number of pairwise attention relationships grows by about four times. This quadratic scaling can sharply increase computation and memory use for long prompts, documents, or conversations.
Architectures mitigate the cost through sparse or local attention, block selection, compressed memory, linear-attention approximations, or hybrids that retain occasional global access. The practical bottleneck depends on implementation and hardware, but quadratic growth remains a central reason that long-context inference needs specialized techniques.
ELI5
Quadratic attention complexity describes how full attention becomes much more expensive as an AI reads longer input. Every token compares with every other token, so doubling the input creates about four times as many token pairs to consider.
For example, processing 2,000 tokens involves roughly four times the pairwise attention work of 1,000 tokens. Long-context systems reduce this cost with methods that focus on selected nearby blocks, compress older information, or approximate full attention.
Why is full attention described as quadratic?
A sequence of n tokens produces roughly n multiplied by n token-pair relationships, so the attention work grows with the square of sequence length.
How can models reduce quadratic attention costs?
They can restrict attention to local or selected tokens, process blocks, compress older context, or use approximate attention mechanisms with better scaling.
