Linear attention uses mathematical rearrangements, kernel functions or recurrent summaries to avoid constructing the complete token-by-token attention matrix. This can make long sequences cheaper to process in memory and computation.
The efficiency gain comes with design tradeoffs. Approximations may lose some expressive detail, and different linear attention methods behave differently on recall and reasoning tasks. Hybrid architectures often combine linear and conventional or sparse attention to balance quality and cost.
ELI5
Linear attention is a way for a model to process long sequences without comparing every token with every other token. It uses summaries or mathematical shortcuts so the work grows more slowly as the sequence gets longer.
For example, a model reading a long document can maintain a compact running summary instead of building every possible word-to-word connection. That saves memory, but some methods may miss details that full attention would preserve.
Why is linear attention useful for long context?
Its computation can grow much more slowly as the sequence becomes longer. That can make large documents, extended conversations or long media sequences practical to process.
Is linear attention always better than full attention?
No. Full attention can represent detailed token relationships directly. Linear methods trade some representation choices for efficiency, so the best approach depends on the task and implementation.
