Linear attention uses mathematical rearrangements, kernel functions or recurrent summaries to avoid constructing the complete token-by-token attention matrix. This can make long sequences cheaper to process in memory and computation.
The efficiency gain comes with design tradeoffs. Approximations may lose some expressive detail, and different linear attention methods behave differently on recall and reasoning tasks. Hybrid architectures often combine linear and conventional or sparse attention to balance quality and cost.
ELI5
Linear attention is a way for a model to process long sequences without comparing every token with every other token. It uses summaries or mathematical shortcuts so the work grows more slowly as the sequence gets longer.
For example, a model reading a long document can maintain a compact running summary instead of building every possible word-to-word connection. That saves memory, but some methods may miss details that full attention would preserve.
