Session-aware cache eviction uses information about conversations or agent tasks to decide which key-value cache entries should remain in scarce accelerator memory. A recently paused session may deserve retention because another turn is likely, even if newer unrelated requests have arrived.
The policy can combine recency, session state, prefix size, reuse probability and reconstruction cost. Poor eviction wastes memory or forces repeated prompt processing, so the optimal balance depends on traffic and available offload tiers.
ELI5
Session-aware cache eviction decides which AI working-memory entries to remove by considering whether an active conversation or agent task is likely to need them again. It uses more information than simply deleting the oldest entry.
For example, a recently paused coding session with a large prompt may stay in fast memory because another turn is likely. Removing it too soon would force the system to process the whole prompt again, while keeping unused sessions wastes scarce memory.
How is session-aware cache eviction different from least-recently-used eviction?
It considers the state and likely return of a session, while a simple least-recently-used policy looks primarily at access recency.
Where can evicted model cache data go?
A system may discard it or move it to a slower memory or storage tier, depending on transfer cost, capacity and expected reuse.
