Key-value cache-aware routing scores inference workers by both capacity and prefix locality. A worker that already holds the attention state for a session or shared prompt can continue generation without repeating expensive prompt processing.
Routing only by cache locality can overload one worker, while routing only by load discards useful cached work. Practical schedulers balance both signals and account for the cost of moving cache state across the network.
It is stored attention state for tokens the model has already processed, allowing later generation to reuse that work instead of recomputing the entire prefix.
Why does cache-aware routing reduce response time?
It can place a follow-up request on the worker that already holds its prefix state, shortening the prompt-processing stage before new tokens appear.
