Key-value cache-aware routing scores inference workers by both capacity and prefix locality. A worker that already holds the attention state for a session or shared prompt can continue generation without repeating expensive prompt processing.
Routing only by cache locality can overload one worker, while routing only by load discards useful cached work. Practical schedulers balance both signals and account for the cost of moving cache state across the network.
ELI5
Key-value cache-aware routing sends an AI request to a model worker that already holds useful attention state for the same session or shared prompt. Reusing that state avoids processing the full prefix again.
For example, later messages in a long chat can return to the worker that cached the earlier conversation. The router must also consider current load because sending every related request to one busy worker can create a new delay.
