What is system prompt caching?

Definition

System prompt caching takes advantage of the fact that many turns in a session begin with identical instructions. After the model processes that prefix, the serving system retains the corresponding key-value cache and reuses it for a later request.

Reuse can substantially reduce prompt-processing work and response startup time. The benefit depends on routing the request to a worker that still holds the prefix and on eviction policies that preserve high-value entries without exhausting memory.

ELI5

System prompt caching saves reusable model state after processing instructions that appear at the beginning of many requests. Later requests with the same prefix can skip repeating part of that computation.

For example, an agent session may reuse a long project policy on every turn. Keeping its processed state on the right worker can reduce response startup time, while eviction rules balance reuse against limited accelerator memory.

What does system prompt caching save?

It saves the intermediate attention state created while processing repeated system instructions, avoiding the need to recompute that prefix for every turn.

When does a system prompt cache miss occur?

A miss occurs when the required prefix is absent, changed, expired or located on a different worker that cannot reuse the stored state efficiently.

  1. Yuchen Fama and Ashish Kamra beside the words Smarter Routing for Agentic Inference
Definition card for system prompt caching: What is system prompt caching?

System prompt caching stores reusable model state for repeated system instructions so later requests can avoid processing the same prefix again.