System prompt caching takes advantage of the fact that many turns in a session begin with identical instructions. After the model processes that prefix, the serving system retains the corresponding key-value cache and reuses it for a later request.
Reuse can substantially reduce prompt-processing work and response startup time. The benefit depends on routing the request to a worker that still holds the prefix and on eviction policies that preserve high-value entries without exhausting memory.
Acronyms and aliases
system instruction caching synonymcached system prompt variant
Related terms
Frequently asked questions
What does system prompt caching save?
It saves the intermediate attention state created while processing repeated system instructions, avoiding the need to recompute that prefix for every turn.
When does a system prompt cache miss occur?
A miss occurs when the required prefix is absent, changed, expired or located on a different worker that cannot reuse the stored state efficiently.
