What is prefix caching?

Definition

When requests begin with the same tokens, a serving system can reuse compatible intermediate model state from that shared prefix. This reduces repeated work on the beginning of the input rather than requiring the model to process it from scratch for every request.

The benefit depends on exact reusable input, cache capacity and request routing. A request sent to a replica without the relevant state may miss the cache. Prefix caching is different from simply returning a saved answer, since the model still processes new input and generates the requested continuation.

ELI5

Imagine several people asking questions about the same long document. A helper can reuse some work already done to understand that common beginning instead of repeating it every time.

For example, a support assistant might receive the same instruction page followed by different customer questions. Reusing the shared opening can save work while each new question still needs its own answer.

Is prefix caching the same as caching an answer?

No. It reuses intermediate computation for a shared input prefix; the system still handles the new input and continuation.

Why does request routing affect prefix caching?

The reusable state must be available where the request runs. Sending related requests to a replica without that state can lose the benefit.

  1. Portrait of Yunmo Koo on a black background with the blue and white headline "INFERENCE BEYOND WEIGHTS".
Definition card for prefix caching: What is prefix caching?

Prefix caching reuses model computation for an identical starting token sequence shared by multiple requests.