In a transformer language model, previously processed tokens contribute key and value representations used by attention. Keeping those representations avoids recomputing the entire preceding sequence for every new token.
The cache can consume substantial memory, especially with long inputs or many simultaneous requests. Reuse also depends on compatible model settings and matching context, so an old cache cannot be applied blindly to unrelated input.
ELI5
A key-value cache keeps useful working information from what a model has already processed. The model can reuse it as it generates the next part of an answer.
For example, while writing a long response, a model can consult retained information about earlier text instead of processing all of that text from scratch at every step. Keeping that working information also takes memory.
Is this the same as saving a final answer?
No. It stores intermediate attention information rather than simply returning a previously completed response.
Why can long conversations use more memory?
More processed tokens can require more cached attention information, depending on the model architecture and runtime.
