Prefill-decode disaggregation separates the compute-heavy stage that processes input tokens from the iterative stage that generates output tokens. Each pool can use scheduling and hardware choices suited to its different compute, memory and latency characteristics.
The design can improve utilization and tail latency for long-context workloads at suitable concurrency. It also introduces the cost of transferring key-value cache state, so a fast network and careful workload analysis are required.
Acronyms and aliases
P-D disaggregation acronymdisaggregated prefill and decode variant
Related terms
Frequently asked questions
What is the prefill stage in model inference?
Prefill processes the supplied prompt in parallel and creates the attention state needed before the model begins generating new output tokens.
When can prefill-decode disaggregation perform poorly?
It may add more transfer and coordination overhead than it saves for short prompts, low traffic, extreme concurrency or slow network fabrics.
