Prefill-decode disaggregation separates the compute-heavy stage that processes input tokens from the iterative stage that generates output tokens. Each pool can use scheduling and hardware choices suited to its different compute, memory and latency characteristics.
The design can improve utilization and tail latency for long-context workloads at suitable concurrency. It also introduces the cost of transferring key-value cache state, so a fast network and careful workload analysis are required.
ELI5
Prefill-decode disaggregation splits AI text generation into two groups of computers. One group reads and processes the prompt, while another group produces the answer one token at a time, so each group can be tuned for its own kind of work.
For example, a long document may need powerful machines for the initial reading stage, while many smaller machines handle the ongoing output. The split can improve speed and capacity, but only if moving the model's temporary working data between the groups is fast enough.
What is the prefill stage in model inference?
Prefill processes the supplied prompt in parallel and creates the attention state needed before the model begins generating new output tokens.
When can prefill-decode disaggregation perform poorly?
It may add more transfer and coordination overhead than it saves for short prompts, low traffic, extreme concurrency or slow network fabrics.
