What is multi-token prediction?

Definition

Multi-token prediction extends the usual next-token objective by producing candidates for multiple positions ahead. A serving system can verify or accept compatible drafts in groups, reducing the number of strictly sequential generation steps.

The method can improve generation speed when future-token drafts are accurate enough. Benefits depend on architecture, acceptance behavior, hardware, and workload, and incorrect drafts still need correction or rejection.

ELI5

Multi-token prediction lets an AI look several words or word pieces ahead instead of producing only the next one. Good drafts can be accepted together, which reduces the number of slow one-by-one steps.

For example, the model may draft a short phrase and then verify that the tokens fit. When the draft is correct, output arrives faster; when it is wrong, the system discards or repairs part of it.

Frequently asked questions

Why can multi-token prediction speed up generation?

It can accept several verified future tokens in one stage instead of waiting for a separate full model step for every token.

Does multi-token prediction always improve speed?

No. Gains depend on draft accuracy, verification overhead, hardware, implementation, and the kind of workload.

Videos explaining multi-token prediction

  1. The words Open Weights Move Faster beside an abstract specialist-model distillation