A drafting process proposes a sequence of text pieces. The target model evaluates the proposals together, accepting eligible pieces and correcting the rest under the method’s verification rules.
Speed depends on accepted drafts and the overhead of proposing and checking them. The original exact method preserves the target model’s output distribution; other implementations need their own correctness and performance evaluation.
ELI5
Speculative decoding speeds up AI text generation by using a cheaper method to draft several likely tokens and letting the main model verify them together. Matching tokens are accepted, while incorrect proposals are discarded or replaced.
For example, a small draft model may suggest the next short phrase and a larger model checks it in parallel. The method helps most when drafts are accurate and verification costs less than generating every token one by one.
Are drafts accepted without checking?
No. Verification by the target model is the essential part of the method.
Does speculative decoding always make generation faster?
No. Draft acceptance and processing overhead determine the benefit for the measured workload.
