Draft Model

How Draft Models Work in Speculative Decoding

A draft model is the fast proposer in speculative decoding. It predicts several next tokens cheaply, then the larger target model checks those proposals, so the system can trade a little extra guessing for much lower end-to-end latency when the draft’s guesses are often accepted.

The key idea is division of labour. The draft model is not trying to be the final authority on output quality, it is trying to produce plausible continuations quickly enough that the target model can validate multiple tokens in fewer expensive forward passes.

Why Acceptance Rate Determines the Speedup

The value of a draft model depends on alignment with the target model. When the draft produces token sequences that the target model would also choose, the acceptance rate rises and the decoder saves work. When the predictions diverge, the system spends time proposing tokens that get rejected, which erodes the latency gain.

This makes draft-model quality a relative property, not an absolute one. A smaller model can be highly effective if it is well matched to the target distribution, even if it would look weak on standalone benchmark tasks.

Model Pairing, Size Trade-offs, and Quality Drift

Draft models are usually selected to be much cheaper than the target model, but not so cheap that their predictions become noisy. The practical question is whether the draft is “good enough” to keep the acceptance pipeline efficient at the prompt types and sequence lengths the system actually sees.

Because speculative decoding depends on continued agreement between two models, drift in prompt mix, domain, or instruction style can reduce gains. A pairing that works well for one workload may perform poorly for another if the smaller model’s token preferences no longer resemble the target model’s behaviour.

Where Draft Models Fit in the Inference Pipeline

Draft models are an inference optimisation, not a separate product category. They sit inside a larger decoding strategy that aims to reduce the cost of generation without changing the final model that users rely on for output quality.

In practice, the draft model must be fast, compatible with the target model’s tokenisation and output style, and stable enough that rejection overhead does not dominate. The best implementations treat it as a throughput tool, then measure acceptance rate and latency together rather than assuming a smaller model automatically improves performance.

Risk and Threat Considerations

Draft models introduce a performance and trust risk when the proposed tokens diverge from the target model too often. In production, that usually shows up as wasted compute, unpredictable latency, and a false sense of efficiency if teams measure only draft throughput instead of end-to-end acceptance.

Failure mechanism: Misalignment between the draft and target model lowers acceptance, increases rejection churn, and can make speculative decoding slower than ordinary generation for some prompts or workloads.

Impact: Systems may miss latency targets, waste inference budget, and deliver unstable user experience, especially when model versions, prompting style, or task mix change.

Practitioner Guidance

Why practitioners should care: A draft model is only useful when its cheap guesses consistently match the target model’s likely next tokens. Treat acceptance rate as the primary operational signal, because raw draft speed alone can hide a poor deployment choice.

What to watch for: Monitor acceptance by workload, not just in aggregate. A pairing that looks strong on short prompts may underperform on longer or more structured generations, so the model should be validated against the real traffic it will serve.

Related resources from NHI Mgmt Group

• What is the difference between a schema draft and a tested authorization model?
• What is the Model Context Protocol (MCP) and why does it matter for security?
• What does AI model abuse reveal about the current NHI threat surface?
• Why do attackers often check model availability before trying to generate content?