Speculative decoding improves local LLM throughput without quality loss

At a glance

What this is: Jozu explains speculative decoding as a way to accelerate local LLM generation by splitting proposal and verification across a draft model and a target model.

Why it matters: It matters because AI teams running local or self-hosted inference need to balance latency, GPU capacity, and model governance without sacrificing output consistency or operational control.

By the numbers:

• The target model can run around 30 to 50 tokens per second on an RTX 4070 Ti for a 7B model, while a modern CPU reaches around 5 to 10 tokens per second.
• The tutorial's comparison shows about 19 seconds with speculative decoding versus about 28 seconds without it for 2048 tokens.

Context

Local LLM inference often shifts privacy and cost concerns in the right direction, but it exposes a different bottleneck: throughput. When generation is sequential, each token depends on a fresh forward pass, so latency becomes visible during long outputs and reasoning-heavy tasks.

That performance problem is not just a model-engineering detail. In production AI systems, inference speed affects user experience, GPU utilisation, and the operational window in which prompts, outputs, and surrounding artefacts must be governed.

Key questions

Q: How should teams decide whether speculative decoding is worth using in local LLM deployments?

A: Use it when generation latency is a real bottleneck and you can support two-model inference without exhausting memory. The best candidates are compatible models from the same family, with a draft model that is much smaller than the target. If acceptance rates are low, the extra draft work can outweigh the benefit.

Q: Why do tokenizer compatibility and model family matter so much for speculative decoding?

A: Speculative decoding only works when the draft model and target model speak the same token language. Shared tokenizers keep proposal and verification aligned, while same-family models usually produce more similar token distributions, which increases acceptance. Without that alignment, the verifier rejects too many tokens and the speedup disappears.

Q: What operational risks appear when local inference runs multiple model artefacts together?

A: The main risk is losing control over what actually ran. When the target model, draft model, configs, and datasets are deployed together, teams need strong versioning, integrity checks, and environment isolation to preserve reproducibility and prevent unapproved changes from entering production.

Q: How should AI teams size infrastructure for speculative decoding?

A: They should size for the combined footprint of the target model, the draft model, and runtime overhead, including KV cache. That means planning memory and storage as a multi-artefact workload, not as a single-model deployment. If the hardware margin is too thin, the optimisation will fail before it creates any value.

Technical breakdown

How speculative decoding reduces sequential token latency

Speculative decoding changes the generation loop from one-token-at-a-time execution to a two-model workflow. A small draft model predicts several future tokens quickly, then the larger target model validates those tokens in one batched pass. When the target agrees, the accepted tokens are emitted at once, which reduces the number of expensive forward passes. The speedup depends on how often the draft model's guesses match the target model's distribution, so the method trades a little extra work on the draft side for fewer target passes overall.

Practical implication: Use speculative decoding where latency matters and measure acceptance rate, not just raw model size, before assuming a throughput gain.

Why tokenizer and model family compatibility decide whether the pair works

The draft and target models must share the same tokenizer because speculative decoding compares proposed token IDs against the target model's expected outputs. If tokenisation differs, the verification step cannot line up cleanly and the process breaks. The strongest results usually come from models in the same family because their learned token distributions are more similar, which raises acceptance rates. A large size gap also matters: the draft has to be cheap enough that its compute cost does not erase the savings from reduced target-model passes. See the LLM Security Guide.

Practical implication: Treat tokenizer compatibility and family alignment as hard requirements when selecting draft-target pairs for production inference.

Why memory planning changes when two models run at once

Speculative decoding loads both the draft and the target model into memory at the same time, so the hardware requirement is the combined weight footprint plus inference overhead and KV cache. Quantisation lowers the footprint of each model, but it does not eliminate the need to budget for both. That is why a deployment that fits one model comfortably can still fail once speculative decoding is enabled. Storage also matters because model loading time becomes more noticeable when multiple artefacts must be read and initialised together. See the AI Infrastructure Workload Identity Guide.

Practical implication: Size GPU memory for the sum of both models plus overhead before enabling speculative decoding in local or containerised deployments.

Breaches seen in the wild

• ShadowRay 2024: Attackers took over internet-exposed Ray AI clusters via the disputed CVE-2023-48022, exposing cloud keys, AI API tokens and GPU compute.
• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Speculative decoding is a throughput optimisation problem, but its governance implications sit in the model pipeline, not the prompt layer. Once teams package a target model and a draft model together, the control question becomes which artefacts were approved, versioned, and deployed together. That puts model provenance, packaging integrity, and inference reproducibility in scope for AI operations. The practical conclusion is that performance tuning should not bypass artefact governance.

Model packaging is now part of AI identity control. Jozu's ModelKit approach treats weights, datasets, configs, and documentation as a single governed artefact, which is the right abstraction for AI supply chains. In practice, the identity of the running model must be traceable back to the exact bundle that produced it, especially when inference behaviour changes with a draft model added to the stack. Teams should govern the bundle, not only the endpoint.

Speculative decoding narrows the latency gap between local and cloud inference, which makes local AI more operationally attractive but also more governable in familiar ways. Once inference shifts on-prem or into self-hosted environments, the hard problems are resource sizing, access to model artefacts, and verification of what actually ran. That means AI platform teams need controls that look a lot like software supply chain and workload governance, not just model selection. The practical conclusion is to connect inference performance work to provenance and deployment controls.

The real risk is confusing faster generation with lower operational risk. A faster local model still handles the same sensitive prompts, the same outputs, and the same deployment artefacts. If speed improvements lead teams to skip scanning, signing, or version discipline, they have traded one bottleneck for another governance gap. The practical conclusion is to pair inference optimisation with explicit controls for artefact integrity and environment isolation.

From our research library:

• 71% of organizations use third-party APIs, according to Gartner’s 2024 data.
• Read next: AI Infrastructure Workload Identity Guide

What this signals

AI throughput is now a governance variable: when teams optimise local inference, they also expand the number of artefacts and runtime conditions that must stay aligned. The control problem shifts from choosing the fastest model to proving that the draft model, target model, tokenizer, and deployment bundle still represent the intended system state.

Model provenance becomes the anchor control for local AI operations. If teams cannot prove which weights, configs, and supporting artefacts were shipped together, faster inference only makes the failure harder to investigate. That is why local inference programmes should pair performance work with bundle-level traceability and integrity verification.

AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026. The practical lesson is that the operational surface around inference often becomes the larger exposure zone, so governance has to follow the supporting stack as closely as the model itself.

For practitioners

• Define draft-target pairing criteria Require identical tokenizers, a shared model family, and a minimum size gap before approving speculative decoding in production workloads.
• Budget memory for both models and cache overhead Size GPU capacity for the combined weights of the target model, draft model, and KV cache instead of assuming the larger model alone determines fit.
• Treat model bundles as governed artefacts Version the model weights, configuration files, and supporting assets together so the deployed inference stack is always reproducible and attributable.
• Measure acceptance rate before tuning for speed Track how often the target model accepts draft tokens, because low acceptance can erase the performance gain and indicate a poor pairing choice.

Key takeaways

• Speculative decoding speeds up local LLM generation by splitting work between a fast draft model and a verified target model.
• The technique depends on shared tokenizers, model-family compatibility, and enough memory to hold both models plus runtime overhead.
• For AI teams, the operational issue is not only latency but also whether the model bundle remains identifiable, reproducible, and policy-governed.

Related NHI Mgmt Group resources: AI Infrastructure Workload Identity Guide · LLM Security Guide · AI Governance Guide

Key terms

• Speculative Decoding: A text generation method that uses a small draft model to propose tokens ahead of time and a larger target model to verify them in batches. It reduces latency by replacing some expensive forward passes with cheaper preview steps while preserving the target model's final output distribution.
• Draft Model: A smaller, faster model used to guess several upcoming tokens during speculative decoding. Its value depends on how well its predictions align with the target model, because higher acceptance rates produce more speedup and fewer wasted proposals.
• Target Model: The primary high-quality model that validates the draft model's proposed tokens and determines the final output. In speculative decoding, it remains the source of truth, so compatibility, alignment, and verification cost all shape the performance outcome.
• ModelKit: A packaged AI artefact bundle that groups weights, datasets, configuration files, documentation, and related assets into one versioned unit. In operational terms, it is the control point that preserves reproducibility, provenance, and deployable state across the AI lifecycle.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore it when your programme needs a stronger bridge between identity controls and AI operations.