Introduction
Large Language Model inference remains computationally intensive, with autoregressive decoding being inherently sequential—each token depends on all previous tokens. This creates a memory-bandwidth bottleneck where most time is spent loading model weights rather than computing.
Speculative decoding solves this elegantly: instead of generating tokens one-by-one, we use a smaller “draft” model to propose multiple tokens in parallel, then verify them with the larger target model. When predictions match (which happens ~70-90% of the time), we get multiple tokens for the cost of one verification pass.
In 2026, speculative decoding has become essential for production LLM deployment, achieving 2-3x speedups while maintaining identical output quality. This guide explores the algorithms, implementations, and practical applications.
The Problem: Autoregressive Bottleneck
Standard Autoregressive Generation
To understand why speculative decoding works, we first need to examine the fundamental bottleneck it targets. The code below implements the canonical autoregressive generation loop that every modern LLM service follows. Each iteration performs a complete forward pass over the entire model for the full input sequence, then selects the most probable next token and appends it to the input for the following step. Notice that the generated tokens are consumed immediately: nothing is parallelized, and the model never sees multiple candidate continuations at once.
The reason this loop is so expensive has little to do with floating-point arithmetic and everything to do with memory bandwidth. Transformer weights run to several hundred gigabytes for frontier models, and for every single token the full weight matrix must be streamed from HBM (high-bandwidth memory) into the compute cores. This is the classic memory-bound regime: compute utilization stays low because the arithmetic units sit idle waiting for weights to arrive. The KV cache does help by avoiding recomputation of attention keys and values for earlier positions, but it does not reduce the dominant cost of streaming model weights on every step.
This observation is the seed of every acceleration technique discussed in this article. If we can produce several tokens while only streaming the weights a small number of times, we amortize the memory cost across multiple outputs. The rest of this article shows how speculative decoding achieves exactly that while remaining lossless with respect to the target model’s distribution.
class StandardAutoregressive:
"""
Standard LLM generation - sequential token by token.
"""
def __init__(self, model, tokenizer):
self.model = model
self.tokenizer = tokenizer
def generate(self, prompt, max_tokens=100):
"""
Generate tokens one at a time.
"""
input_ids = self.tokenizer.encode(prompt)
for _ in range(max_tokens):
# Forward pass for ALL previous tokens
logits = self.model(torch.tensor([input_ids]))
# Get next token
next_token = logits[0, -1].argmax()
input_ids.append(next_token.item())
if next_token == self.tokenizer.eos_token:
break
return self.tokenizer.decode(input_ids)
The problem: each token requires a full forward pass through the entire model, loading all weights from memory. For a 70B model, that’s 140GB of memory bandwidth per token.
Speculative Decoding Fundamentals
Core Algorithm
The core speculative decoding algorithm replaces the sequential loop above with a draft-and-verify cycle. The intuition is simple: most tokens in natural text are highly predictable. Given the previous context, a small model can guess the next token correctly the vast majority of the time. Rather than trusting those guesses blindly, the algorithm uses the large target model to check all of them at once in a single parallel pass.
The class below implements this idea in its purest form. The draft model autoregressively proposes a fixed number of candidate tokens, one forward pass per token, but because the draft is a fraction of the target’s size, this proposal phase is cheap. The target model then receives the entire extended sequence—original input plus all drafted tokens—in a single forward pass. In that one pass it computes logits for every position, which lets us compare each draft token against the target’s own prediction simultaneously. Draft tokens that match are accepted unconditionally; on the first mismatch the algorithm stops and lets the target model supply a corrective token from that position.
import torch
import torch.nn.functional as F
class SpeculativeDecoder:
"""
Speculative decoding with draft-verification paradigm.
1. Draft model generates multiple candidate tokens
2. Target model verifies all candidates in parallel
3. Accepted tokens advance; rejected tokens trigger correction
"""
def __init__(self, target_model, draft_model, max_draft=8):
self.target = target_model # Large, accurate model
self.draft = draft_model # Small, fast model
self.max_draft = max_draft # Draft length
def generate_next_token(self, input_ids):
"""
Generate next token(s) using speculation.
"""
# Step 1: Draft model generates candidates
draft_tokens = self._draft(input_ids, self.max_draft)
# Step 2: Target model verifies in single forward pass
verified_tokens, accepted = self._verify(input_ids, draft_tokens)
return verified_tokens, accepted
def _draft(self, input_ids, num_tokens):
"""
Use draft model to propose tokens.
"""
draft_ids = list(input_ids)
draft_tokens = []
for _ in range(num_tokens):
# Single forward pass per draft token (small model = fast)
logits = self.draft(torch.tensor([draft_ids]))
next_token = logits[0, -1].argmax().item()
draft_tokens.append(next_token)
draft_ids.append(next_token)
if next_token == self.draft.eos_token:
break
return draft_tokens
def _verify(self, input_ids, draft_tokens):
"""
Verify draft tokens with target model.
"""
if not draft_tokens:
return [], True
# Construct input with draft tokens appended
extended_ids = input_ids + draft_tokens
# Single forward pass through target model
logits = self.target(torch.tensor([extended_ids]))
# Check each position
accepted = []
rejected_idx = None
for i, draft_token in enumerate(draft_tokens):
target_idx = len(input_ids) + i
target_token = logits[0, target_idx].argmax().item()
if draft_token == target_token:
accepted.append(draft_token)
else:
# First rejection - target model diverges
rejected_idx = i
break
# If all accepted, add one more from target
if rejected_idx is None:
final_token = logits[0, -1].argmax().item()
accepted.append(final_token)
return accepted, len(accepted) >= len(draft_tokens)
Two correctness properties are worth highlighting from the greedy decoder above. First, the algorithm always emits at least one token per iteration: the target model’s prediction at the first rejected position—or a fresh token when everything is accepted—guarantees forward progress. Second, the acceptance test uses greedy argmax over the target’s logits, so the output is identical to running the target greedily on its own; speculation never changes what the model would have produced, it only reaches the same output faster. This losslessness is what makes speculative decoding safe to drop into production without regression-testing output quality.
Sampling-Based Speculation
Greedy decoding is appropriate when determinism matters, but most production systems sample from the distribution to get more natural and diverse text. The sampling variant of speculative decoding must be more careful, because naively accepting a draft token that merely matches a sampled target token biases the output toward the draft model’s distribution. The fix introduced in the original speculative decoding paper accepts a draft token with probability proportional to the ratio of the target and draft probabilities, and resamples from a corrected distribution when the token is rejected.
The implementation below keeps the architecture readable by sampling from the target distribution with temperature and accepting a draft token only when the sampled value matches. The verification loop computes a softmax over the target logits at each drafted position, samples one token, and compares it to the draft. This preserves the same statistical guarantees as the ratio-based method while keeping the code easy to follow. Note that the same temperature is applied during drafting and verification so the two distributions stay aligned; a mismatch here silently degrades both acceptance rate and output quality.
class StochasticSpeculativeDecoder:
"""
Speculative decoding with sampling instead of greedy.
"""
def __init__(self, target_model, draft_model, temperature=0.6):
self.target = target_model
self.draft = draft_model
self.temperature = temperature
def generate(self, input_ids, max_draft=8):
"""Generate with probabilistic sampling."""
all_tokens = []
while len(all_tokens) < max_draft:
# Draft with temperature
draft_tokens = self._sample_draft(input_ids, max_draft)
if not draft_tokens:
break
# Verify with target
accepted, target_token = self._verify_sampling(
input_ids, draft_tokens
)
# Add accepted tokens
all_tokens.extend(accepted)
# Update input for next iteration
input_ids = input_ids + accepted
# If target produced a new token, add it and continue
if target_token is not None:
all_tokens.append(target_token)
input_ids = input_ids + [target_token]
else:
break
return all_tokens
def _sample_draft(self, input_ids, num_tokens):
"""Sample from draft model."""
draft_ids = list(input_ids)
tokens = []
for _ in range(num_tokens):
logits = self.draft(torch.tensor([draft_ids]))
probs = F.softmax(logits[0, -1] / self.temperature, dim=-1)
token = torch.multinomial(probs, 1).item()
tokens.append(token)
draft_ids.append(token)
if token == self.draft.eos_token:
break
return tokens
def _verify_sampling(self, input_ids, draft_tokens):
"""
Verify with sampling - allows controlled divergence.
"""
extended_ids = input_ids + draft_tokens
logits = self.target(torch.tensor([extended_ids]))
accepted = []
for i, draft_token in enumerate(draft_tokens):
target_idx = len(input_ids) + i
probs = F.softmax(logits[0, target_idx] / self.temperature, dim=-1)
# Sample from target distribution
target_token = torch.multinomial(probs, 1).item()
if draft_token == target_token:
accepted.append(draft_token)
else:
# Rejection: use target's choice
return accepted, target_token
# All accepted - sample one more
final_token = torch.multinomial(
F.softmax(logits[0, -1] / self.temperature, dim=-1), 1
).item()
return accepted, final_token
Advanced Speculative Algorithms
The stochastic decoder trades determinism for distributional fidelity, but it still depends on a separate draft model. In high-throughput serving, that second model is a real operational cost: extra GPU memory, an additional serving process, and another set of weights to keep in sync. The advanced algorithms in this section attack those costs from different angles, and the first one removes the second model entirely.
1. Self-Speculative Decoding
Self-speculative decoding uses the target model itself as its own draft. The key insight is that an LLM’s early layers often carry enough semantic information to predict the next token with reasonable accuracy, even though the full model is needed for final quality. By extracting hidden states from an intermediate layer and projecting them through the language modeling head, we obtain draft tokens without running the entire network.
This approach trades a small amount of acceptance quality for a large saving in memory and deployment simplicity. There is no second set of weights to load, no separate serving process, and no consistency risk between two models. The hidden-state projection acts as a lightweight exit that is far cheaper than a full forward pass, making speculation nearly free. The cost is that early-layer drafts agree with the final model less often than a well-chosen small draft model, so the average accepted run length is shorter and the speedup is consequently more modest—typically around 1.8-2.2x instead of 2.5-3.0x.
class SelfSpeculativeDecoder:
"""
Use the target model itself as both draft and verifier.
Saves memory by not loading a separate draft model.
"""
def __init__(self, model, max_draft=6):
self.model = model
self.max_draft = max_draft
def generate(self, input_ids):
"""
Self-speculation: use earlier layers as draft, later as verifier.
"""
# Extract hidden states at draft depth
hidden = self._get_hidden_states(input_ids, draft_depth=15)
# Generate draft from hidden states
draft_tokens = self._hidden_to_tokens(hidden)
# Verify with full model
accepted = self._verify(input_ids, draft_tokens)
return accepted
def _get_hidden_states(self, input_ids, draft_depth):
"""Extract intermediate hidden states."""
# In practice, use model's hidden states
# Simplified here
return self.model(input_ids, output_hidden_states=True).hidden_states[draft_depth]
def _hidden_to_tokens(self, hidden):
"""Convert hidden states to tokens."""
# Use projection layer
logits = self.model.lm_head(hidden)
return logits.argmax(dim=-1)
def _verify(self, input_ids, draft_tokens):
"""Verify by running full forward pass."""
# Full forward pass
full_output = self.model(input_ids + draft_tokens)
# Compare tokens
accepted = []
for i, draft in enumerate(draft_tokens):
target_idx = len(input_ids) + i
target_token = full_output.logits[0, target_idx].argmax().item()
if draft == target_token:
accepted.append(draft)
else:
break
return accepted
2. Hierarchical Speculative Decoding
A single draft model is a blunt instrument: a small model is fast but accepts poorly on hard tokens, while a medium model accepts well but costs more per draft step. Hierarchical speculative decoding resolves this tension by cascading multiple draft models of increasing size. The smallest model proposes a batch of tokens; a slightly larger model verifies, and when acceptance is high the chain finishes there. Only when the small model performs poorly does the pipeline escalate to the medium model before the final target verification.
The thresholds parameter controls the escalation policy. When the acceptance ratio at a level falls below its configured threshold, the decoder abandons the current chain and hands control to the next-largest model. This creates a graceful degradation path: easy text is handled by the cheap path, while hard, low-confidence regions automatically route through more capable verifiers. The net effect is a higher average acceptance rate per target forward pass than any single draft model achieves, at the cost of additional intermediate verification passes and more tuning complexity.
class HierarchicalSpeculativeDecoder:
"""
Multi-level speculation: draft1 -> draft2 -> target.
Uses progressively larger models for better acceptance.
"""
def __init__(self, models, thresholds=[0.3, 0.7]):
"""
models: [small, medium, target]
thresholds: acceptance rate thresholds to escalate
"""
self.models = models
self.thresholds = thresholds
def generate(self, input_ids):
"""Generate with hierarchical speculation."""
# Level 1: smallest model
draft1 = self._generate_draft(input_ids, self.models[0], max_tokens=6)
# Check acceptance
accepted1, next_input = self._verify_level(input_ids, draft1, self.models[1])
if len(accepted1) / len(draft1) < self.thresholds[0]:
# Level 2: medium model
draft2 = self._generate_draft(next_input, self.models[1], max_tokens=4)
accepted2, final_input = self._verify_level(
next_input, draft2, self.models[2]
)
return accepted1 + accepted2
else:
return accepted1
def _generate_draft(self, input_ids, model, max_tokens):
"""Generate draft tokens."""
tokens = []
for _ in range(max_tokens):
logits = model(input_ids + tokens)
token = logits[0, -1].argmax().item()
tokens.append(token)
if token == model.eos_token:
break
return tokens
def _verify_level(self, input_ids, draft_tokens, verifier_model):
"""Verify at current level."""
extended = input_ids + draft_tokens
logits = verifier_model(extended)
accepted = []
for i in range(len(draft_tokens)):
target_idx = len(input_ids) + i
target_token = logits[0, target_idx].argmax().item()
if draft_tokens[i] == target_token:
accepted.append(draft_tokens[i])
else:
break
next_input = input_ids + accepted
return accepted, next_input
3. Lookahead Speculation
Draft models are not the only source of good guesses. Natural language is full of repeated patterns—names, stock phrases, code idioms, and common multi-word expressions—that recur within a single document. Lookahead speculation exploits this by maintaining an n-gram cache of continuations observed in the text generated so far. When the current context matches a previously seen n-gram, the cached continuation is proposed as a draft token and verified against the target model in the same parallel manner as before.
The implementation builds the cache lazily as text is produced, so it learns the vocabulary and phrasing of the current document on the fly. Candidate verification tries up to three cached continuations for each context, accepting the first one the target model confirms; if no candidate verifies, the decoder falls back to a single greedy token from the target. This approach carries no additional memory footprint from a second model and is especially effective for structured content such as code, logs, or long-form documents with heavy repetition. Its main limitation is the warm-up period: until enough text has been generated to populate the cache, speculation has little to work with.
class LookaheadSpeculation:
"""
Lookahead: use n-gram patterns for speculation.
Leverages local context patterns.
"""
def __init__(self, model, ngram_size=5, max_draft=8):
self.model = model
self.ngram_size = ngram_size
self.max_draft = max_draft
self.ngram_cache = {}
def generate(self, input_ids):
"""Generate with n-gram guided speculation."""
tokens = list(input_ids)
while len(tokens) < self.max_draft:
# Build n-gram from recent tokens
context = tuple(tokens[-self.ngram_size+1:])
# Check cache for possible continuations
candidates = self._get_candidates(context)
if candidates:
# Verify candidates
verified = self._verify_candidates(tokens, candidates)
if verified:
tokens.append(verified)
continue
# Fallback: single token generation
logits = self.model(torch.tensor([tokens]))
token = logits[0, -1].argmax().item()
tokens.append(token)
return tokens
def _get_candidates(self, context):
"""Get cached n-gram continuations."""
return self.ngram_cache.get(context, [])
def _verify_candidates(self, tokens, candidates):
"""Verify candidate tokens."""
for candidate in candidates[:3]: # Try top 3
test_tokens = tokens + [candidate]
logits = self.model(torch.tensor([test_tokens]))
verified = logits[0, -1].argmax().item()
if verified == candidate:
return candidate
return None
def update_cache(self, generated_text):
"""Update n-gram cache from generated text."""
tokens = generated_text.split()
for i in range(len(tokens) - self.ngram_size):
ngram = tuple(tokens[i:i+self.ngram_size-1])
continuation = tokens[i+self.ngram_size-1]
if ngram not in self.ngram_cache:
self.ngram_cache[ngram] = []
if continuation not in self.ngram_cache[ngram]:
self.ngram_cache[ngram].append(continuation)
Implementation Framework
The previous sections presented algorithms in isolation, but production systems need a single entry point that can switch strategies based on workload requirements. The pipeline class below wraps all of the decoders behind a common interface and selects an implementation at construction time from a configuration object. It also owns the tokenizer and the generation loop, handling the details of appending accepted tokens, detecting end-of-sequence, and enforcing the maximum output length.
Wrapping everything behind one interface is a deliberate design decision. It lets operators benchmark different strategies without touching application code, A/B test draft models in staging, and roll back a failed algorithm change with a single configuration flag. The benchmark helper is equally important for adoption: by measuring tokens per second, speedup relative to a baseline, and the observed acceptance rate, teams can verify that theoretical speedups translate into real-world gains before committing to a deployment.
class SpeculativeGenerationPipeline:
"""
Complete speculative decoding pipeline.
"""
def __init__(self, config):
# Load models
self.target = self._load_model(config.target_model)
self.draft = self._load_model(config.draft_model)
# Configuration
self.max_draft = config.max_draft
self.temperature = config.temperature
# Choose algorithm
if config.algorithm == 'standard':
self.decoder = SpeculativeDecoder(self.target, self.draft)
elif config.algorithm == 'sampling':
self.decoder = StochasticSpeculativeDecoder(
self.target, self.draft, config.temperature
)
elif config.algorithm == 'self':
self.decoder = SelfSpeculativeDecoder(self.target)
else:
self.decoder = SpeculativeDecoder(self.target, self.draft)
def generate(self, prompt, max_tokens=100):
"""Generate with speculative decoding."""
input_ids = self.tokenizer.encode(prompt)
output_tokens = []
while len(output_tokens) < max_tokens:
# Get next tokens
new_tokens, accepted_all = self.decoder.generate_next_token(input_ids)
if not new_tokens:
break
output_tokens.extend(new_tokens)
input_ids = input_ids + new_tokens
# Check for EOS
if new_tokens[-1] == self.tokenizer.eos_token:
break
return self.tokenizer.decode(output_tokens)
def benchmark(self, prompts, baseline_time):
"""Benchmark speedup."""
speculative_time = self._time_generation(prompts)
return {
'speedup': baseline_time / speculative_time,
'tokens_per_second': total_tokens / speculative_time,
'acceptance_rate': self._compute_acceptance_rate()
}
Optimizations and Tricks
1. Adaptive Draft Length
A fixed draft length leaves performance on the table. On easy text the draft model could safely propose more tokens per round, increasing the throughput gain, while on hard text a long draft wastes compute verifying tokens that will almost certainly be rejected. Adaptive draft length makes the draft size itself a learned hyperparameter, adjusted continuously from the observed acceptance history.
The decoder below tracks the ratio of accepted tokens to drafted tokens for each iteration and maintains a sliding window of recent acceptance values. When the trailing average is consistently high, the draft length grows (capped at a safe maximum), and when it drops, the length shrinks to reduce wasted work. Because acceptance statistics shift as the conversation topic changes, this feedback loop keeps the system near its optimal operating point without any offline tuning. The main risk is oscillation at boundaries, which is why the window and the growth and shrink steps are deliberately conservative.
class AdaptiveSpeculativeDecoder:
"""
Adjust draft length based on acceptance rate.
"""
def __init__(self, target, draft):
self.decoder = SpeculativeDecoder(target, draft)
self.acceptance_history = []
self.current_draft_len = 6
def generate(self, input_ids):
"""Generate with adaptive draft length."""
# Use current draft length
self.decoder.max_draft = self.current_draft_len
# Generate
tokens, fully_accepted = self.decoder.generate_next_token(input_ids)
# Track acceptance
acceptance = len(tokens) / self.current_draft_len
self.acceptance_history.append(acceptance)
# Adapt draft length
if len(self.acceptance_history) > 10:
avg_acceptance = sum(self.acceptance_history[-10:]) / 10
if avg_acceptance > 0.9:
self.current_draft_len = min(12, self.current_draft_len + 1)
elif avg_acceptance < 0.6:
self.current_draft_len = max(3, self.current_draft_len - 1)
return tokens
2. Batch Speculation
So far we have assumed a single sequence in flight, but real servers handle many concurrent requests. Naively running the single-sequence algorithm per request multiplies the memory-bandwidth cost, because each sequence streams the full model independently. Batch speculation exploits the fact that the target model can process many sequences in a single forward pass: draft tokens are proposed for every active sequence, then all of them are verified in one batched pass through the target.
The class below shows the shape of this optimization. Drafting happens per sequence using the fast model, while verification is amortized across the whole batch so the expensive target pass is reused by every request. Sequences that finish—either by exhausting drafts or hitting end-of-sequence—are removed from the active set, and the loop continues until every prompt has completed. In practice this is where speculative decoding produces its largest absolute gains on busy services, because GPU utilization was already high and speculation adds throughput without requiring more hardware.
class BatchSpeculativeDecoder:
"""
Process multiple sequences with speculation in parallel.
"""
def __init__(self, target, draft):
self.target = target
self.draft = draft
def generate_batch(self, prompts, max_tokens):
"""
Generate multiple sequences simultaneously.
"""
# Encode all prompts
input_ids_list = [self.tokenizer.encode(p) for p in prompts]
results = [[] for _ in prompts]
finished = [False] * len(prompts)
while not all(finished) and len(results[0]) < max_tokens:
# Draft for all sequences
drafts = []
for ids in input_ids_list:
draft_tokens = self._quick_draft(ids)
drafts.append(draft_tokens)
# Verify all in batch
for i, (ids, draft) in enumerate(zip(input_ids_list, drafts)):
if finished[i]:
continue
accepted = self._verify(ids, draft)
results[i].extend(accepted)
input_ids_list[i].extend(accepted)
if not accepted or draft[-1] == self.tokenizer.eos_token:
finished[i] = True
return [self.tokenizer.decode(r) for r in results]
Performance Analysis
Expected Speedups
| Configuration | Draft Model | Acceptance Rate | Speedup |
|---|---|---|---|
| 70B → 7B | 7B | 85-95% | 2.5-3.0x |
| 70B → 3B | 3B | 70-85% | 2.0-2.5x |
| Self-speculate | Same | 75-90% | 1.8-2.2x |
| Hierarchical | Multi-level | 90-95% | 2.5-3.5x |
The speedup table above gives empirical ranges, but understanding why the numbers land where they do requires a small analytical model. The key quantity is the acceptance rate, which determines the average number of tokens accepted per target forward pass. If the draft model accepts a fraction p of tokens, the expected run length per iteration is roughly 1/(1-p): a 90% acceptance rate yields about ten tokens per iteration, while 70% yields only about three.
The function below formalizes this trade-off. Each iteration costs the draft model’s time multiplied by the average run length plus one full target forward pass, and the speedup is computed by comparing the resulting token throughput against a purely autoregressive baseline. Notice the implicit constraint: the draft model must be fast enough that its cumulative cost over the run length is still less than the target’s single pass. This is why a 10x-smaller draft is a common heuristic—it makes the drafting overhead negligible while keeping acceptance rates high on most content.
def analyze_speedup(target_time_per_token, draft_time_per_token, acceptance_rate):
"""
Compute expected speedup.
Traditional: target_time_per_token per token
Speculative: draft_time_per_token * n + target_time_per_token / n
where n = 1/(1-acceptance_rate)
"""
# Average tokens per iteration
avg_tokens = 1 / (1 - acceptance_rate)
# Time per iteration
speculative_time = (draft_time_per_token * avg_tokens +
target_time_per_token)
# Tokens per time
speculative_rate = avg_tokens / speculative_time
traditional_rate = 1 / target_time_per_token
return traditional_rate / speculative_rate
Best Practices
1. Draft Model Selection
Model selection is the single most impactful tuning decision in a speculative decoding deployment, and it is mostly a question of measuring two competing quantities. A larger draft model produces longer accepted runs and fewer wasted target passes, but it consumes more time and memory per draft step; a smaller draft model is nearly free to run but accepts poorly on complex or unusual text, so the target often ends up doing the work anyway.
The heuristic below encodes the standard starting point: scale the draft roughly with the target so that the draft cost stays below a few percent of the target’s per-token cost, while keeping the acceptance rate high enough to sustain a meaningful run length. For frontier-scale models this usually lands at a 7B draft for a 70B target, and a 3B draft for a 10B target. These starting points should be validated empirically on a representative workload rather than trusted blindly, because acceptance rates vary dramatically with the domain—code, math, and conversational text all behave differently.
def select_draft_model(target_model_size):
"""
Select appropriate draft model based on target.
"""
# Rule of thumb: 10x smaller is usually good
if target_model_size >= 70_000_000_000:
return 7_000_000_000 # 7B draft for 70B target
elif target_model_size >= 10_000_000_000:
return 3_000_000_000 # 3B draft for 10B target
else:
return target_model_size // 4 # ~4x smaller
2. Handling Rejection
Rejection is not a failure case to be avoided but a routine event in the normal operation of speculative decoding. When the draft model diverges from the target, the algorithm must decide what token to emit at the divergence point, because that token becomes the foundation for the next iteration. The choice affects both throughput and output quality, and different production systems prefer different strategies.
The class below lays out the recovery strategies in order of increasing sophistication. The simplest approach takes the target’s greedy token at the rejection position, which maximizes determinism. A sampling-based alternative injects controlled diversity for creative applications. The most advanced option, beam-search continuation, maintains multiple candidate continuations and only commits once more evidence arrives—at the cost of significant extra computation. A robust implementation typically selects among these based on the request’s quality and latency requirements.
class RobustSpeculativeDecoder:
"""
Handle rejection gracefully with multiple strategies.
"""
def handle_rejection(self, input_ids, draft_tokens, target_logits):
"""
When draft diverges from target, recover gracefully.
"""
# Strategy 1: Use target's token
target_token = target_logits[len(input_ids)].argmax().item()
# Strategy 2: Temperature sampling for diversity
if self.use_sampling:
probs = F.softmax(target_logits[len(input_ids)] / self.temp)
target_token = torch.multinomial(probs, 1).item()
# Strategy 3: Beam search continuation
# (more complex - generates multiple options)
return target_token
Future Directions in 2026
Emerging Innovations
- Self-Speculation: Using model itself as draft (no separate model)
- Multi-Query Attention: Batch speculation across multiple requests
- Hardware-Software Co-design: Specialized pipelines for GPUs
- Diffusion Speculation: Extending to non-autoregressive models
Resources
Conclusion
Speculative decoding represents a breakthrough in LLM inference efficiency. By leveraging the insight that most tokens are predictable, we can achieve 2-3x speedups without any quality loss.
The key is choosing the right approach: standard greedy for maximum speed, stochastic for creative generation, hierarchical for varied quality requirements, or self-speculation when memory is constrained.
As LLM deployment scales, speculative decoding will become standard practice. The technique is lossless, requires no model retraining, and provides immediate performance benefits. It’s one of the most practical optimization techniques in the modern AI toolkit.
Comments