Speculative Decoding: Making LLMs 3x Faster with Zero Quality Loss

If you have ever run a local model like Llama 3 or Mistral, or watched ChatGPT think, you have felt the latency: you send a prompt and the text arrives one word at a time.

That is not a flaw in the implementation. It is the architecture. Modern LLMs are autoregressive: to generate a sequence of K tokens, the model runs its entire network K times, in series. It cannot begin word five until the arithmetic for word four has finished. It is the slowest possible way to produce a sequence, and every transformer does it.

Here is where it gets frustrating. Measure a GPU running a large transformer and you find the compute cores sitting idle for most of the time.

The reason is that loading the model's weights out of high-bandwidth memory into the cores takes longer than the arithmetic those cores then perform. The memory bus is the traffic jam; the engine is revving, waiting for the road to clear. In the standard terminology, single-stream inference from a large model is memory bandwidth bound, not compute bound.

Yaniv Leviathan, Matan Kalman and Yossi Matias at Google Research started from an observation that is obvious once stated: not all inference steps are alike. Some genuinely need a very large model. Others are approximated perfectly well by a small one.

• Easy. The prompt is "The sky is". The next token is almost certainly "blue". A child could guess it; so could a model a hundred times smaller.
• Hard. The prompt is "The chef added a pinch of". It could be salt, sugar, paprika, cumin. This one needs the context depth the big model has.

So why spend an eleven-billion-parameter forward pass on "blue"? Why not let something small and fast do the guessing, and keep the large model for what it is uniquely good at judging whether the guess was right?

Two models do the work, and it helps to name them.

The Intern proposes. The Professor disposes. Everything else is the machinery that makes that exchange lossless.

Whether this pays off comes down to two quantities.

Every generation step runs the same loop.

This is the part that makes the technique a breakthrough rather than a trick. Usually, speeding up a model means trading away accuracy. Quantisation and distillation both give you outputs that are similar, not identical. Speculative decoding gives you the same distribution, exactly.

The mechanism is rejection sampling, and the intuition is short. You want to sample from p, the Professor. You sample from q, the Intern, instead.

• Where q ≤ p. The Intern was cautious. Accept always. You have captured exactly q(x) of the mass, the part of p that q covers.
• Where q > p. The Intern was overconfident. Accept with probability p/q, which keeps exactly p(x). On the rejection, sample from the residual max(0, p − q).

Sum the accepted-via-q mass and the rejected-via-residual mass and the q terms cancel. The algebra lands on p(x).

The paper's description translates almost directly into code. This assumes Mp and Mq are callables returning probability distributions over the vocabulary.

import numpy as np

def speculative_decoding_step(Mp, Mq, prefix, gamma):
    # 1. Sample gamma guesses x1..x_gamma from Mq, autoregressively.
    x, q = [], []
    for _ in range(gamma):
        qi = Mq(prefix + x)
        xi = np.random.choice(len(qi), p=qi)
        q.append(qi)
        x.append(xi)

    # 2. Run Mp on every position. One batched call in production.
    p = [Mp(prefix + x[:i]) for i in range(gamma + 1)]

    # 3. Walk the guesses in order and find n, the number accepted.
    n = gamma
    for i in range(gamma):
        r = np.random.uniform(0, 1)
        if r > p[i][x[i]] / q[i][x[i]]:
            n = i
            break

    # 4. Corrected token, or a free bonus one if nothing was rejected.
    if n == gamma:
        p_prime = p[n]
    else:
        p_prime = np.maximum(p[n] - q[n], 0)
        p_prime = p_prime / p_prime.sum()

    # 5. n accepted draft tokens, plus exactly one token from Mp.
    t = np.random.choice(len(p_prime), p=p_prime)
    return prefix + x[:n] + [t]

The full annotated version is in this post's codes/python/ directory.

What each block is doing

• The draft loop. Both the tokens and their distributions are kept, because q is needed again at verification time and recomputing it would throw away the saving.
• The comprehension over Mp. Written serially for readability, but this is the one line that must become a single batched forward pass in a real implementation. Run as written, it is slower than plain decoding.
• The dice roll. r > p/q rejects. When q ≤ p the ratio is at least one, r is drawn from [0, 1), and acceptance is automatic: the two branches of the maths collapse into one line of code.
• The residual. max(0, p − q) strips out what the Intern already claimed, and renormalising makes it a distribution again.

The paper tests T5-XXL (11B) as the Professor against three Interns (T5-Small at 77M, about 100× smaller; T5-Base at 250M, 40×; and T5-Large at 800M, 13×) on English-to-German translation and on CNN/DailyMail summarisation.

The dumbest Intern wins

T5-Small is the least accurate draft model and the fastest overall. T5-Large is measurably smarter, with α of 0.82 against 0.75, and finishes well behind it, because the extra accuracy costs more in c than it returns in α. Speed beats cleverness here, and by a wide margin.

Greedy beats sampling

Temperature zero always outruns stochastic sampling. Forced onto the single most likely token, the two models agree far more often. Add randomness and the Intern starts picking creative tokens the Professor will not endorse, and every one of those is a rejection.

Translation beats summarisation

English to German is close to a deterministic mapping, so a small model tracks it well. Summarisation requires deciding what matters in an article, which is exactly the judgment a 77M-parameter model does not have.

What if you have no small transformer at all, just the weights of the giant one? The authors tried an Intern that is not a neural network in any sense: a bigram model. A lookup table that takes the previous token and returns whatever most often followed it in the training data. No grammar, no syntax, no meaning. Counting.

It guessed right twenty percent of the time. And because a table lookup is free, with c effectively zero, that α = 0.2 still bought a 1.25× speedup.

Reach for it when

• Memory bandwidth is the bottleneck. True of nearly any large model at batch size one on a single GPU, the case where the compute cores are idle and you have capacity to spend.
• A smaller sibling exists. T5, GPT, LaMDA and Llama all ship distilled or smaller variants. That is your Intern, already trained.
• The sequences are long. More tokens means more chances to accept a draft and amortise the setup.

Leave it alone when

• Compute is the bottleneck. Serving a hundred concurrent users on one GPU already saturates the cores. There is no idle capacity left to spend, and the extra parallel runs will cost you throughput rather than buy it.
• The Intern is not cheap enough. A draft model only twice as fast as the target puts c at 0.5, and α rarely clears that.
• You need bit-exact reproducibility. The distribution is identical; the specific path through it is not.

Be clear about what is being spent. Standard decoding runs M_p once per token. Speculative decoding runs it γ + 1 times per iteration. That is strictly more total arithmetic, and when the Intern guesses badly, most of it is discarded.

It still wins, because the arithmetic was never what you were waiting for. The weights had to be read out of memory regardless; the extra multiplications ride along in cycles that were going to be idle. You are not buying speed with accuracy. You are buying it with work the hardware was going to waste anyway.

This is not a research curiosity waiting on productionisation.

• vLLM and ExLlamaV2 both ship variants of speculative decoding already.
• Medusa takes a different route, attaching fine-tuned draft heads to the main model rather than running a separate one.
• DeepMind (Chen et al., 2023) arrived at the same technique independently and confirmed 2–2.5× on Chinchilla 70B shortly after the original paper.

If you run models locally, look for the speculative flags in your inference engine. If you are building a serving system, a small draft model on the side is a low-cost addition for a large return.

What makes this elegant is where it aims. The obvious target is the serial nature of autoregressive generation, and the obvious attack is to make the model smaller or cheaper, which costs quality. Leviathan and colleagues went after the observation underneath it instead: the bottleneck is not the arithmetic, it is the memory. Spend the idle compute on parallel verification and let a tiny model guess ahead.

And they did it with a proof that the output distribution is untouched. No retraining, no new models, no change to what comes out. Where memory bandwidth is the limit and compute is going spare, it is close to a free default.

We are moving into a period where the interesting question is not how smart a model is, but how efficiently it can be run. This is a large step in that direction.

1. Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. Proceedings of the 40th International Conference on Machine Learning.
2. Chen, C., Borgeaud, S., Irving, G., Lespiau, J., Sifre, L., & Jumper, J. (2023). Accelerating Large Language Model Decoding with Speculative Sampling. arXiv.
3. Raffel, C., et al. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR.