Exploring Next Overview: Speculative Decoding

Justy Okay, we have to do this one properly. Speculative decoding keeps popping up in our conversations like it assumes everybody already knows the trick, and that is very rude of it.

Cody Yeah, and it sounds way fancier than it is. It's basically a fast helper model doing the rough draft while the big model does the actual checking.

Justy Which, honestly, is such an Exploring Next thing. We keep saying it like it's obvious and then act surprised when nobody in the room is nodding.

Cody Right. And I think the reason it matters is that it feels like cheating until you see the asymmetry.

Justy Mm-hm.

Cody A big language model writes one token at a time. Token means the little chunk of text the model picks next, and autoregressive generation means each new token depends on the ones before it. We did a whole episode on that, episode six hundred thirty, but the short version is: the model can't just spray out the whole answer in one shot.

Justy Okay, so the model is like a painfully careful editor who has to type every word themselves.

Cody Exactly. Speculative decoding says, 'what if a smaller, faster writer drafts a few words, and the big editor only checks them?' The helper is the draft model. The big one is the target model. And decoding strategy just means the rule you use for choosing the next token or tokens while you're generating.

Justy Right, so this isn't some mystical new kind of model. It's a different way of doing the same generation loop.

Cody Yeah. And the key trick is that writing is sequential, but checking is cheaper and can be done in parallel. The target model can look at a proposed batch of tokens in one forward pass and decide where it agrees, instead of laboring through each one from scratch.

Justy Hold up. Forward pass is one run through the network, right?

Cody Yeah. One sweep through the model to get its predictions. And because decode is autoregressive, the model normally has to do that sweep again for every token. Speculative decoding tries to get multiple accepted tokens out of one target-model pass.

Justy That is the part that sounds illegal on a podcast.

Cody I know. But the reason it works is the draft is only guessing. The target model still gets the final say on every accepted token, so the output stays identical to what the target would have produced on its own.

Justy Okay, but if the target is still doing the real work, where's the speedup actually coming from?

Cody From the fact that verification is cheaper than generation. The target model is expensive when it has to invent the next token. It's much less expensive when it only has to score a candidate sequence and confirm whether it would have made the same choices.

Justy Mm-hm.

Cody We should probably gloss conditional probability here too, since it's doing a lot of hidden work. We did a whole episode on that, episode six hundred seventy-one. Plain version: the model is constantly asking, 'given everything so far, what's the probability of each next token?' Speculative decoding just changes who proposes the candidate and who checks that probability.

Justy So the little model is basically saying, 'I think the next few words are probably this,' and the big model is saying, 'sure, up to the point where I'd have said something else.'

Cody Yes. And the classic version goes like this: the draft model proposes a short run of tokens, often four to eight. Then the target model runs once over that whole proposal. It accepts the longest prefix it agrees with, and the first disagreement becomes the point where the target supplies the next token itself.

Justy So if the draft gets the first five right and the sixth wrong, the first five survive and the sixth gets replaced.

Cody Exactly. And everything after that wrong token is tossed, because once the sequence diverges, the rest of the draft was based on the wrong path.

Justy Mm. That's the part I kept mentally hand-waving, and then it turns out the hand-wave was the whole point.

Cody Yeah, the whole thing lives or dies on acceptance rate. If the draft's guesses survive often, you leap forward several tokens for about the cost of one target pass. If they don't, you've just added overhead.

Justy So the draft model has to be close enough to the target to make good guesses, but not so huge that it eats the savings.

Cody Then there's n-gram speculation and lookahead decoding. N-gram speculation doesn't even use a neural draft model. It looks at sequences already in the text, builds a little dictionary of observed token runs, and reuses those as guesses during decode. Lookahead decoding is the version that can construct those patterns during inference rather than relying as much on the repeated input.

Justy So it's basically pattern reuse instead of prediction from another model.

Cody Exactly. And that makes it weirdly strong for code completion and code revision, where the output often mirrors the input structure. In that lane, the repeated syntax gives you long speculative runs for very little cost.

Justy That's the part I love. The thing that sounds most hacky sometimes ends up being the best fit for the most mechanical workloads.

Cody Yeah. And the examples that matter here are the ones people actually ship with: vLLM, SGLang, and TensorRT-LLM all support flavors of this stuff in production serving stacks. The details differ, but the idea is common enough now that it's not some exotic lab trick.

Justy So this is still live, not some old clever paper everyone admired once and then forgot?

Cody No, it's very live. In 2026 it's basically one of the standard levers for making single-stream LLM serving faster and cheaper. The exact algorithm shifts, but the core pattern is still in use.

Justy Okay, good. Because I was about to get melodramatic about it and say it got absorbed by the void.

Cody Not yet. The field has definitely moved toward better speculators and better integration, but the basic draft-and-verify loop is still doing work.

Justy And that's where the product story comes in. If you're building on top of a provider API, you just feel it as 'the model got faster this quarter.' You don't care whether the under-the-hood reason was better batching, better caching, or speculative decoding, as long as the bill and the wait both shrink.

Cody Right, but the engineering story is where the catch lives. Under heavy batching, the GPU time spent verifying speculation for one request is GPU time not serving another request. So a technique that helps a single user can hurt fleet throughput if you crank it at the wrong time.

Justy That is the part that sounds unromantic but real.

Cody It's the whole production tradeoff. Single-request latency and server-wide throughput are different objectives. Speculation is often great when the system has spare compute, and less attractive when the server is already busy.

Justy So the stack has to decide when to use it.

Cody Yeah. Dynamic tuning matters. Some systems raise or lower speculation based on load, because the best move for one request is not always the best move for the whole fleet.

Justy And the draft model itself is another moving part. If it gets stale or doesn't match the target's style, acceptance falls off and the whole savings story gets wobbly.

Cody Exactly. That's one reason the draftless variants exist. They reduce the overhead of carrying a separate draft model around, or they train the speculator more directly against the target's own behavior.

Justy Okay, let me try the whole thing back in one sentence, and tell me if I mangle it. The junior model guesses ahead, the senior model checks in one sweep, and if the guesses are good enough you get the senior's exact answer with fewer expensive passes.

Cody That's it. And the reason it works at all is the asymmetry between generating and verifying.

Justy Mm-hm.

Cody Writing the answer is sequential and costly. Checking a candidate sequence is cheaper and parallelizable. Speculative decoding exploits that gap, and everything else is just different ways of making the gap bigger or the guesses better.

Justy And the thing I want to keep in my head is just that picture of the fast drafter and the exact checker. Because now when somebody says 'they're serving it with spec-dec,' I won't just nod like an impostor.

Cody We have been doing this since November, and somehow that still counts as progress.

Justy Yeah, shocking. Come on, Cody, let's not act like the show has finally become responsible. That would ruin the bit.