Overview: KV Cache

Cooper Okay, I’m just saying it now because we keep doing this dance: KV cache is in, like, half the things we talk about lately. We should probably stop pretending it’s background noise and actually do the whole thing properly.

Miles Yeah. It’s one of those terms that sounds like a tiny implementation detail, and then you look closer and realize it’s sitting under the whole inference stack. The annoying part is that it’s actually simple once you stop letting the acronym bully you.

Cooper That is such an Exploring Next move, honestly. We keep circling the same little piece of machinery and acting surprised that it matters.

Miles I know. And I think the cleanest way to think about it is a notebook. The model is writing down a sentence one word at a time, and KV cache is the part where it keeps the useful notes instead of rereading the whole page every time.

Cooper Mm-hm.

Miles So when it’s generating text, it doesn’t just magically know the next token. It has to look back at what came before, and in a transformer, that looking-back is done through attention. If you want the deep version of attention, we did episode six oh seven, but the quick version is: attention is the mechanism that lets one token look at earlier tokens and decide what matters.

Cooper Right, and the other prerequisite here is autoregressive generation, which we did in episode six thirty. That just means the model makes one token, then uses that token plus the history to make the next one, then repeats.

Miles Exactly. And transformers are the model family that made this practical at scale. Quick gloss: a transformer is a neural network built around attention blocks, so instead of processing a whole sequence like a fixed blob, it can compare tokens to each other and decide what to use from the past.

Cooper So the thing that’s burning work is that repeated looking back. Every new token says, okay, what do I need from all the old tokens? And without cache, it’s like you keep re-asking the same room to reintroduce itself.

Miles That’s the right annoyance. The model computes attention relationships using three vectors: Query, Key, and Value. Query is the thing asking, Key is the label on the thing being looked up, and Value is the content you actually pull back. You don’t need the full math yet to get the idea.

Cooper Wait, but why those three? Why not just store the whole hidden state and call it a day?

Miles Because attention doesn’t use the whole hidden state in the same way. It uses those transformed versions of the token representation. The key and value are the pieces that matter for later lookup, so if they don’t change, there’s no reason to recompute them over and over.

Cooper Okay, that helps. So the cache is basically the model’s memory of the old keys and values.

Miles Yes. More precisely, as each token gets processed, the model computes fresh Key and Value vectors for that token once, stores them, and then reuses them for later tokens. The new token still gets a fresh Query, because it’s the thing doing the asking right now.

Cooper Right.

Miles And that’s the entire win. During generation, the old Keys and Values are unchanged, because those earlier tokens are already fixed. So if you cache them, the model only does the new work once instead of redoing the same work for every next token.

Cooper So the model is not remembering the words in some mystical human sense. It’s remembering the expensive intermediate stuff that would otherwise get recalculated.

Miles Exactly. I’d put it this way: the cache stores what the model has already understood well enough to reuse. It’s not storing the final answer. It’s storing the scratch work that turned the past into something usable.

Cooper That’s a much better way to say it than my notebook thing, which was flirting with being cute in a bad way.

Miles No, the notebook is fine. It’s just the notebook with the actually important pages marked.

Cooper Okay, so where does the speedup actually come from? Because if you’re still doing attention, I can already hear my own skepticism getting annoying.

Miles No, that’s exactly the right question. Without a KV cache, every new token would force the model to recompute the Keys and Values for all previous tokens again, even though those old tokens haven’t changed. With the cache, it only computes the new token’s pieces and appends them. That cuts a huge amount of repeated work during decoding.

Cooper Decoding meaning generation, right?

Miles Yeah. In this context, decoding just means the phase where the model is producing tokens one by one. Training is different. During training, the model usually sees full sequences at once, so KV cache is not the thing you care about there.

Cooper Mm-hm.

Miles And the reason this matters so much is latency. If you’re waiting for the next token in a chat or a code completion, the user feels every little delay. KV cache is one of the reasons those systems can feel interactive instead of glacial.

Cooper That’s the product story I can actually feel. Like, if the answer shows up in time, nobody cares that the backend had a little computational epiphany. They just care that it didn’t sit there thinking for an hour.

Miles Right, and the rough claim from the seed material is that it can cut per-token compute by something like seventy-five to ninety percent, depending on sequence length. I’d treat that as a useful ballpark, not a sacred number, but it’s enough to explain why this is such a big deal.

Cooper Sure.

Miles Now, the catch. The cache is not free. It takes memory, and it takes more memory as the sequence gets longer. So you’re trading compute for memory, which is why people obsess over how to store it efficiently.

Cooper Okay, that’s the part I wanted to get to. Because every time somebody says, ‘we made inference faster,’ there’s usually a second sentence hiding under the table that says, ‘by spending a bunch of some other scarce thing.’

Miles Exactly. This one is very honest about the bill. If the conversation gets longer, the cache gets bigger. More tokens means more stored Keys and Values, so memory use grows linearly with sequence length.

Cooper So a short prompt is cheap, and then as the chat goes on, the working set keeps ballooning.

Miles Yep. And that’s why long-context serving gets interesting. You can be totally fine at a few thousand tokens and then start sweating when you push much further, because the cache is now a major resident object in memory.

Cooper Okay, can we put a little shape on the mechanics? Like, what happens on token number two versus token number two hundred?

Miles Sure. On the first token, there’s almost nothing to reuse, so the cache is tiny. On each later token, the model computes that token’s Query, Key, and Value, uses the existing cache to attend over the past, and then appends the new Key and Value to the stored history. By token two hundred, the cache has two hundred tokens’ worth of those stored vectors, so the memory footprint is much larger.

Cooper I see.

Miles And because the cache grows with sequence length, the system has to manage that growth carefully. If you’re serving lots of users at once, every active conversation is carrying around its own cache state. That turns into a scheduling and memory-management problem pretty fast.

Cooper Which is why all the inference-engine people sound like they live inside a spreadsheet sometimes.

Miles That’s not wrong. They’re basically trying to make the notebook fit in the backpack without ripping the straps off.

Cooper Okay, that’s actually the nicest version of this I’ve heard.

Miles Let’s ground it in concrete systems, because otherwise this stays abstract. We looked at OpenAI’s GPT-five point six family, where the Responses API is built for programmatic tool calling and fast interactive use. KV cache is part of what makes that kind of back-and-forth feel immediate instead of recomputing the whole conversation every step.

Cooper Right, and that’s the user story. If you’re building something where the model is supposed to answer, then call a tool, then keep going, you really feel the cost of every extra token if the runtime is sloppy.

Miles Exactly. Another one we talked about was the ZML inference product. The whole pitch there was speeding inference across lots of AI chips, and again, cache handling is one of those invisible pieces that decides whether the speed claim is real or just a demo artifact.

Cooper Mm-hm.

Miles And then there was the local models for coding episode. That one mattered because coding assistants are a perfect KV cache use case. You’re generating line by line, the context can get long, and every little latency win is visible in the editor. If the cache management is bad, the whole experience feels sticky.

Cooper That’s the thing I keep coming back to. KV cache is not glamorous, but it’s exactly the kind of boring plumbing that decides whether a product ships cleanly or feels like a science project.

Miles Yep. And the Nemotron TwoTower thirty B model is another nice example from the stack side. A model like that, especially when you’re trying to serve it efficiently, lives or dies on how the runtime handles these stored attention states. The architecture can be good, but if the cache is wasteful, the serving story gets ugly fast.

Miles That’s annoyingly good. Yeah, that’s it.

Cooper Alright, I can’t believe this is how we spent a Wednesday, but at least the notebook has the right pages now.