How to Handle Small Context Window Limits in RAG Systems
Justy and Cody dig into a hands-on technique for making RAG work when your context window is tiny: route with summaries, answer with raw chunks, and keep an explicit budget. Cody questions whether the toy demo obscured the real complexity, and Justy sizes up who this actually saves.
Transcript
Justy Okay—so we keep pretending the context window just needs to be BIGGER to fix RAG?
Cody Yeah, that’s the knee-jerk: ‘throw VRAM at it.’
Justy Exactly. Like running unoptimized graphics on a four-hundred-watt GPU and calling it a win.
Cody Right— … and the GPU melts anyway once the scene gets complex.
Justy So instead we’re supposed to route with summaries, answer with raw chunks, and babysit a context budget—is this gonna take more than a few hours to grok?
Cody You say that like this is novel—trying to fit the whole kitchen sink into the prompt and praying it fits has been the default since day one.
Justy Fair, but the difference is you don’t realize the retriever ‘found’ the golden chunk because the prompt builder silently dropped it, leaving you with ‘I don’t know.’ … Which is exactly what happens on a 12-gig VRAM laptop.
Cody Okay — that’s the moment the abstraction leaks.
Justy So the article’s real audience is people shipping on constrained hardware, not the folks with eight A100s in the cloud.
Cody Not exclusive—the trick helps anywhere the cost curve matters, but yeah, local devs and memory-constrained pipelines feel it first.
Justy And the best part? A three-level trace that tells you which doc, which chunk, which slice of raw text actually made it into the prompt—or got evicted by the budget.
Cody If you treat that trace as a first-class feature, not a footnote.
Justy So the summaries aren’t evidence—they’re signposts that say ‘look over here.’ Then you fetch the raw paragraphs and answer from those.
Cody Which means you’ve moved the complexity from ‘every chunk in the prompt’ to ‘every chunk summarised, indexed, and routed.’
Justy And you still have to replace the toy summarizer and tokenizer with your own stack— … but the pattern travels.
Cody Yeah, no— … the repo is cute, but the ideas outlive the examples.
Justy So who should actually care? Local-model devs, edge pipelines, anyone whose compute budget is a hard stop.
Cody Everyone whose prompt bill is proportional to tokens.
Justy Fair. This isn’t a paradigm shift— … it’s a practical retrofit that turns a mysterious ‘I don’t know’ into a debuggable bottleneck.
Cody And if you’re already neck-deep in LangGraph or LlamaIndex you’ll probably roll your own routers instead of bolting on summaries.
Justy Which is fine—the pattern’s the portable bit, not the library.
Cody I can get behind that.
Justy Cool. Let’s log the budget, not the GPU.