12 Ways to Reduce LLM Latency and Inference Costs in Production KDnuggets
A practical KDnuggets piece argues that most LLM production latency/cost gains come from cutting unnecessary work instead of bigger models or more GPUs. They list 12 levers: measure the right metrics, cut output tokens, route to smaller models, collapse LLM calls, prefix caching, add multiple cache layers, control RAG context, batch offline work, tune batching for user latency, and manage KV cache. Tyler pushes back on the article’s overgeneralization of cache reuse across all tasks, the thin technical depth behind some tips, and the implication that routing to small models never backfires. Pippa highlights the piece’s strongest point—measuring TTFT, P95/P99, and queue time—because that’s where teams most often mis-diagnose bottlenecks. They land on: the article’s monitoring advice and batch-tuning guidance are solid; several recommendations work only for read-heavy workloads; and routing to tiny models is risky until you have cheap, high-confidence evaluators. They wrap with a Build Next command to try vLLM continuous batching and two open-source RAG-caching projects (Harmonia and From Prefix Cache to Fusion RAG Cache).
Transcript
Pippa Ugh, another article telling you to just buy more GPUs. I mean, come on, the title alone is a joke.
Tyler Yeah, and the headline claim is still 'scaling LLMs isn't about more GPUs, it's about removing wasted work.' That’s the part I’m skeptical about.
Pippa Okay, listen—it’s not wrong. The actual insight is measurement.
Tyler You say that now, but they list twelve ways and half of them sound like cargo-cult ops mantras.
Pippa Alright, but they start with the right metrics—TTFT, p-ninety-fives, p-ninety-nines, queue time, cache hit rate. That’s where teams always trip up—misdiagnosing the bottleneck.
Tyler Sure, that part’s solid. Queue time and TTFT are real pain points. But once they leave measurement… they start recommending 'route everything to the smallest model' like it’s a magic wand.
Pippa Tyler, come on, you know teams already do that when it’s safe—sentiment, extraction, FAQ fallbacks. The trick is cheap confidence estimators.
Tyler That ‘trick’ is a nontrivial research project. The article just says 'tune a classifier' and moves on.
Pippa Okay, fair. But the prefix-caching section is legit—put your stable context at the front, let the cache do the work.
Tyler Legit until your workload isn’t read-heavy. Cache hit rates crater if half your prompts are interactive agent loops where the user’s asking for the same thing phrased differently.
Pippa That’s a workload caveat, though—not a strike against prefix caching itself.
Tyler Fine. Cache layers section then—exact response cache, semantic cache, retrieval cache, tool result cache. Four kinds of cache, all needing tenant isolation, versioning, and similarity thresholds? That’s a distributed-systems tax most teams haven’t budgeted for.
Pippa You’re not wrong about the tax. But the article’s point stands: don’t reprocess what you already know.
Tyler Exactly. And that’s why the quality section is thin group therapy. They tell you to ask for shorter outputs and stop sequences without quantifying when brevity hurts accuracy. 'Do not pay for tokens the user won’t read'—great advice for a product owner, terrible validation criteria for an engineer.
Pippa Tyler, you’re being unfair—they’re speaking to product people.
Tyler Then it shouldn’t double as a production guide. The RAG section repeats the same mistake: 'control your context budget' without real guidance on how to triage relevance or deduplicate without losing signal.
Pippa Alright, so what would you salvage?
Tyler Three things the article nails. One: measure TTFT, P95/P99, queue time, cache hit rate—actually look at the levers you can pull. Two: tune batching against user latency SLOs, not raw throughput—continuous batching in vLLM is the best example they didn’t cite.
Pippa Three: collapse superfluous LLM calls. A single structured prompt replacing three sequential steps is pure latency win. That part’s golden.
Tyler Everything else is 'run offshore' advice. Routing to tiny models? Only after you’ve paid the evaluation tax. Cache layers? Only if your workload fits. Batch throughput? Only if it doesn’t blow TTFT.
Pippa So the article’s strongest section is the measurement table and the batch-tuning section—rest is 'apply judiciously.'
Tyler Judiciously, yeah, which is exactly what the article doesn’t say enough.
Pippa You just don’t like any advice that isn’t seven pages of equations.
Tyler I like advice that tells me the failure modes up front. This one buries them.
Pippa Okay, fine. You want to put a number on it?
Tyler Seventy-thirty the article’s blanket recommendations either backfire or don’t generalize—outside the three wins I named.
Pippa And I’ll say forty-sixty those three wins move the needle for teams already running agents. The rest is noise.
Tyler …Forty-sixty? Bold.
Pippa I obviously didn’t sleep on this—literally. Wake up and measure your TTFT and P95 first.
Tyler Fair enough. But given half the article’s tips need a PhD in ops, I’d say Build Next is in order.
Pippa Go on, then—make it concrete.
Tyler vLLM continuous batching plus Harmonia for RAG caching, and also the paper 'From Prefix Cache to Fusion RAG Cache'—both are open source, both ship, both have code.
Pippa You just want me to run the stack so you can critique it.
Tyler I want you to stop burning money while arguing about model routing.
Pippa Fine. Wednesday, done.