Topic
Vllm
7 episodes
-
Exploring Next Overview: Speculative Decoding
We finally slow down and unpack speculative decoding from the ground up: the draft model, the verify step, and why it can make generation faster without changing the output. We keep it concrete, because that trick sounds like cheating until the mechanism actually clicks.
-
12 Ways to Reduce LLM Latency and Inference Costs in Production KDnuggets
A practical KDnuggets piece argues that most LLM production latency/cost gains come from cutting unnecessary work instead of bigger models or more GPUs. They list 12 levers: measure the right metrics, cut output tokens, route to smaller models, collapse LLM calls, prefix caching, add multiple cache layers, control RAG context, batch offline work, tune batching for user latency, and manage KV cache. Tyler pushes back on the article’s overgeneralization of cache reuse across all tasks, the thin technical depth behind some tips, and the implication that routing to small models never backfires. Pippa highlights the piece’s strongest point—measuring TTFT, P95/P99, and queue time—because that’s where teams most often mis-diagnose bottlenecks. They land on: the article’s monitoring advice and batch-tuning guidance are solid; several recommendations work only for read-heavy workloads; and routing to tiny models is risky until you have cheap, high-confidence evaluators. They wrap with a Build Next command to try vLLM continuous batching and two open-source RAG-caching projects (Harmonia and From Prefix Cache to Fusion RAG Cache).
-
Overview: State Management in Language Models
We finally do the episode we keep circling back to: state management in language models. We walk through the idea from the ground up, using the cache-and-notes picture to show why models don’t have to recompute everything every token, and where that trade-off starts biting.
-
Micro Agent: Beat Frontier Models with Collaboration inside Model API
Justy and Cody dig into vLLM Semantic Router's Micro-Agent argument: the real product isn't a bigger model, it's a router that turns one model call into a bounded collaboration loop. They like the serving-layer abstraction, push on where the benchmark story is still thin, and land on who should actually care right now.
-
The Infrastructure Behind Making Local LLM Agents Actually Useful | Towards Data Science
A conversation about making local LLM agents actually usable, focusing on the infrastructure challenges of running scientific agents with open-weight models. The hosts discuss the author's experience building a single-cell RNA-seq analysis agent, the problem of fixed prefix costs in long tool-use loops, vLLM optimizations for inference speed, and context management for long-running sessions.
-
The team behind continuous batching says your idle GPUs should be running inference, not sitting dark
The team behind continuous batching says your idle GPUs should be running inference, not sitting dark Sean Michael Kerner March 12, 2026 Credit: Image generated by VentureBeat with Nano-Banana-2 Every GPU cluster has dead time. Training jobs finish, workloads shift and hardware sits dark while power and cooling costs keep running.
-
recipes/GLM/GLM5.md at main · vllm Project/recipes
Episode 183 explores GLM5, a new language model architecture that's pushing boundaries in multimodal understanding and reasoning. Izzo and Boone dive deep into how it handles mixed text-image inputs, its novel attention mechanisms, and why vLLM is building dedicated recipes for deployment at scale.