Domain
Inference
107 episodes
-
Overview: Structured Output
We slow down and explain structured output from the ground up: why free-form model text is awkward for software, how schemas and constrained decoding make it usable, and where the format guarantee stops.
-
Overview: Decoding Strategy
We finally slow down on decoding strategy, the rule that turns a model's next-token odds into the actual words you see. We use one hallway-and-doors picture to make greedy decoding, sampling, top-k, top-p, beam search, and newer decoding work feel less like magic knobs.
-
Overview: Sampling and Temperature
We’re finally slowing down and explaining sampling and temperature from the ground up, because we keep circling back to it and it’s one of those knobs that sounds simple until it isn’t. We talk through why the same prompt can produce different answers, how temperature changes the shape of the next-token choices, and where the trade-offs actually land.
-
Introducing Cursor Router · Cursor
Cursor Router is Cursor's new Teams and Enterprise model-routing layer, using a classifier trained on more than six hundred thousand live requests to select models by task, context, complexity, and domain. Jessica sees a clean adoption story for teams stuck paying frontier rates for routine coding work; Cathy likes the production-oriented evaluation and cache-aware accounting, while keeping an eye on how much trust enterprises place in Cursor's routing judgment.
-
Overview: Context Window Management
We finally slow down and explain Context Window Management from the ground up, because we keep hand-waving it whenever agents, memory, cost, and long tasks come up. The whole thing is the fixed-desk problem: what stays on the desk, what gets compressed, and what falls off.
-
Overview: Active vs Total Parameters
We finally slow down on active vs total parameters, because we keep throwing the phrase around like it explains itself. This is us making the difference click: what a model stores versus what it actually uses when it answers.
-
Overview: Model Routing
We finally pin down model routing, because we throw the term around all the time and somehow never actually define it. We walk through how a router sends each request to the model most likely to handle it well, and why that can save cost, latency, and a lot of dumb overgeneralization.
-
Overview: Router
We slow down on Router, the little decision-maker inside many AI systems that sends each input to the right expert, model, or retrieval path. We use the triage-desk mental model and build from intuition to mechanism, trade-offs, and where routers still matter now.
-
Overview: Conditional Computation
We finally slow down and explain conditional computation, the idea we keep casually name-dropping whenever sparse models, routers, and mixture-of-experts come up. We use the same receptionist-and-specialists picture all the way through, so the mechanism, the savings, and the catch actually stick.
-
VideoChat3:Fully Open Video MLLM for Efficient and Generalist Video Understanding
VideoChat3 is a fully open-source video multimodal LLM (4B parameters) that tackles three concrete problems: generalization across short/long/streaming video, computational efficiency for video token explosion, and reproducibility through complete open-sourcing. The core innovation is I3D-ViT (Inflated 3D Vision Transformer) plus adaptive frame resolution, which compresses spatiotemporal redundancy early in the pipeline instead of treating each frame as an independent image. Three curated datasets (2M academic + 116K long-form + 617K streaming = 3M samples total) and multi-stage curriculum learning enable the model to handle diverse video scenarios. Jessica sees a shippable foundation for real-world video apps; Cathy pushes on whether the efficiency gains hold under production load and whether the data pipeline's scale claim is reproducible.
-
Concurrent Image Understanding and Generation: Self Correcting Coupled Markov Jump Processes
Fern and Lintel dig into a new paper on doing image understanding and image generation at the same time, inside one decoding loop. The hook is simple: most systems either describe first and draw later, or they run both sides in parallel without letting the latest text and image decisions correct each other mid-step. This paper tries to fix that with a coupled masked-diffusion sampler that can both coordinate and backtrack.
-
Exploring Next Overview: Speculative Decoding
We finally slow down and unpack speculative decoding from the ground up: the draft model, the verify step, and why it can make generation faster without changing the output. We keep it concrete, because that trick sounds like cheating until the mechanism actually clicks.
-
Kimi K3 Kimi API Platform
Two friends unpack the Kimi K3 API docs, debating its 1M‑token claim, hybrid attention, and tool dynamics, and weigh who should pay the price for the hype.
-
Model Behavior: Week of July 13, 2026
We read this week as the moment the race got less obsessed with tallest-model bragging and more obsessed with who gives builders the best menu. The funny part is that the open-weight crowd is making the incumbents act practical faster than they probably wanted.
-
Overview: Token Economics
We finally slow down on Token Economics: why tokens are the meter for cost, speed, memory, and product decisions in language models. We keep using the tiny-slip postage analogy until the whole thing clicks, from tokenization to context windows to real API bills.
-
Overview: Sparse Activation
We finally sit down with sparse activation and make the idea click from the ground up: why only part of a model wakes up on each input, how routing makes that happen, and where the real trade-offs show up. We keep it concrete, because this one has been lurking under a lot of the stuff we keep talking about.
-
12 Ways to Reduce LLM Latency and Inference Costs in Production KDnuggets
A practical KDnuggets piece argues that most LLM production latency/cost gains come from cutting unnecessary work instead of bigger models or more GPUs. They list 12 levers: measure the right metrics, cut output tokens, route to smaller models, collapse LLM calls, prefix caching, add multiple cache layers, control RAG context, batch offline work, tune batching for user latency, and manage KV cache. Tyler pushes back on the article’s overgeneralization of cache reuse across all tasks, the thin technical depth behind some tips, and the implication that routing to small models never backfires. Pippa highlights the piece’s strongest point—measuring TTFT, P95/P99, and queue time—because that’s where teams most often mis-diagnose bottlenecks. They land on: the article’s monitoring advice and batch-tuning guidance are solid; several recommendations work only for read-heavy workloads; and routing to tiny models is risky until you have cheap, high-confidence evaluators. They wrap with a Build Next command to try vLLM continuous batching and two open-source RAG-caching projects (Harmonia and From Prefix Cache to Fusion RAG Cache).
-
Overview: KV Cache
We finally sit down with KV cache and make the whole thing click: what it stores, why generation gets faster, and why memory is the bill you still have to pay. We keep coming back to the same working-memory picture so the mechanics stay grounded instead of turning into acronym soup.
-
Overview: State Management in Language Models
We finally do the episode we keep circling back to: state management in language models. We walk through the idea from the ground up, using the cache-and-notes picture to show why models don’t have to recompute everything every token, and where that trade-off starts biting.
-
OpenAI Releases GPT 5.6 (Sol, Terra, Luna): A Three Tier Model Family With Programmatic Tool Calling in the Responses API
OpenAI's GPT-5.6 family — Sol, Terra, and Luna — introduces three permanent capability tiers with distinct cost profiles and a new cache billing model, plus a multi-agent Ultra mode that runs four agents in parallel by default.
-
Overview: Autoregressive Generation
We finally slow down and make autoregressive generation click: the whole thing is just a model writing one token, then using what it wrote to choose the next one. We keep the focus on the loop, the trade-offs, and why that one-step-at-a-time setup is still the backbone of modern language models.
-
How to Run Open Source AI Models
Sid Saladi argues that frontier AI vendors (Claude, GPT) bundle model, compute, access, and application into one proprietary stack—trapping users in unpredictable pricing and competitive capture. The counter: open-weight models like GLM-5.2, DeepSeek V4, Qwen, and Kimi are now frontier-adjacent in capability (GLM-5.2 beats GPT-5.5 on coding benchmarks, matches Opus 4.8 on others) and cost roughly one-sixth as much. The real problem isn't model quality anymore; it's that companies like Tesla, Uber, and Meta are hemorrhaging money on metered AI because they can't decouple the stack. The guide walks four layers—model, compute, access, harness—and shows how to own each one deliberately instead of letting a vendor own all four by default.
-
Tuning the harness, not the model: a Nemotron 3 Ultra playbook
A LangChain/NVIDIA case study claims harness tuning alone can push Nemotron 3 Ultra to 0.86 on Deep Agents at ~$4.48/run vs $43.48 for Opus 4.8, with parity latency. The hosts parse the real mechanism (matched scaffolding vs post-training), test limits (where harness hits a ceiling), and weigh who actually benefits. They surface concrete repos (langchain-ai/deepagents, langchain-ai/deepagentsjs) and a vendor profile workflow, then poke at the article’s reliance on Deep Agents and the cost math. Final take: a plausible win for teams already deep into harness work, not a universal unlock, with the hosts pushing back on ‘ten-x cheaper’ framing and under-specified benchmark footnotes.
-
I built Andrej Karpathy's "LLM Council" on my own hardware, and now no single model gets the last word
Jessica and Cathy dig into a local rebuild of Karpathy's LLM Council and land on the real claim: the win is not voting, it's structured synthesis across models with different failure modes. They like the practical adaptation to Ollama on a single twelve-gigabyte GPU, but push on where the article overreaches and where the product value is actually real.
-
Hot French startup ZML releases free product to speed inference across lots of AI chips | TechCrunch
Laura and Harper dig into ZML's new free inference server and the bigger claim underneath it: that the real leverage now is software that decouples models from chip vendors. Harper likes the direction but doubts the article proves the hard part, while Laura thinks the product story is strong even if the benchmarks are still missing.
-
Viability of local models for coding
Birgitta Böckeler tests local LLM viability for coding after a year away from the space. She maps a complex web of factors—RAM, model size, quantization, tool calling, context windows, reasoning modes—that determine whether small models actually work for agentic coding on consumer hardware (M3 Max / M5 Pro). Her core finding: locals are runneable and faster than a year ago, but tool calling is still shaky, reasoning can backfire, and quality is hit-or-miss. She's not claiming local models are ready to replace cloud APIs; she's charting what actually works and what doesn't on real machines.
-
Overview: Tokenization
We slow down and explain tokenization from the ground up: how raw text becomes numbered pieces a model can process, why those pieces are usually subwords, and why the tokenizer quietly affects cost, context, language handling, and product behavior.
-
🤗 Kernels: Major Updates
Cody and Justy dig into Hugging Face’s revamped Kernels post as an argument about whether the project is becoming a real product surface or just a more polished infrastructure layer. They focus on the new kernel repo type, security model, CLI separation, framework support, and the agentic-kernel story, while keeping an eye on what actually ships and who benefits.
-
Morphing into Hybrid Attention Models
Talon and Wildflower discuss FlashMorph, a new method for choosing which Transformer layers should keep full attention when converting pretrained LLMs into hybrid attention models.
-
How to Use RLMs in Deep Agents
Exploring Recursive Language Models (RLMs) and their implementation in Deep Agents for handling long contexts efficiently.
-
Micro Agent: Beat Frontier Models with Collaboration inside Model API
Justy and Cody dig into vLLM Semantic Router's Micro-Agent argument: the real product isn't a bigger model, it's a router that turns one model call into a bounded collaboration loop. They like the serving-layer abstraction, push on where the benchmark story is still thin, and land on who should actually care right now.
-
\ours: Advancing Masked Discrete Diffusion for High Resolution Image Synthesis
Discussion of \(\ours\) (NLD-Image), a masked discrete diffusion model that tackles two core problems in high-resolution text-to-image synthesis: the lack of self-correction in MDMs and the training difficulty with large codebooks. The paper introduces token editing for iterative refinement and Grouped Cross-Entropy (GCE) to alleviate codebook sparsity, achieving SOTA scores on GenEval, DPG, and HPSv3. Hosts debate its product readiness, mechanism soundness, and whether the gains justify the complexity.
-
AI agent memory: MRAgent cuts token use up to 27x | VentureBeat
MRAgent from NUS replaces static retrieve-then-reason memory with active reconstruction during reasoning, cutting token use 27x over competing frameworks. The system treats memory as an interactive graph where agents dynamically refine retrieval paths based on intermediate evidence, using a three-layer Cue-Tag-Content structure and automated ingestion pipelines.
-
Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks
GitHub Copilot's agentic harness is a single cross-experience SDK component that orchestrates tools, context, and workflow across CLI, app, and code review. The team claims it delivers task-resolution parity with model-vendor harnesses while cutting token usage across several configurations, backed by public and internal benchmarks and real-world metrics. We debate technical validity, practical stakes for teams, and whether the harness should get most of the credit.
-
nvidia/Nemotron TwoTower 30B A3B Base BF16 · Hugging Face
Justy and Cody dig into NVIDIA’s Nemotron-TwoTower-30B-A3B-Base-BF16 and whether block-wise diffusion decoding is a real systems win or just a benchmark-shaped detour. Cody is skeptical about the headline throughput claim and the way the model compares itself to a single autoregressive baseline, while Justy focuses on who actually benefits from faster generation without a big quality drop. They land on cautious interest: interesting infrastructure idea, but not a universal replacement for standard decoding.
-
OpenAI and Broadcom unveil LLM Optimized inference chip
OpenAI and Broadcom unveil Jalapeño, a custom AI inference chip designed for LLM workloads, promising substantial performance-per-watt improvements.
-
What 50,000 Runs of a 5 Line Eval Taught Us
The VS Code team ran a simple 5-line eval 50,000 times to test AI models' efficiency in completing a basic task: writing a string to a file. The goal was to see how reliably models could finish the work and what kinds of failures show up. The eval, called 'say_hello,' asks the model to add 'HELLO' to a file named 'HELLO.txt.'
-
How to Handle Small Context Window Limits in RAG Systems
Justy and Cody dig into a hands-on technique for making RAG work when your context window is tiny: route with summaries, answer with raw chunks, and keep an explicit budget. Cody questions whether the toy demo obscured the real complexity, and Justy sizes up who this actually saves.
-
Glint Research (GlintResearch)
The duo digs into Glint-Research’s release of svelte generative models (1M–10M parameters) that favours transparency, small-scale training, and hard limits over glossy numbers. They argue whether this bet changes anything practical and where the tech could actually break.
-
A Startup Claims It Broke Through a Bottleneck Thats Holding Back LLMs
Subquadratic claims its SubQ model breaks the dense attention bottleneck in LLMs, offering 12x context, near-SOTA performance, and radical efficiency. Third-party Appen benchmarks validate speed/cost claims, but full reproducibility and open access remain pending. Cody questions the 'transformer-replacement' hype; Justy sees a niche for high-throughput document and code analysis.
-
AI optimizer beats Claude Code, Codex by 2
Researchers introduce Arbor, a framework that structures AI optimization as a cumulative learning tree instead of isolated trial-and-error loops, delivering 2.5x verifiable gains over Claude Code and Codex on the same compute. It solves the core AO bottleneck: agents forgetting what they’ve learned across long-running experiments.
-
How to Build a Production Architecture for Small Language Model Fleets
The article discusses building a production architecture for small language model fleets, focusing on avoiding model rot. It proposes a solution involving a Model Registry, Gateway Pattern, and Manifest-based Delivery System.
-
Stanford's DeLM cuts multi Agent costs 50%
Justy and Cody dissect Stanford's DeLM framework, which claims to halve multi‑agent costs by removing a central orchestrator. They weigh the technical merits, practical impact for product teams, and the hidden scheduling trade‑offs.
-
When is Your LLM Steerable?
A new paper investigates whether you can predict if a steering attempt will succeed by looking at the model's hidden states from just the first few tokens—before decoding the full response. The authors build ASTEER, a dataset of 1.4M steered generations labeled for success/failure across 150 concepts, then train a GBDT classifier on early hidden-state features to predict three outcomes: under-steer, success, or over-steer. The predictor hits ~0.7 macro-F1 on unseen concepts and cuts the cost of finding good steering strengths without expensive full rollouts.
-
Skip a Layer or Loop It? Learning Program of Layers in LLMs
Justy and Cody dig into PoLar, a paper arguing that a pretrained LLM does not need to run its layers in one fixed order for every input. They talk through the main idea of treating layers like reusable functions that can be skipped or repeated, how the authors used Monte Carlo Tree Search to discover better layer programs, and why a lightweight predictor makes the idea more practical. The conversation centers on what problem this solves, why dynamic depth methods have been too narrow, and whether this feels like a research curiosity or an actual path to deployment.
-
End to End Context Compression at Scale
Justy and Cody dig into Latent Context Language Models (LCLMs) — encoder-decoder compressors that shrink long prompts into short latent sequences, cutting memory and latency at ratios up to 1:16 while staying competitive on accuracy. They cover the architecture search, the training recipe, the agent use-case, and what production deployment actually looks like.
-
Lip Forcing: Few Step Autoregressive Diffusion for Real time Lip Synchronization
Justy and Cody dig into Lip Forcing, a paper on making diffusion-based video-to-video lip sync actually fast enough for streaming. They unpack the core problem, the teacher-student distillation setup, the key mid-trajectory guidance insight, and what the reported speedups might mean for real products like live translation, avatars, and dubbing systems.
-
FlashMemory DeepSeek V4: Lightning Index Ultra Long Context via Lookahead Sparse Attention
Researchers propose Lookahead Sparse Attention (LSA) with a Neural Memory Indexer to slash GPU memory usage for ultra-long LLM context by pre-predicting which KV cache chunks matter, trained independently without the full backbone. FlashMemory-DeepSeek-V4 cuts physical KV cache to 13.5% of baseline on average while maintaining or improving accuracy (+0.6% abs) across LongBench-v2, LongMemEval, RULER—at 500K tokens, it suppresses KV overhead by over 90%. Project paused due to org changes; code not yet public.
-
Serving Multiple Users at Once: How Continuous Batching Keeps LLM Inference Efficient MachineLearningMastery
Continuous batching is a scheduling technique that keeps LLM inference servers from wasting GPU cycles on padding. Instead of forcing short requests to wait for long ones in a fixed batch, continuous batching frees up slots the moment a request finishes and admits new work immediately, eliminating idle padding tokens and improving throughput.
-
Shopify’s journey to faster breadth first GraphQL execution (2026) Shopify
Justy and Cody discuss Shopify's new breadth-first GraphQL execution engine, 'Cardinal,' which claims up to 15x faster execution and 90% less memory for large, nested queries by resolving fields once across all objects instead of per-object.
-
The Infrastructure Behind Making Local LLM Agents Actually Useful | Towards Data Science
A conversation about making local LLM agents actually usable, focusing on the infrastructure challenges of running scientific agents with open-weight models. The hosts discuss the author's experience building a single-cell RNA-seq analysis agent, the problem of fixed prefix costs in long tool-use loops, vLLM optimizations for inference speed, and context management for long-running sessions.
-
Replacing RAG with bash cut AI retrieval costs 30%
Justy and Cody dig into the argument behind direct corpus interaction, where agents use terminal tools like grep and find instead of relying only on vector search. They like the core point that retrieval interfaces can bottleneck reasoning, but they keep it grounded: this looks strongest for exact-evidence tasks in changing workspaces, and weakest as a blanket replacement for broad recall across huge corpora.
-
RecursiveMAS cuts multi agent AI costs by 75%: researchers
Justy and Cody dig into RecursiveMAS, a research framework that lets multi-agent systems pass latent embeddings instead of text, cutting token usage and speeding up inference while keeping base model weights frozen.
-
Hermes Unlocks Self Improving AI Agents, Powered by NVIDIA RTX PCs and DGX Spark
Hermes is a rapidly growing, self-improving AI agent framework that runs locally on NVIDIA RTX PCs and DGX Spark, using small but powerful Qwen models to do what previously required data-center scale.
-
How Lakebase Architecture Delivers 5x Faster Postgres Writes
Justy and Cody dig into Databricks Lakebase claiming much faster Postgres writes by turning off full page writes at the compute layer and pushing page image generation into distributed storage. Cody likes the architectural trick but questions where the complexity moved, while Justy argues the real win is for teams hitting write bottlenecks without wanting to re-architect their app.
-
Implementing Prompt Compression to Reduce Agentic Loop Costs MachineLearningMastery
Justy and Cody kick around whether prompt compression is actually a smart production habit or just another neat demo. Cody starts skeptical about summary drift and hidden complexity, then they get concrete on why long agent loops get expensive fast, what the article's Python example is really proving, and where compressed history plus distilled instructions make sense right now.
-
Local First AI Inference: A Cloud Architecture Pattern for Cost Effective Document Processing
Justy and Cody debate Local-First AI Inference — a pattern that routes most documents to deterministic local extraction while falling back to cloud AI for edge cases. They unpack the signal in the noise: who actually benefits, the clever confidence-gated routing, the real cost savings, and the architectural trade-offs. Then they lay out concrete ways to test the claims over a weekend.
-
Scaling real time performance with Bigtable in memory tier | Google Cloud Blog
Justy and Cody geek out over Bigtable's new in-memory tier, which uses RDMA to deliver sub-millisecond reads. Justy sees a product manager's dream for removing cache-layer nightmares, while Cody explains how direct memory access avoids CPU bottlenecks and why the hotspot resistance is the real game-changer.
-
Stop Wasting Tokens: A Smarter Alternative to JSON for LLM Pipelines KDnuggets
Cody is skeptical that TOON is a universal fix for JSON in LLM pipelines, and Justy pushes that the real win is for repeated structured records where token cost and clarity both matter. They land on TOON as a useful pre-LLM transport format, not a replacement for JSON everywhere.
-
Speeding up agentic workflows with WebSockets in the Responses API
Justy and Cody dig into OpenAI’s writeup on speeding up agentic workflows with WebSockets in the Responses API. Cody is skeptical of the hype around raw model speed, while Justy keeps pulling it back to user pain: long, repetitive agent loops that make coding tools feel sluggish. They land on a practical read — the transport change matters most when the model is fast enough that API overhead becomes the bottleneck — and they sketch a weekend experiment for building a tiny stateful agent loop.
-
The context window has been shattered: Subquadratic debuts a 12 Million Token window
Cody is skeptical that a 12-million-token context window is broadly useful today, while Justy pushes the angle that it solves a very real pain point for teams with giant codebases, logs, and long-running workflows. They land on it as a real technical milestone with a narrow early market, plus a lot of unanswered questions about cost, latency, and whether most users need this kind of scale.
-
How NetEase Games cut LLM cold starts from 42 minutes to 30 seconds
Justy and Cody unpack how NetEase Games used Kubernetes-native data orchestration with Fluid to shrink LLM inference cold starts from 42 minutes to about 30 seconds, and what that means for teams running their own models.
-
Google AI breakthrough means chatbots use six times less memory during conversations without compromising performance
Google's TurboQuant compresses AI working memory (the KV cache) by up to 6x in real time using two novel techniques — PolarQuant and QJL — without degrading model performance. Justy and Cody dig into what this actually means for inference costs, who benefits first, and why the 'DeepSeek moment' framing is both apt and a little overblown.
-
Agentic AI: How to Save on Tokens | Towards Data Science
Cody and Justy examine whether the token-saving techniques in Ida Silfverskiöld's article (prompt caching, semantic caching, lazy-loading, routing, context cleanup) are practical wins or theoretical cost-cutting that introduces real friction. Cody opens skeptical: the savings are real but the tradeoffs are often hidden or underestimated. Justy counters that for production teams already bleeding money on agentic AI, even 20-30% savings justifies the engineering lift. They land on a nuanced take: prompt caching is genuinely low-risk and worth it; semantic caching and aggressive routing are trickier and need honest trade-off audits before deployment.
-
You don't need an expensive GPU to run a local LLM that actually works
Cody and Justy examine the claim that you don't need an expensive GPU to run capable local LLMs. Cody opens skeptical about quantization trade-offs and real-world inference speed; Justy pushes back with the actual user story—cost-conscious builders and privacy-first home automation. They dig into what 'works' really means, explore the CPU-only vs. GPU trade-off, and land on a nuanced take: smaller quantized models on mid-range hardware are genuinely usable now, but marketing around this can oversell the experience. Build Next includes testing Ollama on a specific budget GPU and benchmarking a 7B quantized model on a CPU-only rig.
-
Cut AI token usage by 96%? Here's how AWS Strands Agents does it.
AWS Strands Agents is a way to cut agent token usage by making models ask for only the context they need, when they need it. Instead of stuffing huge prompts up front, it uses tools, memory, and session state to keep agents lean, which matters for cost, latency, and scaling.
-
Stop Hitting Claude Code Limits
Claude Code's usage limits aren't the real problem—how you set it up is. Four controllable causes drive 85% of overspend: cache misses, context bloat, wrong model routing, and token-heavy input formats. One user cut costs from $1,389/mo to $200/mo by locking tools at session start, disabling 1M context, delegating to cheaper subagents, and swapping screenshots for accessibility trees. Real fixes are copy-paste configuration changes and workflow tweaks, not waiting for Anthropic.
-
Recursive Multi Agent Systems
RecursiveMAS is a new multi-agent framework from researchers at UIUC, Stanford, NVIDIA, and MIT that replaces text-based agent handoffs with latent-space recursion — cutting token usage by up to 75%, speeding up inference 2.4x, and improving accuracy by 8.3% across nine benchmarks. Justy and Cody dig into why passing hidden states instead of words is such a big deal, what the RecursiveLink module actually does, and whether any of this is shippable today.
-
Stochastic KV Routing: Enabling Adaptive Depth Wise Cache Sharing
Justy and Cody dig into Stochastic KV Routing, a paper on cutting transformer KV cache memory by sharing caches across layers instead of only squeezing along the token axis. They unpack random cross-layer attention, why it helps models tolerate missing per-layer caches, and where this could matter in real serving stacks.
-
Contexts are Never Long Enough: Structured Reasoning for Scalable Question Answering over Long Document Sets
SLIDERS solves the aggregation bottleneck in document question answering by extracting information into a relational database and reasoning over structured data via SQL instead of concatenating chunks. It uses data reconciliation to fix duplicates and inconsistencies, outperforming GPT-4 on long-context benchmarks and scaling to 36M tokens.
-
Contexts are Never Long Enough: Structured Reasoning for Scalable Question Answering over Long Document Sets
A new QA framework called SLIDERS handles document sets that outgrow any context window by storing extracted facts in a relational database and reasoning over them with SQL.
-
DeepSeek V4 arrives with near state of the art intelligence at fraction of the cost of Opus 4.7, GPT 5
Justy and Cody unpack DeepSeek-V4, an open-weight MoE model that gets close to top closed models on several practical benchmarks while landing in a much lower price tier. They focus on why cheaper frontier-class inference changes what teams can afford to automate, where DeepSeek still trails GPT-5.5 and Claude Opus 4.7, and what builders can try this weekend.
-
6 Things I Learned Building LLMs From Scratch That No Tutorial Teaches You | Towards Data Science
Justy and Cody dig into what actually changes when you stop calling an LLM API and start building pieces yourself: why fine-tuning tricks like RsLoRA matter, why RoPE won, where weight tying still makes sense, why Pre-LN became the default, and how KV cache buys speed by spending memory.
-
Moonshot AI and Tsinghua Researchers Propose Prfaas a Cross Datacenter Kvcache Architecture That Rethinks How LLMs Are Served at Scale
Justy and Cody unpack PRFaaS, a cross-datacenter KV-cache serving design from Moonshot AI and Tsinghua that tries to make LLM inference less wasteful by treating prefills as reusable networked assets instead of repeating them in every region.
-
The Complete Guide to Inference Caching in LLMs
Justy and Cody dig into inference caching for LLMs and why it matters right now for anybody paying real model bills or waiting on sluggish responses. They unpack the three layers from the article — KV caching inside a single generation, prefix caching across requests with identical leading tokens, and semantic caching using embeddings plus vector search to skip model calls entirely. The episode stays grounded in production reality: prompt structure, exact-match requirements, provider behavior, GPU memory trade-offs, and when semantic caching is actually worth the extra moving parts.
-
Cloudflare Launches Code Mode MCP Server to Optimize Token Usage for AI Agents
Cloudflare's new Model Context Protocol (MCP) server powered by Code Mode reduces token usage for AI agents, making it possible to interact with complex APIs more efficiently.
-
selimaktas/MiniMax M2.75 460B A20B · Hugging Face
Exploring the capabilities and potential applications of the MiniMax-M2.75-460B-A20B model, a text generation transformer that outperforms its base model on Single-turn SWE-Bench and has achieved impressive results in software engineering, professional work, and entertainment.
-
We're bringing the advisor strategy to the Claude Platform. Pair Opus as an advisor with Sonnet or Haiku as an execu...
Claude AI's advisor strategy and its implications on AI development
-
Alright agent nerds, if you care about your tokens and usage limits, pay attention to the tools you give to your agen...
Episode 280 of Exploring Next dives into the importance of choosing the right browser tools for agents, exploring their impact on token usage and latency.
-
Rightnow AI Releases Autokernel an Open Source Framework That Applies an Autonomous Agent Loop to GPU Kernel Optimization for Arbitrary Pytorch Models
Exploring the release of Autokernel, an open-source framework for autonomous GPU kernel optimization in PyTorch models
-
Andrej Karpathy Just 10x’d Everyone’s Claude Code
Episode 271 of Exploring Next dives into Andrej Karpathy's recent work on Claude, which has significantly improved its capabilities. The discussion revolves around the substance of the project, its architecture, and how it works, with a focus on the product angle and technical aspects.
-
Running local models on Macs gets faster with Ollama's MLX support
Ollama just added MLX support for Apple Silicon Macs, promising significantly faster local LLM performance through better unified memory usage. We break down what this actually means, why it matters as local models gain momentum, and the technical architecture that makes it work.
-
Prismo Optimize AI Costs
Prismo is an AI cost optimization platform that acts as a drop-in proxy between your application and AI providers like OpenAI and Anthropic. By routing requests through Prismo's gateway, teams get real-time spend tracking, automated budget enforcement, and intelligent model routing that can reduce costs by up to 40%. The platform requires just a one-line code change to integrate and provides full visibility into AI spending across teams, services, and models.
-
Reasoning as Compression: Unifying Budget Forcing via the Conditional Information Bottleneck
Chain-of-Thought prompting makes LLMs more accurate but expensive. This research reframes efficient reasoning as a compression problem, introducing a conditional information bottleneck approach that preserves essential reasoning while cutting cognitive bloat. Instead of naive length penalties, they use semantic priors based on token surprisal to compress reasoning traces intelligently.
-
How xMemory cuts token costs and context bloat in AI agents
Featured How xMemory cuts token costs and context bloat in AI agents Ben Dickson March 25, 2026 Image credit: VentureBeat with ChatGPT Standard RAG pipelines break when enterprises try to use them for long-term, multi-session LLM agent deployments. This is a critical limitation as demand for persistent AI assistants grows.
-
The team behind continuous batching says your idle GPUs should be running inference, not sitting dark
The team behind continuous batching says your idle GPUs should be running inference, not sitting dark Sean Michael Kerner March 12, 2026 Credit: Image generated by VentureBeat with Nano-Banana-2 Every GPU cluster has dead time. Training jobs finish, workloads shift and hardware sits dark while power and cooling costs keep running.
-
Netflix Uncovers Kernel Level Bottlenecks While Scaling Containers on Modern CPUs
Netflix discovered that scaling hundreds of containers simultaneously hits deep kernel-level bottlenecks in the Linux virtual filesystem, where thousands of mount operations create lock contention that varies dramatically across different CPU architectures. Their solution involved redesigning overlay filesystems to reduce mount operations from O(n) to O(1) per container.
-
New KV cache compaction technique cuts LLM memory 50x without accuracy loss
MIT researchers developed Attention Matching, a KV cache compaction technique that achieves 50x memory reduction in LLMs without accuracy loss, solving a critical bottleneck for enterprise applications handling long contexts.
-
Top 7 Small Language Models You Can Run on a Laptop MachineLearningMastery
Izzo and Boone explore seven small language models that run locally on laptops, diving deep into the technical trade-offs, hardware requirements, and real-world use cases. They break down everything from Phi-3.5 Mini's long-context capabilities to Llama 3.2's versatility, examining why local inference matters and how to choose the right model for your specific needs.
-
recipes/GLM/GLM5.md at main · vllm Project/recipes
Episode 183 explores GLM5, a new language model architecture that's pushing boundaries in multimodal understanding and reasoning. Izzo and Boone dive deep into how it handles mixed text-image inputs, its novel attention mechanisms, and why vLLM is building dedicated recipes for deployment at scale.
-
Transformers.js v4 Preview: Now Available on NPM!
Transformers.js v4 brings massive performance improvements with a new C++ WebGPU runtime, modular architecture, and standalone tokenizer library. Now runs state-of-the-art AI models directly in browsers, Node, and Deno with hardware acceleration.
-
'Observational memory' cuts AI agent costs 10x and outscores RAG on long Context benchmarks
Observational memory is a new approach to AI agent memory that uses two background agents to compress conversation history into dated observation logs, achieving 10x cost savings through stable context windows that enable prompt caching while outperforming traditional RAG systems on long-context benchmarks.
-
Context Engineering: An Introduction to the Information Environment for LLMs
A deep dive into context engineering reveals how structuring information for large language models enhances their performance and relevance. It’s more than just managing prompts—it's about creating a dynamic environment that allows AI to engage intelligently. This discussion explores why these strategies matter, who stands to benefit, and practical examples of their application.
-
10 strategies to reduce MCP token bloat
In today's tech landscape, managing token bloat is critical for efficient application performance. This dialogue dives into strategies for reducing MCP token bloat, emphasizing its importance for developers and organizations alike. The hosts explore practical solutions and real-world implications, showcasing how these strategies can lead to smoother operations and enhanced user experiences.
-
Latent Chain of Thought as Planning: Decoupling Reasoning from Verbalization
This episode explores the innovative PLaT framework for reasoning in large language models, which introduces a two-part system separating reasoning from verbalization. It addresses the challenges of computational efficiency and interpretability, paving the way for more effective AI solutions across various domains. By discussing practical implications and potential use cases, we highlight how this research can transform the landscape of AI applications and improve user experiences.
-
MAXS: Meta Adaptive Exploration with LLM Agents
MAXS introduces an innovative framework for improving the reasoning capabilities of LLM agents, addressing critical issues in multi-tool reasoning. The integration of lookahead strategies and trajectory convergence allows for more stable and efficient performance, making it highly relevant for developers and practitioners.
-
Webgpu Changed How I Think About Web Performance D63e771d1cee
WebGPU is revolutionizing web performance by drastically enhancing graphics and data processing speeds, showing a 23x improvement over WebAssembly in practical applications. This shift in technology not only benefits developers looking for efficient solutions but also enhances user experiences in data-intensive applications.
-
TimeBill: Time Budgeted Inference for Large Language Models
This episode dives into the innovative framework of TimeBill for time-budgeted inference in Large Language Models (LLMs), exploring its implications in time-sensitive applications and its adaptive mechanisms that enhance performance.
InferenceResearch Paper -
Multi hop Reasoning via Early Knowledge Alignment
The research on Early Knowledge Alignment enhances how Large Language Models retrieve and reason with information, particularly for complex queries. This innovation improves precision and efficiency, benefiting developers in creating more effective AI systems.
-
MIT offshoot Liquid AI releases blueprint for enterprise Grade small Model training
Liquid AI's new blueprint for small-model training positions enterprises to leverage AI on-device efficiently, ensuring privacy and operational reliability without reliance on cloud-based solutions. This shift could transform how businesses implement AI, enabling real-time applications that enhance productivity and data security.
-
Ibms Open Source Granite 4 0 Nano AI Models Are Small Enough to Run Locally
In this episode, we explore IBM's Granite 4.0, a breakthrough in nano-AI models that can run locally, transforming how AI is integrated into everyday devices and applications. We discuss the implications for privacy, efficiency, and accessibility, and share real-world scenarios that highlight its potential impact on industries.
-
New Markovian Thinking Technique Unlocks a Path to Million Token AI
In this episode, we dive into a groundbreaking technique in AI that dramatically expands the token processing capabilities of language models, paving the way for more advanced applications.
-
New Token Oriented Object Notation (TOON) Hopes to Cut LLM Costs by Reducing Token Consumption
InfoQ Homepage News New Token-Oriented Object Notation (TOON) Hopes to Cut LLM Costs by Reducing Token Consumption Development New Token-Oriented Object Notation (TOON) Hopes to Cut LLM Costs by Reducing Token Consumption Nov 23, 2025 2 min read by Bruno Couriol Write for InfoQ Feed your curiosity. Help 550k+ global senior developers each month stay ahead.
-
Will DeepSeek's new AI model break the 'long context' bottleneck holding back LLMs?
Tech AI Will DeepSeek's new AI model break the 'long-context' bottleneck holding back LLMs? South China Morning Post Wed, October 22, 2025 at 9:30 AM UTC DeepSeek's new artificial intelligence model that converts images into text is not just a document parsing tool but a potential preview of its next generation of large language models (LLMs), according to AI experts.
-
7 LLM Generation Parameters What They Do and How to Tune Them
Editors Pick Agentic AI Staff Tech News 7 LLM Generation Parameters—What They Do and How to Tune Them? By Michal Sutter - October 14, 2025 Tuning LLM outputs is largely a decoding problem: you shape the model’s next-token distribution with a handful of sampling controls— max tokens (caps response length under the model’s context limit), temperature (logit scaling for more/less randomness), top-p / nucleus and top-k (truncate the candidate set by probability mass or rank), frequency and presence penalties (discourage repetition or encourage novelty), and stop sequences (hard termination on delimiters).
InferenceBlog -
JavaScript Library Runs Machine Learning Models in Browser
A new JavaScript library enables developers to run machine learning models directly in the browser, making AI more accessible and efficient.
-
I switched from LM Studio/Ollama to llama.cpp, and I absolutely love it
I switched from LM Studio/Ollama to llama.cpp, and I absolutely love it Credit: By Dhruv Bhutani Published Nov 2, 2025 Dhruv Bhutani has been writing about consumer technology since 2008, offering deep insights into the personal technology landscape through features and opinion pieces. He writes for XDA-Developers, where he focuses on topics like productivity, networking, self-hosting, and more.