AI agent tool routing cuts token use 99% | VentureBeat

Cooper Ninety-nine percent token savings is such a ridiculous headline that I immediately assumed you were going to hate this.

Miles I mean… I hated the headline, yes. The actual idea, less so. This is basically them saying the agent doesn't need to see the whole phone book of tools every time it wants to do one job.

Cooper Right, and that's the part that matters right now. Everybody's wiring agents into giant tool shelves, especially around M C P, and then acting shocked when the model gets weird once there are two thousand things in reach.

Miles Yeah.

Miles The central claim isn't really token savings, even though that's the flashy number. It's that decomposition quality is the routing bottleneck. If the model breaks a task into vague steps, retrieval goes bad. If it breaks the task into steps using the vocabulary of actual skills in the library, the whole plan snaps into place.

Cooper Which is very you. Constraint language, harness language, same disease. We keep ending up at the serving layer being the benchmark again.

Miles It really is the same move. And, Cooper, I actually think this one earns it because they aren't pretending the model just got smarter. They built a loop where the model drafts a plan, retrieval finds loosely matching tools, then those tool names get fed back in as hints so the model rewrites the plan around reality.

Cooper My week is apparently just me reading papers where the answer is, wow, the system should know what tools it has. Very glamorous Wednesday behavior for almost seven months into this extremely unserious podcast.

Miles That's a brutal description of what we do, and also fair.

Cooper But it does feel practical. If your internal agent has a tiny tool set, who cares. If you've got hundreds of connectors and wrappers and little business-specific actions, this is suddenly product-shaping, because the failure mode stops being intelligence and starts being discoverability.

Miles Mm-hm.

Miles Their pipeline is clean. Decompose the user request into atomic sub-tasks. Retrieve top candidates for each sub-task with embeddings, they used MiniLM with F A I S S. Then compose a final execution graph by checking whether outputs and inputs between selected skills are compatible, and they represent that as a D A G so independent branches can run in parallel.

Cooper And the example in the piece is the boring enterprise one, which honestly is good. Download a dataset, transform it, make visual reports. That's exactly the kind of thing where one-shot tool choice is fake competence, because the task was never one tool in the first place.

Miles Exactly.

Miles The interesting technical bit is the Skill-Aware Decomposition loop. Vanilla seven B decomposition got the right number of steps only fifty-one point zero percent of the time. Turn on S A D and it goes to sixty-seven point seven. On the larger Qwen-Max setup they say ninety-two percent. Hard tasks, four to five skills, improved by fifty percent.

Cooper Those are real gains, though the ninety-nine point nine percent token cut is also doing a slightly easier trick, right? If your baseline is shoving all two thousand two hundred nine tools into the prompt, of course not doing that wins.

Miles Right, right.

Miles Yes. That's where the article is a little too pleased with itself. Going from roughly eight hundred eighty-four thousand tokens to about one thousand one hundred sixty is dramatic, but the baseline is kind of absurd. No careful team would ship the brute-force version unless they were speedrunning a cloud bill.

Cooper Some teams absolutely would. Do not underestimate the industry's commitment to yelling "just give the model everything" and calling it architecture.

Miles Fair. I retract my faith in people.

Cooper Also the bigger-model-does-worse thing is catnip for people who want a dunk, but the nuance matters. The fourteen B model over-decomposed into tiny nonsense steps when unguided. That's not proof small models are better. It's proof that without anchors, more verbal fluency can just generate more wrong structure.

Miles Yes, completely. And I liked that they showed ReAct basically face-planting here, zero percent decomposition accuracy, because that lines up with the old thing we keep coming back to. A loop that chooses the next action is not the same as an explicit multi-step plan with compatibility checks.

Cooper This is such an Exploring Next problem. We keep discovering that one universal loop is not the answer, and then some benchmark comes along and rediscovers it with a new noun.

Miles No way.

Miles But seriously, I think the argument holds up technically with one caveat. They only get top-one exact tool ranking around thirty-seven percent with the bi-encoder, even though top-ten recall is close to seventy percent. So in practice you probably need a reranker layer, either cross-encoder or L L M judge, before I'd trust this in a messy enterprise stack.

Cooper That's the practical cut for me too. Who should care? Teams with large, messy tool inventories. Platform teams building internal agent layers. Anybody trying to expose a huge skill catalog to a model and wondering why usage is flaky. If you're building a simple agent with six tools, this changes approximately nothing.

Miles I’d put maybe seventy-thirty on this pattern becoming standard in serious tool-heavy agents by year end. Not necessarily this framework by name, but the retrieve-rewrite-route loop. It solves too obvious a problem.

Cooper I'd go even higher on the pattern, lower on the branding. SkillWeaver as a name, whatever. The loop itself feels sticky. And since they haven't shipped code, it's more recipe than product anyway. Prompt templates in the paper, standard libraries like LangChain or LlamaIndex, raw Python if you're stubborn.

Miles The indexing setup is refreshingly un-dramatic too. They say embedding and indexing all two thousand two hundred nine skills took fifteen seconds, retrieval added under fifteen milliseconds. That's not where the pain is. The pain is keeping the skill docs clean enough that retrieval means anything.

Cooper Which is where your beloved boring work comes back. You don't get to have a magical agent catalog if your tools are named like internal jokes and half the descriptions were written at two in the morning by somebody mad at YAML.

Miles Honestly, that may be the whole episode.

Cooper Yeah, okay. Let's not spend episode five ninety-two becoming the documentation police. Go be gloomy somewhere else, Miles.