Ep 1140 Research Paper 5:48 w/ Justy & Cody

Why Do Multi Agent LLM Systems Fail?

A close look at MAST, a taxonomy and dataset for diagnosing why multi-agent LLM systems fail. The paper’s practical contribution is turning vague coordination failures into observable categories, while its evidence also shows why labeling root causes remains difficult.

Embed this episode

Paste this on any site — the player is a self-contained iframe with no cookies or trackers.

<iframe src="https://sandrise.io/exploring-next/embed/1140"
  width="100%" height="180" style="max-width:640px;border:0;border-radius:12px;overflow:hidden"
  title="Exploring Next — Episode 1140 audio player"
  loading="lazy" allow="autoplay" referrerpolicy="strict-origin-when-cross-origin"></iframe>
Embed & API docs →
Script GPT-5.6 luna Voice ElevenLabs v4

Transcript

Justy The striking number in this paper is not a new benchmark win. It’s that seven multi-agent systems fail between forty-one and eighty-six point seven percent of the time.

Cody And that range is across their task runs, not a claim that every multi-agent system is doomed. But it does explain why adding agents can feel like adding a project manager, three meetings, and a shared document to a task that was already shaky.

Justy Exactly. The paper is asking the question teams keep getting stuck on: when the whole system fails, which agent, handoff, instruction, or missing check actually caused it?

Cody Right.

Justy That’s the useful angle here. MAST is a failure taxonomy, meaning a shared set of labels for recurring breakdowns. MAST-Data is the accompanying collection of more than sixteen hundred execution traces, so people can study those breakdowns instead of trading screenshots of spectacular demos.

Cody The traces matter because a final wrong answer hides the path. An agent may misunderstand its role, another may accept a bad intermediate result, and the evaluator may never verify the actual task. From the outside, that looks like one failure. Inside the trace, it can be a chain of failures.

Justy I had to stop myself from mentally turning every one of those traces into a little org chart. We’ve had this argument before: multiple agents sound like specialization, but every handoff creates another place where state can drift.

Cody Mm-hm.

Justy The researchers build the taxonomy by closely examining more than one hundred fifty traces, some averaging over fifteen thousand lines of text. Six expert annotators worked through the patterns, then three annotators independently labeled fifteen traces until their agreement reached a kappa of zero point eighty-eight.

Cody That statistic is measuring how consistently the humans applied the labels, beyond what random agreement would produce. It’s strong evidence that the categories are usable. It is not evidence that the categories capture every possible failure, or that a label automatically identifies the true root cause.

Justy That distinction is the whole production story for me. The fourteen modes sit in three broad buckets: system design problems, agents getting out of sync with one another, and task verification problems. Those are categories a team can actually map onto logs and workflow stages.

Cody And the paper gives concrete examples. A system can violate a role specification, fail to pass useful context, or let an agent terminate before the group has reached the required decision. Verification failures are especially dangerous because the system can produce a polished answer without checking whether it completed the task.

Justy Yeah, the polished wrong answer is the expensive one.

Cody The researchers then use an LLM annotator to scale the labeling. It uses OpenAI’s o1 model and reaches kappa values of zero point seventy-seven against human labels, and zero point seventy-nine on two additional unseen systems and benchmarks.

Justy On a normal episode like this, we’re trusting the paper and our read of it rather than re-deriving every annotation decision ourselves, so one small exact figure could be off. But the direction is clear enough to discuss, and the paper is unusually open about where the measurement comes from.

Cody Right, and the measurement still has a causality problem. The annotator sees the trace after the fact. If agent two receives confused context because agent one made a bad plan, do we label the planner, the handoff protocol, the prompt, or the missing validator? The taxonomy can expose the pattern without proving which intervention would fix it.

Justy That’s why I like the ChatDev example. The CPO agent could end the conversation without consensus from the CEO agent. Changing the workflow so the CEO had final say increased overall task success by nine point four percent. That’s not a universal recipe, but it shows the labels can point to a shippable experiment.

Cody Exactly.

Justy The experiment is small enough to imagine in a real codebase: capture the full trace, tag the failure mode, change one coordination rule, and measure task completion again. That is much more useful than saying the agents need to be smarter.

Cody I’m a little more cautious about the headline failure rates, because the systems, models, and benchmarks differ. The paper covers coding, math, and general agent tasks, with models including GPT four, Claude three, Qwen two point five, and Code Llama. That breadth is valuable, but it also makes a single pooled number hard to interpret.

Justy Fair. Still, the paper lands on a practical conclusion without pretending it solved architecture. A taxonomy is a debugging surface. It doesn’t tell you that one agent beats seven, or that seven agents are worth the cost.

Cody And that connects to our recurring multi-agent question. The live fight is still specialization versus coordination overhead. This paper is another entrant, not the final answer: it says the overhead is visible and classifiable, while leaving open how much of it better orchestration can remove.

Justy For builders, the release gives you MAST-Data, the smaller MAST-Data-human set with twenty-one traces annotated by three experts, and the MAST library. The paper says you can use it through the Python package with pip install agentdash.

Cody I’d use that as an inspection layer before adding another worker. Compare failure labels across your own traces, check whether the LLM annotator agrees with a human sample, and then test one workflow change at a time. Otherwise the taxonomy becomes another dashboard that reports trouble without reducing it.

Justy That’s a good place to leave it. The agents can have a committee meeting after we’ve checked whether anyone wrote down the objective.

Cody A surprisingly expensive way to rediscover a missing validator.

Justy And that’s enough multi-agent governance for one sitting. Cody, I’ll see you in the next pile of traces.