Tracing Agentic Failure from the Flow of Success

Jessica Okay, this is weirdly satisfying. A paper about figuring out which step went wrong, but you train only on the good runs. That feels very Exploring Next, honestly.

Cathy Yeah, because the failure-label bottleneck is the whole mess here. If your agent trajectory is two hundred steps long and the mistake gets half-buried by recovery steps, asking humans to annotate the exact bad step is just brutal.

Jessica Mm-hm.

Cathy And the older fixes are either expensive prompting pipelines or post-training on failure traces with step-level labels. So the paper’s move is: stop waiting for the bad data, learn normal behavior from success, then treat deviations as suspicious.

Jessica How's your week going, by the way? Mine's been very "let's read one more failure-attribution paper and pretend that's normal."

Cathy Ha! Normal enough. I mean, I did not sleep on this one, literally, but the idea is clean. They call the model Oat, which is a little too cute for the amount of machinery underneath it.

Jessica Right, Oat is doing that one-class thing in latent space. So you take each step in the trajectory, embed it with an LLM, and model the whole sequence as a continuous path instead of a bunch of disconnected logs.

Cathy Exactly. The key is Neural CDEs, which are a principled way to handle irregular trajectories. Here, they use them to learn the dynamics of successful agent behavior, not just static step fingerprints.

Jessica I like that, because the product version of this is not "give me a vibe score for the whole episode." It’s "tell me which step I should inspect first." That’s what people can actually use when they’re debugging a broken agent.

Cathy Right, and the paper is pretty explicit that the anomaly score comes from deviation from the learned normal flow. At inference time, each step gets scored against the expected successful path, and the high-deviation steps get flagged as error steps.

Jessica Oh interesting.

Cathy The part I care about is the gated control path. They’re not just blindly trusting every control signal from an unseen trajectory. The gate can suppress out-of-distribution inputs, which matters if you want this to survive outside the exact benchmark domain.

Jessica That sounds like the difference between a demo and something you can park inside a real agent stack. Because real systems do not politely stay inside the training distribution. They wander off and then act offended about it.

Cathy No way.

Jessica And they trained it on only one hundred successful trajectories. No failure data. No step annotations. That is such a tiny supervision budget that I almost don’t believe it until I read the rest of the paper.

Cathy I buy the direction, but I’d still want to know how sensitive it is to the quality of those successes. If the success traces are narrow or overly templated, one-class methods can just learn a brittle shell and call everything else weird.

Jessica Sure, but even with that caveat, the operational story is strong. The model itself is just a lightweight three-layer MLP on top of the latent machinery, and at inference it needs zero token budget. That’s the kind of thing teams will actually ship into a debugger.

Cathy And the reported speed gap is big: two hundred to five thousand times faster than prompting baselines. That’s not a small engineering win. That’s the difference between something you run once in a postmortem and something you can use while the failure is still fresh.

Jessica Ha! Okay, yes, that is a very non-academic amount of faster.

Cathy The results are also better on both in-domain and out-of-distribution data, with plus twenty percent F one in-domain and plus seven percent O O D. The O O D piece is the one I’d watch, because that’s usually where these neat latent tricks fall apart.

Jessica And they say it beats prompting frontier models, including GPT-five. That part matters less as a trophy and more because it changes the cost picture. If a tiny model can do the attribution job better, you stop paying the frontier tax for every failure trace.

Cathy Yeah, though I’d still ask for more detail on failure modes. Like, does it localize the first bad step, or the most anomalous step, and those aren’t always the same thing. In agent traces, later compensation can hide earlier causality.

Jessica That's fair. But I do think the framing is smart: they’re not pretending to solve every causal question, just the practical debugging question of where to look. That’s usually where useful tools start, not with philosophical purity.

Cathy Mm-hm.

Jessica So for who builds with this, I’d say anybody shipping long-horizon agents, especially coding and tool-using systems, where trace review is already part of the pain. This feels less like a research toy and more like a component in an evaluation or observability stack.

Cathy Yeah. If I were wiring it in, I’d probably want it beside whatever deterministic checks or human review you already have. Not as the final judge, just as a fast triage layer that points at likely bad steps before you spend frontier-model money.

Jessica Okay, that's the exact kind of boring-adjacent infrastructure I trust. Also, Oat is a ridiculous name for something that is basically "normal-flow debugger," which I say with affection.

Cathy Ha! You'd rename everything into a product label if you could.

Jessica Maybe. But this one earns it. A tiny model, no failure labels, no inference tokens, and it still beats the expensive stuff? That is the rare paper where the shipping story is not an afterthought.

Cathy I’m with you, mostly. I’d just want broader replications, more messy real-world trajectories, and a little more stress on how the gate behaves when the success set is small. But as a direction, this is solid.

Jessica Yeah, okay, that’s a very Cathy ending. Suspicious, but not annoyingly so.

Cathy I contain multitudes.

Jessica Sure you do. Anyway, I cannot believe we spent episode seven hundred on a model named Oat, but here we are.