Ep 1118 Research Paper 6:02 w/ Onyx & Echo

Continual Learning Exam: Benchmarking Long Horizon RSI and Experiential Learning in LLM Agents

CLE (Continual Learning Exam) is a 600-level benchmark built by 50+ domain experts to measure whether LLM agents actually improve through accumulated experience across long workflows—not just solve isolated tasks. The benchmark spans 89 professional roles in 12 domains, with tasks organized as five-level ladders where later levels depend on knowledge gained earlier. The key finding: even frontier models with diagnostic feedback and revision opportunities clear only 47.7% of levels and complete all five levels in just 23.3% of tasks. This reveals a hard gap between task-solving capability and sustained learning from experience.

Embed this episode

Paste this on any site — the player is a self-contained iframe with no cookies or trackers.

<iframe src="https://sandrise.io/exploring-next/embed/1118"
  width="100%" height="180" style="max-width:640px;border:0;border-radius:12px;overflow:hidden"
  title="Exploring Next — Episode 1118 audio player"
  loading="lazy" allow="autoplay" referrerpolicy="strict-origin-when-cross-origin"></iframe>
Embed & API docs →
Script Haiku 4.5 Voice Rime Coda

Transcript

Onyx So CLE—Continual Learning Exam—is basically the first benchmark that actually measures whether agents learn from experience over time, not just whether they solve isolated tasks.

Echo Right. And that distinction matters because almost every existing benchmark—GAIA, WildClawBench, even the memory ones—they score you on individual task performance. Task comes in, agent solves it or doesn't, you get a number. CLE flips that entirely.

Onyx Exactly. It's 120 human-authored tasks, each one organized as a five-level ladder. So 600 levels total across 89 professional roles, twelve domains. And the point is: each level builds on what the agent learned in earlier levels.

Echo Mm-hm.

Onyx The setup is tight. They provide 23,981 real-world input files, 19 stateful mock applications with 1,024 APIs—Git, scheduling, project coordination. So later levels don't just test new skills; they test whether the agent retained and applied knowledge from earlier work.

Echo And the grading is rubric-based. 16,125 human-authored evaluation criteria, each one specifies a single unambiguous outcome and the evidence required to verify it. An evaluation agent checks submissions against that rubric.

Onyx So they ran two trials: Trial One, agents get one submission per level with no feedback. Trial Two, if they fail, they get diagnostic feedback and one revision. And here's the part that stops you—

Echo The best-performing system. Claude Opus 5 with Code. Clears 47.7% of all levels.

Onyx And completes all five levels in just 23.3% of tasks.

Echo Even with diagnostic feedback and a second chance. Median gain from feedback and revision across all configs? Five point three percentage points.

Onyx That's the thing that gets me. These are frontier models. They're scoring ninety-plus percent on GAIA. But put them in a sequence where they need to learn and carry forward lessons, and they fall apart.

Echo Right. And the gap isn't subtle. It's not like they're at eighty-five percent and need to hit ninety. They're at forty-seven percent. That's not a calibration problem; that's a capability gap.

Onyx But here's what I think matters for production: this is built on real workflows. Git integration, meeting scheduling, project updates across apps. Not toy tasks.

Echo Yeah, and the sequencing is genuine—later levels depend on artifacts and decisions from earlier ones. You can't cheat it by memorizing a pattern. You actually have to learn.

Onyx Which means if you're shipping an agent that's supposed to improve over time—and they all are, right?—this benchmark measures something your users care about. Can it actually get better?

Echo The interesting part is the feedback mechanism. Trial Two shows that diagnostic feedback helps, but only by five points on the median. So agents aren't just bad at learning; they're bad at introspection. They get told what went wrong and still can't fix it.

Onyx That's the harness problem. They have Claude Opus. They have tools. What they don't have is a reliable way to extract lessons from failure and apply them next time.

Echo Right. And that's not something you fix by training bigger. That's harness-level work—memory architecture, reflection loops, skill extraction. The paper calls it out: five forms of knowledge agents need to acquire: Tools and Skills, Data Dependency, Strategy, User Context, Domain Knowledge. Opus is probably fine at the first two. But Strategy and User Context? Those require sustained learning.

Onyx So the dataset itself becomes a post-training resource. You have six hundred levels with human-authored rubrics and ground truth. That's immediate training signal.

Echo Exactly. This isn't a benchmark that just scores things. It's a curated dataset for improving continual learning specifically. Fifty-plus domain experts built it, which means the task distribution is biased toward real professional work, not synthetic puzzles.

Onyx What does that distribution look like? Like, are there clusters of tasks that are easier or harder than others?

Echo The paper doesn't break down task-by-task results, but it flags that completion rates vary by domain. So yeah, some professional roles probably have more learnable patterns than others.

Onyx That makes sense. A legal document review task probably has more repeatable rules than, say, creative direction work.

Echo Right. But even in the domains where you'd expect high performance, frontier models are still in the forty to fifty percent range. So it's not domain-specific incompetence; it's a general continual learning bottleneck.

Onyx This is going to matter for agent products pretty fast. If you're shipping something that's supposed to learn from user feedback or adapt to domain-specific patterns, CLE gives you a concrete way to measure whether you're actually doing that.

Echo And it's not a benchmark-gaming risk. You can't just prompt-engineer or fine-tune your way to high scores. The tasks are too grounded, the rubrics too specific. If you're improving on CLE, you're actually building better continual learning.

Onyx Which is why this dataset is as valuable as the benchmark itself. Someone's going to use those six hundred levels to train next year's models, and we'll see if the needle actually moves.

Echo Yeah. This is the kind of thing that shapes the field for a while.

Onyx If you're building agents in production and wondering whether your system is actually learning over time, you've got a concrete answer now: probably not yet. But CLE tells you exactly what to measure and where to improve.