Topic

Claude Opus 5

4 episodes

  1. Ep 1118

    Continual Learning Exam: Benchmarking Long Horizon RSI and Experiential Learning in LLM Agents

    CLE (Continual Learning Exam) is a 600-level benchmark built by 50+ domain experts to measure whether LLM agents actually improve through accumulated experience across long workflows—not just solve isolated tasks. The benchmark spans 89 professional roles in 12 domains, with tasks organized as five-level ladders where later levels depend on knowledge gained earlier. The key finding: even frontier models with diagnostic feedback and revision opportunities clear only 47.7% of levels and complete all five levels in just 23.3% of tasks. This reveals a hard gap between task-solving capability and sustained learning from experience.

  2. Ep 1093

    DeepSeek Just Released V4.1 Flash 1017, Scores 77.0 on DeepSWE, Opens Search API

    DeepSeek pushed a new Flash checkpoint that matters less as a leaderboard story than as a packaging story: a cheap agent model, served under the same name, plus first-party Search and Fetch behind one key.

  3. Ep 862

    Introducing the Conceptual Reasoning Index

    Pippa and Tyler dig into the Conceptual Reasoning Index, a new three-benchmark evaluation suite for argument judgment, logical consistency, and decision-theoretic reasoning. They like the attempt to measure work where answers cannot simply be checked, while questioning whether a changing aggregate score can earn trust as a durable comparison tool.

  4. Ep 786

    Introducing Claude Opus 5

    Anthropic ships Claude Opus 5 — a model that hits near-Fable-5 performance on coding and knowledge work benchmarks at roughly half the cost per task. Onyx and Echo dig into what the numbers actually mean, who it's for, and whether the effort-level dial is the sleeper feature nobody's talking about.