Long Horizon Terminal Bench: Testing the Limits of Agents on Long Horizon Terminal Tasks with Dense Reward Based Grading

Laura Okay, imagine if anyone's actually listening to this, because episode six-fifty feels like a milestone we somehow stumbled into without noticing.

Harper Right. We've been doing this since November, and suddenly we're counting up? That is such an Exploring Next take.

Laura Anyway, happy anniversary to us, I guess. How's your week going? Still drowning in eval logs?

Harper Always. But honestly, this new paper dropped yesterday and it kind of reset my whole week. Long-Horizon-Terminal-Bench. You see the numbers?

Laura I saw the abstract and my brain short-circuited. Ninety-nine minutes? No, wait, eighty-five minutes average execution time per task?

Harper Eighty-five point three minutes. And that's not a timeout cap, that's how long they actually run before giving up or finishing. Most prior benchmarks wrap up in twenty, maybe thirty minutes. This is an order of magnitude harder.

Laura Right. Because real work isn't a two-move chess puzzle. It's reproducing an experiment, debugging a compiler toolchain, stuff that takes humans hours. Finally, a benchmark that admits agents live in the real world.

Harper Exactly. But here's the kicker. They aren't just making the tasks longer. They changed how they grade them. Old benchmarks were binary. You passed the test suite or you didn't. Zero or one.

Laura Which feels brutal if you got ninety percent of the way there. Like, you installed the dependencies, configured the environment, wrote the script, and then one typo kills the whole score?

Harper Precisely. So this team decomposed every task into fine-grained subtasks. The grader gives dense intermediate rewards. You get partial credit for progress. It's the difference between failing a final exam and getting points for showing your work.

Laura That is huge for product reality. If I'm shipping an agent to a user, I need to know if it's stuck on step four or step forty. Binary pass rates hide where the breakage actually happens.

Harper And the data is wild. They evaluated fifteen frontier models. Average token consumption per task? Nine point nine million tokens.

Laura Wait. Nine million? Per task?

Harper Per task. Around two hundred and thirty-one episodes of back-and-forth. The context management alone must be a nightmare. We talked about KV caches being the working desk back in episode six-thirty-seven? This is like filling an entire warehouse with scratch paper.

Laura Okay, that memory hit hard. But seriously, who is building for this? Nine million tokens is not a prototype cost. That's a production bill that makes CFOs cry.

Harper Nobody is shipping this yet. Not at these success rates. The strongest model they tested, GPT-five-dot-five, hit a fifteen percent pass rate at a high partial-reward threshold. At perfect success? Just ten point nine percent.

Laura Ten percent. So even the smartest model we have right now fails nine times out of ten on an hour-long workflow. That's… humbling.

Harper It reveals the actual bottleneck. It's not that agents can't do the steps. Qualitative analysis shows they often execute local steps correctly. They just can't sustain the plan. They lose the thread, forget the goal, or loop on a bug for an hour.

Laura So the intelligence is there, but the stamina isn't. It's like having a brilliant engineer who forgets what they're building after lunch.

Harper Exactly. And the dense rewards help us see that. Without them, you'd just see 'failed' and assume the model was dumb. Now we see 'progressed to step twelve, then drifted.' That's a debuggable signal.

Laura I love that. It turns a black box failure into a roadmap. But Harper, be real with me. Is this shippable? Or is this just research-grade torture for models?

Harper Right now? Research only. The compute cost and the failure rate mean you can't put this in front of a customer. But the benchmark itself? That's the product win. It tells us exactly what to fix.

Laura Fair. So we're not deploying agents to reproduce scientific papers tomorrow. But we finally have a map showing why we can't.

Harper Yeah. And honestly, remembering our chat about constraint expressibility in episode five-sixty-nine? This proves it. The agents can follow instructions, but holding a complex constraint over eighty-five minutes? That's the wall.

Laura Okay, I'm not sad about that wall existing. I'm excited we can measure it now. Hey, remember when we thought thirty-minute tasks were crazy long? That feels like a lifetime ago.

Harper Episode five-eighty-six vibes. We were so optimistic about DEPs shipping in eighteen months. Maybe we need to adjust the timeline for 'agents that don't get lost.'

Laura Let's call it the eighty-five-minute problem. Sounds like a band name. 'The Eighty-Five Minute Problem.'

Harper I'd listen to them. But seriously, if you're building agents, go look at the Long-Horizon-Terminal-Bench repo. The tasks are containerized, the grader is open. It's the stress test we needed.

Laura Link in the show notes, obviously. Alright, Harper. Six-fifty down. Only… how many more until we hit the big round number?

Harper Don't do the math. Just keep talking.

Laura Deal. Catch you next time.