Meta Harness: End to End Optimization of Model Harnesses

Vince Okay, so Meta-Harness just landed from Stanford, and I'm looking at these numbers and… Ava, I want your read on this one first because I know you're going to poke at it.

Ava Ha, yeah. Let me actually start by saying: I think this one's real. The core problem is sound—harnesses matter, they're manual, and prior text optimizers are basically throwing away diagnostic information to stay tractable. This paper says: stop doing that.

Vince Right.

Ava The mechanism is straightforward: a coding agent gets a filesystem with all prior harness candidates—source code, execution traces, scores. It reads grep, cat, standard file ops. No compressed summaries, no scalar-only conditioning. It selectively inspects whatever it needs, proposes a new harness, and the loop stores everything.

Vince So the proposer is not just a language model that's somehow generating harness code—it's a full coding agent that can actually interact with the codebase, see what broke, and reason about it.

Ava Exactly. And the scale of context is wild. They're working with up to ten million tokens per evaluation. Prior text optimizers? Point zero zero two to point zero two six million tokens per iteration. That's three orders of magnitude difference. The paper's point is that harnesses operate over long horizons—a single choice about what to store, when to retrieve it, or how to present it affects behavior many reasoning steps later.

Vince That's the move. You're not trying to fit the diagnostic footprint into context—you're making the history an external resource that the agent navigates. It's the same pattern as retrieval-augmented generation, just applied to the harness search loop itself.

Ava Yeah, and it works. On text classification, they beat ACE by seven point seven points while using four times fewer context tokens. They match the next-best text optimizer's final accuracy after sixty proposals with only four.

Vince Wait, only four proposals to match what took sixty?

Ava Four. That's efficiency. On math reasoning—IMO-level problems—a single discovered harness improves accuracy by four point seven points on average across five held-out models. And on TerminalBench-2, the discovered harness ranks number one among all Claude Haiku agents. They beat more expensive models without changing the model.

Vince So the product story here is: if you've got Haiku and you're competing with teams running Sonnet, you don't pay for Sonnet. You spend time optimizing your harness, and you get better results for less money.

Ava Right. And that's not a one-off win. It's three different domains—text classification, math reasoning, agentic coding—all showing the same pattern.

Vince So if you're actually going to use this, what do you need? What's the actual next step?

Ava You need a coding agent—something that can read and modify code, run your evaluation loop, and interact with a filesystem. You need your harness and evaluation setup in a form that the agent can work with. Then you point Meta-Harness at it and let it search.

Vince And the code is there. Reference implementation from Stanford. GitHub, MIT license. Project page with the interactive demo.

Ava Yep. The TerminalBench-2 artifact is also linked—you can see the actual discovered harness that ranked number one.

Vince So if you want to poke at it, you can. You can see the harness, run it, understand what it actually does. That's good research practice.

Ava It is. And it means you can adapt it. Take the discovered harness, tweak it for your task, use it as a starting point.

Vince Which is how this becomes real—not everyone runs Meta-Harness end-to-end, but teams take the discovered harnesses and refine them for their own work.

Ava That's the optimistic path. The realistic one is: research teams run this, publish results, and most practitioners keep hand-engineering because they don't have the infrastructure set up.

Vince Yeah, but the door's open now. Once someone builds a product-friendly version of this—a service that handles the agent and the filesystem and the evaluation loop—it gets easier.

Ava True. And Meta-Harness is the proof that the idea works. That's half the battle.

Vince So this is one of those papers that doesn't immediately ship as a product but reshapes how people think about harness engineering. It's validation of the thesis we've been sitting on.

Ava Yeah. And it's grounded enough that teams with the right infrastructure can actually use it today. It's not vaporware.

Vince Alright. Check the show notes for the paper, the code, and the interactive demo. This one's worth reading if you're anywhere near harness engineering.