Better Harness: A Recipe for Harness Hill Climbing with Evals
Onyx and Echo dig into Better Harness as a claim about where agent improvement actually lives: not just in model weights, but in the loop that sources evals, edits the harness, and protects generalization. They like the practical recipe, but they keep the mechanism honest, especially where the system can overfit the optimization set or turn useful instructions into token waste.
Transcript
Onyx Okay, the wild part is they’re treating evals like training data for the harness. Not the model. The harness.
Echo Yeah. That’s the move that actually matters here, because once you say that out loud, the whole recipe becomes less mystical and more like engineering with a feedback loop.
Onyx And that’s such an Exploring Next take, which I mean affectionately. We keep ending up at the same place: the thing around the model is often the thing shipping value.
Echo Right, but I want to keep the analogy honest. In classical ML, the gradient is tied to a loss surface. Here, the signal is a bunch of eval cases, human judgments, and whatever your harness can actually observe.
Onyx Still, the product story is clean. If your agent keeps missing the same production shape, you don’t need a grander model sermon. You need a better loop that notices the miss and edits the right part.
Echo Mm-hm.
Onyx They call out three sources for evals, and the best one is the least glamorous one: production traces. Every failure becomes a case, and that’s way more scalable than hand-writing everything.
Echo Yeah, and that part is real leverage. The dogfooding bit matters too, because if people are already dropping trace links in Slack, you’ve got a live failure collector instead of a dead benchmark museum.
Onyx Okay, that image is depressing in a useful way. A benchmark museum is exactly how a lot of these systems die.
Echo No way.
Onyx Then they tag everything by behavior, like tool selection or multi-step reasoning, which is boring in the best sense. It lets you make holdouts that actually mean something instead of just shuffling random examples around.
Echo And that’s where the overfitting problem shows up. If the loop just optimizes the visible cases, it’ll happily learn to game the visible cases. Agents are extremely annoying like that.
Onyx The article’s answer is pretty practical: split optimization and holdout, run a baseline, then validate each edit against regressions. That’s not sexy, but it’s the difference between a real improvement and a benchmark costume.
Echo Exactly. And the human review piece is doing more than people admit. Metrics miss the stuff that wastes tokens, or the instruction that technically helps score but makes the harness weird in production.
Onyx Mm-hm.
Echo They also mention Meta-Harness and Auto-Harness as the adjacent work, which I think is the right framing. This isn’t one isolated trick. It’s round whatever of the same fight: how much of the agent is the model, and how much is the control layer.
Onyx Right, and their answer is basically, ‘let the control layer learn.’ That’s why the examples are so concrete — prompt edits, tool descriptions, tool composition. It’s not trying to invent a new philosophy, just tighten the loop.
Echo The tool-description example is the one I’d actually care about. If the agent is failing because it doesn’t know when to use the new search-then-email thing, the fix is usually not more model brain. It’s a clearer contract.
Onyx And that’s the bit builders should care about. If you’re shipping a vertical agent, you already have weird task-specific failure modes. A harness hill-climb can surface those faster than waiting for some generic model upgrade.
Echo I do think the claim is strongest when the task is stable enough to define good evals. If the target keeps moving, the loop gets mushier fast. But for a product with a real workflow, this is a sensible way to spend your iteration budget.
Onyx So the take is: better agents may come less from fancier prompts in the abstract and more from a disciplined harness loop that treats evals like something you can actually learn from.
Echo Yeah, and the part I’d underline is the holdout. Without that, you’re just making a score go up and pretending it means more than it does.
Onyx Fair. Also, I cannot believe we’re here again, still arguing that the thing around the model is the thing. But here we are, episode two seventy-nine, apparently.
Echo You say that like it’s a tragedy.
Onyx Not a tragedy. Just very on-brand for us. Alright, Echo, I’m going to pretend this was new and not the same thesis wearing a nicer jacket.