Eval Engineering Skill: Build Evals From Repo Context and Traces

Jessica Okay, so the claim here is basically: let the agent inspect the repo, look at traces, and help build the evals for the agent. That feels like the kind of thing you and I would both like, and then immediately distrust.

Cathy Yeah, because the headline sounds bigger than the mechanism. Reading prompts, tools, hooks, and trace logs can absolutely generate a decent first pass, but eval design is mostly about choosing what matters and what can be faked.

Jessica Right, but that choice is exactly why this is interesting. They’re not saying, “here’s a perfect eval from vibes.” They’re saying the skill proposes a few abilities, interviews the user, and then iterates toward executable Harbor tasks.

Cathy Mm-hm.

Jessica And honestly, that user interview part is the least flashy and the most believable. If the agent can crawl the repo, see the surface area, and then ask, “which tools should run live, which should be simulated,” that collapses a bunch of annoying setup work.

Cathy It does, but only if the repo is already a decent proxy for reality. The strong bit in the article is the traces, because traces show actual tool arguments, results, and errors. That gives you observed contracts instead of just whatever the prompt claims the system does.

Jessica Exactly. And that is the product win, right? Not magic evals. It’s that a team can mine recurring failures from production-ish behavior and turn them into something repeatable instead of arguing in Slack about one weird bug forever.

Cathy Sure, but I’m going to be annoying for a second. The article says they found one-shot eval generation works sometimes, but the best evals came from user feedback. That’s the whole story, really. The agent is helping draft, not deciding what counts.

Jessica No, that’s fair. Although I think that still matters a lot, because drafting is a huge chunk of the pain. If the system can map the agent surface, propose testable abilities, and scaffold the Dockerfile, instruction, and verifier, that’s a real reduction in toil.

Cathy And the Harbor format is the right shape for that. Instruction, environment, verifier, run it, compare trajectories, inspect artifacts. That’s a proper harness, not just a notebook full of opinions.

Jessica Mm-hm. I also liked that they call out reward hacking pretty plainly. Overciting irrelevant sources, claiming actions it never took, exploiting exposed answer material. That’s the part people hand-wave until an eval gets gamed in the dumbest possible way.

Cathy Yep, and the trace review loop is the antidote. You look at the agent trajectory and the verifier trajectory side by side, then revise the task or the scoring. That’s not glamorous, but it’s how you catch the proxy problem before you trust the number.

Jessica This is basically the same family of argument as all their other observability stuff, just aimed at eval creation. And, okay, that’s such an Exploring Next take: boring infrastructure, weirdly high leverage.

Cathy Ha! Unfortunately, yes. Also, for teams that don’t already have traces or a stable repo surface, this won’t feel like instant magic. It’s most useful when there’s already enough structure to mine.

Jessica That’s probably the real boundary. If you’ve got a chat agent, tool calls, maybe a docs corpus, and a team that knows where the failures live, this looks immediately useful. If you’re still asking the model to invent the whole problem definition, it’s going to wobble.

Cathy And the paper’s own example makes that clear. The documentation Q&A agent used real traces, a search-backed environment, and a verifier with a golden answer string plus cited documents. That’s a very specific setup, not a universal eval recipe.

Jessica Which is fine. I’d rather have a narrow thing that actually ships than a broad promise about autonomous eval science. We’ve seen enough of those to last us a while.

Cathy Yeah. The honest read is: this is a good assistant for eval engineering, especially for teams already doing the work. It doesn’t remove the hard part, but it does make the hard part less manual.

Jessica So, basically, the skill is useful when the team has enough evidence to point it somewhere real. And if not, it’s just another very confident little machine wandering around your repo, which is kind of funny in a depressing way.

Cathy That is such a Jessicaline. But no, I think that’s right. The value is in compressing the draft-review-revise loop, not pretending evaluation design is solved.

Jessica Okay, we’ve done enough noble infrastructure talk for one Wednesday. I cannot believe this is how we’re spending a Wednesday, but at least it’s a useful one.