Evals are the new PRD

Laura The strongest line here is that a P R D can describe an AI product, but an eval can actually catch it getting worse. That feels very real.

Harper Yeah. The post is basically saying the spec has to become executable because the same prompt does not reliably produce the same behavior. I buy that, with one important caveat.

Laura Mm-hm.

Harper An eval is not the whole P R D. It can encode acceptance criteria beautifully. It cannot decide whether a recipe feature is worth building, who it serves, or whether people will trust it.

Laura Right, but that is still a pretty meaningful job shift. Instead of handing engineering “helpful and concise,” the P M owns a testable picture of good.

Harper And Braintrust’s recipe-video example is unusually clean. Check that ingredients come before steps with a rubric. Match transcript ingredients against the recipe deterministically. Then judge whether the instructions are short and scannable.

Laura Exactly.

Harper Those are three different measurement problems, which is the useful part. A string match is cheap and reliable. An A I judge for scannability needs calibration. Pretending those are equally trustworthy is how teams end up worshipping a dashboard.

Laura Also, every old P R D has that one majestic Google Doc nobody opens after launch. It becomes digital sediment.

Harper Okay, that’s good.

Laura It is! The eval suite at least has the decency to fail in C I and make itself everyone’s problem. Very democratic.

Harper Sure.

Laura The part I’m most sold on is the flywheel. Production traces show where users got a bad answer. Those failures become cases. The next candidate has to clear the expanded suite before it ships.

Harper That is the same control-layer story we keep tripping over. The model is only one moving piece. The durable asset is the loop that turns observed failures into constraints on the next release.

Laura And it gives product people something concrete to own. The post’s weekly cadence is almost aggressively practical: review bad traces, turn a handful into cases, compare the candidate, then ship or do not.

Harper Right, right.

Laura That changes something for teams with real traffic. A thumbs-down, an edited answer, or somebody rephrasing the same request three times is not just support noise. It can become evidence about what the system missed.

Harper But the post gets a little too smooth when it says every failure should feed the suite. Production data is messy. A rephrase might mean the model failed, or it might mean the person changed their mind halfway through.

Laura Yeah, fair. Someone has to curate that signal, especially before a weak proxy turns into a hard release gate. This is not “collect more logs and receive quality.”

Harper Mm-hm.

Harper It also lands directly in our old judge-reliability fight. The architecture-sufficiency camp says structured judges can generalize. The task-specificity camp says a judge that behaves on clean tool calls may fall apart on ambiguous production work. That debate is absolutely not settled.

Laura No, and the article does acknowledge it with the human-aligned judge category. For subjective dimensions, people still provide the ground truth and the judge only approximates it. That is the honest version.

Harper The failure mode is Goodhart, obviously. “Make this number go up” is great engineering direction only if the number remains attached to task completion, satisfaction, or retention. Otherwise the team learns to impress its own grader.

Laura Which is why I’d frame the actual takeaway a little differently than the headline. Evals are the new acceptance criteria, and the best teams make them part of product discovery instead of a pre-launch ritual.

Harper Yeah. A living suite beats vibes. It just does not absolve anyone from deciding what good means in the first place.

Laura That’s episode two sixty-six, Harper. Please do not let your eval suite become a very confident little bureaucracy.