Compare AI agent changes with Online Experiments in Agent Observability | Datadog
Datadog’s Online Experiments ties agent evals to classic A/B testing: you compare prompt/model/tool variants on real traffic, with quality, cost, latency, and errors in one view. Their internal GPT‑5.6 vs GPT‑6 Luna case shows how this surfaces a clear cost win but ambiguous reliability, forcing you to dig into traces before flipping a switch.
Transcript
Pippa Datadog basically just said, stop shipping AI agent changes on vibes. Run real experiments on live traffic instead. And they brought receipts.
Tyler This is their Online Experiments thing wired into Agent Observability, right? Classic A/B testing, but for prompts, models, tools… the whole agent.
Pippa And the central move is, your offline evals stay. They're just not the last word anymore.
Tyler Because offline, you've got a curated dataset and an LLM judge with a rubric, and that's only as good as the cases you remembered to include.
Pippa So online, they split real users across variants and score behavior in production. Quality, latency, cost, and errors, all measured together.
Tyler And randomize who sees which agent version, which is the part I actually care about. That's how you separate 'this prompt is better' from 'this week's tickets happened to be easy.'
Pippa They're not hand-waving it either. The Agent Observability evaluators plug straight into Datadog Experiments, and Feature Flags control the traffic split. You start with, say, ten percent on the new variant. No redeploy.
Tyler mm, feature flags.
Tyler The bit I liked is the metric pairing. They're explicit that your measures should match the decision. A revised search tool might improve evidence quality but add extra tool calls.
Pippa Or the cheaper model needs more retries.
Tyler Right. So they line up evaluation scores with duration, estimated cost, token usage, and workflow errors in one comparison view.
Pippa That's the product win for me. Most teams I talk to track 'did it 5xx' OR some quality score, never both. Here, same traffic slice… did it answer better, did it get slower, did it blow up costs, did it spam tools.
Tyler Okay, the Luna test, because that's where it stops being abstract. They ran their own Insights agent, the one that investigates agent behavior from traces. Half on GPT-5.6 Luna, half on GPT-6 Luna.
Pippa Luna 5.6 as control A, Luna 6 as variant B. And the primary metric was an LLM judge scoring investigations one to five.
Tyler With a rubric covering accuracy, completeness, whether the conclusions actually followed from the trace evidence… including when the agent admitted uncertainty.
Pippa right
Pippa Then duration, estimated cost, estimated tokens, and workflow errors came off the same traces as secondary metrics.
Tyler And Experiments computed relative changes and confidence intervals on all of it. They show the intervals, not just point estimates… which won me over a little, honestly.
Pippa The cost result is the LOUD one. Luna 6 came in about forty-four percent lower on estimated cost.
Tyler Forty-four? That's not noise.
Pippa The whole confidence interval stayed below zero. Roughly thirty-seven to fifty-one percent cheaper, end to end.
Tyler Quality, though. 3.84 versus 3.86 on that one-to-five scale. Those two numbers are basically sitting on top of each other.
Pippa And they say it outright, 'close averages alone do not establish equivalent quality.'
Tyler Which I appreciate. Then it gets murky. Best estimates were something like thirteen percent higher duration and fourteen percent more errors on Luna 6.
Pippa But the intervals ran from slightly better to meaningfully worse. So you can't say 'it's slower' or 'it's buggier' with confidence. Just… there might be a tradeoff.
Tyler Which is exactly what noisy production data looks like. The only thing that really cleared the bar was cost. Everything else was 'could be a wash, could be a regression.' So before anyone
Pippa I'd ship it anyway.
Tyler Of course you would. Forty-four percent cheaper does that to people. Which is why they read the traces first… and found Luna 6 sending invalid tool parameters more often than 5.6, usually fixing itself after a retry.
Pippa Fine, traces first. But the quality judge never caught that? It looked at overall investigation quality and tool use, it just didn't specifically say 'hey, you're burning extra calls on bad arguments.'
Tyler That's the subtle point. A single scalar quality score can hide operational pain. The experiment told them cost is clearly better, quality looks similar, reliability maybe worse… and the traces gave them the actual failure mode to fix or guardrail before flipping everything to Luna 6.
Pippa And who should care feels clear to me. If your agent drafts emails for five people, wiring all this up is probably overkill. But if you've got real production traffic and you're arguing about prompts, or whether to downgrade models to save money… this turns a hallway debate into an experiment.
Tyler It nudges teams toward treating agents like any other system behind a feature flag. You don't 'ship the new prompt.' You ship a variant behind Experiments, watch quality versus cost versus errors for a week, read some traces, then ramp it up or kill it.
Pippa If nothing else, it's a reminder that 'felt nicer in staging' is NOT a metric. Run the split, look at the intervals, then argue.
Tyler I'll take that over yet another 'we tuned the prompt and everything got better' blog post any day.
Pippa Same. Let's park it there before we start designing our own fake metric dashboards.