Agentic Testing: Where Agents Fit in the E2E Testing Stack

Justy Okay, so Slack just ran two hundred plus agent-driven E2E tests and published the whole thing, and I think this is the clearest explanation I've read yet of where agents actually fit in testing.

Cody Yeah?

Justy They didn't try to crown agents as the new testing paradigm. They measured three different approaches—MCP, CLI, and generated code—on two workflows, and the results are… actually honest.

Cody Honest how?

Justy They show where each one breaks. MCP is the most reliable but slow. Generated tests are fast but fragile on complexity. And they're transparent about cost—fifteen to thirty bucks per run. That's real.

Cody Mm-hm.

Justy But the actual insight isn't about speed or cost. It's the distinction they lead with: traditional tests enforce a journey. Agents verify a goal.

Cody That's the whole thing, yeah. You're testing whether the user can accomplish what they set out to do, not whether they took a specific path to get there. The agent tries different valid sequences and still lands on the same outcome.

Justy Right. And they measured this—only about twenty percent of runs followed the exact same action sequence. Same final state, totally different paths.

Cody That's actually interesting because it means the agent is exploring the UI surface, not following a script.

Justy Exactly. So the first thing to understand is the execution models. They tested three: agent plus Playwright MCP, agent plus Playwright CLI, and then a separate path where the agent generates deterministic Playwright code upfront.

Cody The difference being what?

Justy MCP gives the agent a persistent view of the browser state. The agent sees DOM snapshots and logs and decides the next move. CLI is step-by-step—the agent runs a command, sees the result, then decides what's next. Generated code is pre-written, deterministic, just runs like a normal test.

Cody Got it. And the reliability numbers?

Justy MCP crushed it on simple flows—zero percent failures on thread reply. On the harder search discovery flow, it hit about twelve percent. CLI was rougher. Twelve to twenty percent failure rates, mostly timing and session stuff. And generated tests…

Justy …eight percent on simple, but forty-eight percent on complex. They were dying late in the flow.

Cody So the reliability gap widens as the flow gets harder. That tells you something about state management.

Justy Yeah, and they actually call this out. MCP keeps a live, stable view. CLI is rebuilding state from snapshots at each step. Small timing inconsistencies compound over a longer flow.

Cody Right. And there's the in-session context thing too—the agent in MCP is probably reusing successful interactions from earlier in the same run, while CLI feels like starting from scratch each time.

Justy Speed numbers?

Cody Generated tests were fastest overall, about three minutes including the upfront generation cost. But raw execution per test run was like thirty to forty-five seconds. MCP averaged five to eight minutes. CLI nine to eleven. So deterministic is way faster.

Justy But you're paying the agent cost every run with MCP and CLI, while generated tests pay it once.

Cody Exactly. In CI, that one-time cost becomes nothing. You run the test fifty times, the generation is amortized away. Agent-driven workflows pay the full cost every single execution.

Justy Which brings us to cost. Fifteen to thirty bucks per run. They broke down token usage too—MCP was more efficient than CLI or code generation. The execution model mattered more than which model you picked.

Cody So you can't just swap in a cheaper model and solve the cost problem. The architecture of how you're running the agent is the load-bearing part.

Justy Right. And here's the actual insight—and I think this is why I liked this piece so much. They don't try to claim agents are the new testing paradigm. They say agents are a new layer. Complementary.

Cody Mm-hm.

Justy You still need deterministic tests for regression and CI speed. Agents are for exploratory validation. Catching UI flexibility, state variability, weird edge cases where the user can reach the goal through a path the original test writer didn't think of.

Cody That's actually the right way to think about it. Deterministic tests are brittle by design—they enforce a specific sequence. That's a feature for regression. Agents are flexible by design. They adapt. That's a feature for exploration.

Justy And they're expensive and slow, so you don't run them on every commit. You run them periodically or before major releases. Different layer, different purpose.

Cody The stack is now unit tests, traditional E2E for regression, and agentic E2E for exploratory validation. That's a coherent story.

Cody Fair. It's solid work.

Justy Yeah. This one's the kind of piece that lands because it's not trying to sell you on agents—it's trying to help you use them right.