What 50,000 Runs of a 5 Line Eval Taught Us
The VS Code team ran a simple 5-line eval 50,000 times to test AI models' efficiency in completing a basic task: writing a string to a file. The goal was to see how reliably models could finish the work and what kinds of failures show up. The eval, called 'say_hello,' asks the model to add 'HELLO' to a file named 'HELLO.txt.'
Transcript
Cody I've gotta say, I'm skeptical about the claims here. Running 50,000 runs of a 5-line eval seems like overkill for a simple task like writing to a file. What are they really testing here?
Justy I think it's actually kind of genius. By running this simple eval so many times, they're able to see how models handle even the most basic tasks. It's not just about the task itself, but about how the models approach it.
Cody But isn't this just a controlled environment? I mean, it's an empty workspace, a fixed prompt... it seems like a best-case scenario for the models. What about real-world complexity?
Justy That's a fair point, but I think the beauty of this eval is that it removes variables. By keeping it simple, they can isolate specific behaviors and see how models differ. And honestly, it's surprising how many models don't take the most direct path.
Cody Yeah, I was surprised by that. I mean, some models always take the direct path, while others add all these unnecessary steps. But what's the actual cost of that inefficiency?
Justy The article mentions that the extra steps translate directly into output token usage, which has an actual cost. And they show this chart with four clear bands of models based on their efficiency. It's eye-opening to see how much variation there is.
Cody Okay, I see what you're saying. So, what do you think about the claim that choosing a model that overthinks less can save time and money?
Justy I think that's a no-brainer. If you can choose a model that's more efficient for a task, that's a clear win. And the fact that they're providing this data to help developers make informed choices is huge.