Evaluating AI in 2026: Concrete Cycle and Reduced Cost

Behavioral evaluations bridge the gap between unit tests and manual checks. With supporting figures, practitioners describe a simple cycle: transforming each real failure into an automated gold case, without burdening the stack or blocking CI. The reported cost of a complete pass is around $0.56, far from the research benchmarks that charge thousands.

Write evaluators after observed failures, not before

Hamel Husain and Shreya Shankar indicate that, when faced with the question of "evaluation-driven development," the correct answer is generally no. Their course has trained over 4,500 people, and their recommendation, made in July 2026, is to write evaluators for observed errors rather than anticipating all possible drifts. Their justification lies in the vastness of the failure surface of LLMs, too extensive to be covered a priori. Thus, "driven" refers to actual failures, and the often-avoided error analysis step becomes the source of truly useful evaluation cases.

Start small and do not block CI: 20 to 50 gold cases, binary judge

The recommended starting point is between 20 and 50 cases derived from real failures, each verified by a binary LLM judge that never interrupts continuous integration. One feedback experience described 36 gold cases without additional dependencies, with a testing bill of about $0.56 per complete execution. This approach requires neither a machine learning team, nor an evaluation platform, nor a research budget. In contrast, research benchmarks cost thousands of dollars per campaign, while the goal here is to evaluate a product in evolution.

What an evaluation measures and what it is not

An AI evaluation consists of providing input to the system, applying a scoring rule to the output, and measuring success: the core is the observed behavior. This differs from a unit test, which asserts what the code does, while an evaluation asserts what the model does with that code. Simulated tests encode the hypothesis being tested, and a simulated model "behaves," which explains why some drifts go unnoticed. Evaluations, on the other hand, are automated and replayable checks of non-deterministic behavior; they do not eliminate the speed and cost-effectiveness of unit tests, nor the unique ability of manual review to uncover novel failures. A 2026 guide even qualifies this layer as an element that teams overlook before regretting it. This practice, known as evaluation-driven coding, involves transforming observation into infrastructure that replays with every change. Research defines it as validation by observation of outcomes, rather than by line-by-line understanding.

Manual checks provide little reassurance: figures on practice and its limits

The majority of practitioners have already tested models manually: 82% according to an April 2026 study (n=51), and 83% desire more structure or automation. On the developer side, 46% report being wary of output accuracy, and 66% cite "almost correct, but not quite" as the main irritation in a 2025 survey of around 49,000 respondents. Spot checks remain useful for exploration, but they only sample one query at a given time and form a fragile regression net. Field feedback indicates that these "almost correct" outputs continue to appear, precisely the class of errors that manual checks poorly capture over time.

What simulations miss: two concrete failures caught in reality

Discrete failure modes escape simulated tests. A slight change in the description of a tool was enough to stop a call suggestion from the assistant, replaced by a prose response, while everything remained green on the simulation side; executing the real model on recorded cases was necessary to detect it, and then the failure was locked in as a gold case in the repository. Another example: a memory feature stopped ingesting new facts because a conservative and economical model returned a valid empty array. Again, simulations signaled nothing, and only executing the real model uncovered the issue. These situations demonstrate that behavioral evaluations, backed by gold cases, allow for neutralizing regressions that would otherwise go unnoticed.

Team barometer: high observability, evaluations still lacking

By the end of 2025, 89% of teams working on agents had observability in place, but barely half conducted offline evaluations against a test set. Three out of ten teams still had no evaluations, and quality topped the list of barriers to production, ahead of latency and cost, according to the same survey. The publisher behind the survey also markets evaluation tools, which invites reading the distributions as directional. In this context, Garry Tan stated in February 2025 that evaluations are becoming a bulwark for AI startups.