AI Evaluations: A Financial Sinkhole for Developers

AI Evaluations: A Costly Bottleneck

The evaluation of artificial intelligences has reached a cost threshold that profoundly changes who can engage in it. The Holistic Agent Leaderboard (HAL) recently spent around $40,000 to run 21,730 agent deployments across nine models and nine benchmarks. A single run of GAIA on a state-of-the-art model can cost up to $2,829 before any caching optimizations. Research conducted by Exgentic revealed a cost variation of 33 times on identical tasks, identifying the choice of infrastructure as a key cost determinant. Furthermore, the UK-AISI has recently expanded agentic steps to millions to study inference time computation. In the realm of scientific machine learning, The Well requires about 960 H100 hours to evaluate a new architecture and 3,840 H100 hours for a full sweep of four benchmarks. While compression techniques have been proposed for static benchmarks, new agent benchmarks are noisy, sensitive to infrastructure, and only partially compressible. Loop training benchmarks are inherently costly, and when seeking to add reliability to these evaluations, repeated executions further multiply the cost.

Reducing Costs of Static LLM Benchmarks

The cost problem began before the emergence of agents. When Stanford's CRFM released HELM in 2022, the costs per model indicated in the paper ranged from $85 for OpenAI's code-cushman-001 model to $10,926 for AI21's J1-Jumbo (178B), and from 540 to 4,200 GPU hours for open models, with BLOOM (176B) and OPT (175B) at the top of the scale. Perlitz et al. (2023) reaffirmed the broader cost pattern of HELM, and IBM's research noted that running Granite-13B through HELM "can consume up to 1,000 GPU hours." Of the 30 models and 42 scenarios in HELM, the total reported costs and GPU computation amounted to approximately $100,000.

Another striking observation comes from Perlitz et al.'s analysis of EleutherAI's Pythia checkpoints: developers pay for evaluation repeatedly during model development. Pythia released 154 checkpoints for each of the 16 models covering eight sizes, totaling 2,464 checkpoints if each model checkpoint is counted separately, allowing the community to study training dynamics. Running the LM Evaluation Harness on all these checkpoints turns evaluation into a multiplier on training: Perlitz et al. (2024) noted that evaluation costs "can even exceed those of pre-training when evaluating checkpoints." For smaller models, evaluation becomes the dominant cost element throughout the development cycle. As we increase inference time computation, we also increase evaluation costs.

Perlitz et al. then asked how many HELM rankings were actually carried. The result was striking: a reduction of 100 to 200 times in computation preserved almost the same order, with even larger reductions still useful for coarse grouping in the document's tiered analysis. Flash-HELM transformed this discovery into a coarse-to-fine procedure: first running inexpensive evaluations, then spending high-resolution computational resources only on the best candidates. Much of HELM's computation confirmed rankings that the field could have inferred much more easily.

Other work has reached the same conclusion from different angles. tinyBenchmarks compressed MMLU from 14,000 items to 100 anchor items with about 2% error using Item Response Theory. The Open LLM Leaderboard went from 29,000 examples to 180. Anchor Points showed that as few as 1 to 30 examples could rank 87 language model/prompt pairs on GLUE, and others followed, reducing dataset sizes by 90%. Static benchmarks had a weakness that could be exploited: differences between models often concentrate in a small subset of items, allowing rankings to survive aggressive subsampling.

This trick diminished significantly once benchmarks transitioned from static predictions to agents.

Increased Complexity of Agent Evaluations

A very interesting public accounting of agent evaluation comes from the Holistic Agent Leaderboard (Kapoor et al., ICLR 2026). HAL runs standardized agent harnesses across nine benchmarks covering programming, web navigation, scientific tasks, and customer service, with shared infrastructures and centralized cost tracking. The main cost: $40,000 for 21,730 deployments across nine models and nine benchmarks.

Behind this aggregate, the cost of a single benchmark run varies by four orders of magnitude across HAL tasks, and by three orders within certain individual benchmarks.

Behind these figures lies a brutal pricing fact. Claude Opus 4.1 charges $15 per million input tokens and $75 per million output tokens. Gemini 2.0 Flash charges $0.10 and $0.40, a variation of two orders of magnitude on input alone. Agent benchmarks rarely evaluate "the model" in isolation. They evaluate a product of model × infrastructure × token budget, and small infrastructure choices can multiply costs by 10 times.

Worse still, a higher budget does not always buy better results. On Online Mind2Web, using the browser with Claude Sonnet 4 cost $1,577 for 40% accuracy. SeeAct with GPT-5 Medium achieved 42% for $171. The HAL paper notes "a 9-fold difference in cost despite only a 2 percentage point difference in accuracy." On GAIA, a HAL generalist with o3 Medium cost $2,828 for 28.5% accuracy, while another agent achieved 57.6% for $1,686.

The tool from the static era should have helped, but it only did to a certain extent. Ndzomga's average difficulty filter, which selects tasks with historical success rates of 30 to 70%, achieves a reduction of 2 to 3.5 times while preserving rank fidelity under infrastructure and temporal changes. This is useful, but it still falls far short of the 100 to 200 times gains available for static benchmarks. When each item is a multi-turn deployment with its own variance, the inevitably long single-question trajectory becomes the costly object.

Training and Evaluation: An Asymmetrical Relationship

Some benchmarks completely escape the API cost problem because their evaluation protocol trains models from scratch.

The Well provides a very interesting example of this. It aggregates 16 datasets of scientific machine learning covering biological systems, fluid dynamics, magnetohydrodynamics, supernova explosions, viscoelastic instability, and active matter, totaling 15 TB. Using the grid of 16 datasets from the paper, the protocol leaves little room for economy: training each benchmark model for 12 hours on an H100, trying five learning rates per pair (model, dataset), repeating across four architectures and 16 datasets. This grid sweep consumes 3,840 H100 hours, or about $9,600 based on the conversion assumptions below. A new architecture costs around 960 H100 hours, or about $2,400.

Training a single neural operator may require a single run of 12 hours H100, while evaluating it across the benchmark requires 80 of these trainings. This asymmetry is what makes The Well significant. In this corner of machine learning, evaluation computation exceeds training computation by about two orders of magnitude, reversing the traditional mental model of deep learning.

The same pattern repeats across SciML. PDEBench covers 11 families of PDEs and reports epoch time tables across datasets and model families, but a clean dollar figure per architecture depends on the chosen training protocol and hardware. MLE-Bench (OpenAI) sits between agent and training regimes. Each agent attempt in one of the 75 Kaggle competitions lasts 24 hours on a single A10 GPU, leading to real machine learning pipelines. The paper is explicit: "A single run of our main 24-hour competition attempt setup requires 24 hours × 75 competitions = 1,800 GPU hours of computation," plus o1-preview consuming 127.5M input tokens and 15M output tokens per seed. At $1.50 per hour for A10, the base GPU cost alone is $2,700; adding the use of the o1-preview API brings a single seed run to about $5,500. Thus, three seeds × six models would reach nearly $100,000 before any additional scoring or retry costs.

METR's RE-Bench limits each of the seven research engineering environments to 8 hours on 1 to 6 H100. A single pass through the set is therefore 56 to 336 H100 hours before adding repeated attempts, multiple seeds, or multiple agents; the human reference, with 71 expert attempts, drives the implicit budget much further. Because the benchmark gives agents and humans the same real-time computation, a real-time training process fixes the base cost. A token budget no longer limits it from above.

ResearchGym (ICLR 2026) ensures that the agent conducts real machine learning research. Five test tasks (39 subtasks) drawn from ACL, ICLR, and ICML papers, including ACL Highlights, ICML Spotlight, ICLR Spotlight, and ICLR Oral categories, with proposed methods retained. The budget is tight: $10 in API plus 12 to 24 hours on a single GPU under 24 GB per task. A complete pass (5 tasks × 24h × 3 seeds) consumes about 360 GPU hours per agent.

The cost table becomes brutal in PaperBench. Twenty ICML 2024 Spotlight or Oral papers must be replicated from scratch, scored according to rubric trees with 8,316 leaf criteria. Each deployment uses an A10 GPU for 12 hours, and the calculations per paper are straightforward:

• $400 in API per o1 IterativeAgent deployment, multiplied by 20 papers, totaling about $8,000 per evaluation.

Scoring costs $66 per paper with the o3-mini judge, or $1,320 for the entire benchmark. Using o1 as a judge would raise the scoring to about $830 per paper.