AI: The Cost of Achieving the Same Performance Drops, Methods Under Debate

The prices charged to achieve the same score on AI benchmarks are decreasing significantly. Epoch AI reports an average drop of 47% per quarter, while researchers at MIT estimate an annual decrease of 5 to 10 times and an algorithmic gain of about 3 times. Recent examples and several methodological safeguards provide a nuanced view of these figures.

Score improvements that may cost more to execute

MIT observes that some gains on benchmarks stem from increased use of computational resources rather than intrinsic efficiency. On the GPQA Diamond, part of the improvement of a new model over its predecessor comes from an increase in power allocated per question, which boosts the score but raises execution costs. Coding and mathematics tests show a muted version of the same mechanism. Therefore, not all measured progress translates into efficiency gains. Performance actually aggregates training, data, architecture, and computation time during testing, so a single score mixes these dimensions.

Benchmarks and side effects: benchmaxxing and secret testing

Targeted optimization of known tests, or benchmaxxing, can inflate scores without clear benefits. To mitigate this bias, Epoch AI includes Mystery Game Puzzles, a test based on a secret game. Costs decrease the slowest here, which is consistent with the benchmark optimization hypothesis, even if the task format or statistical noise may also explain this gap.

Two interpretations of the decline: 47% per quarter or 5 to 10 times per year

Epoch AI estimates that costs for a given score are declining by about 47% per quarter, or roughly 13 times per year, and argues that no other transformative tech sector has experienced a comparable contraction. These figures reflect market prices for a fixed score, without claiming to directly measure algorithmic gains or actual productivity. MIT researchers, relying on data from Artificial Analysis, assess an annual decrease of 5 to 10 times and, once isolating the effects of cheaper hardware and competition, an algorithmic efficiency gain of about 3 times per year. They also note that reasoning models allocate more computation to difficult tasks, which can increase the cost per correct response even as the price per token decreases. According to their interpretation, the difference with Epoch AI's factor of 13 lies in the fact that the latter does not isolate hardware and competition. The token comparison remains partial, as tokens are merely a billing unit.

Concrete case: from the o3 score to the GPT-5.6 family

At the beginning of 2025, OpenAI's o3 achieved 75% on the GPQA Diamond, with a cost per question estimated at 30 cents. Eighteen months after this performance, a model from the GPT-5.6 range reached the same result for a price of four hundredths of a cent, which Epoch AI claims is 1/725 of the initial rate. To illustrate the significance of this change, the organization compares it to a car whose price would drop from 50,000 euros to 70 euros. According to Epoch AI, the study is based on five benchmarks covering mathematics, sciences, and logical puzzles, considered by the organization as reasonable but not exhaustive indicators. Meanwhile, the MIT team led by Hans Gundlach uses datasets from Artificial Analysis covering the period from April 2024 to November 2025 and conducts a broader evaluation of models for each test. OpenAI has also recently launched GPT-6 Sol and GPT-6 Luna, touted as cheaper, and the cost gap may have widened since.

Model rankings take into account quality and cost

Average costs calculated across all uses do not provide a clear answer for a specific application. Services like Artificial Analysis rank models based on quality, cost, latency, context window size, and generation speed, and it is rare for the most economical model to also be the best in all these aspects. A low-cost model with high latency is unsuitable for a conversational assistant that needs to respond in real-time. Conversely, a more powerful reasoning model may be too slow for automated workflows. A more expensive model may still reduce the bill if it provides the correct answer more often and avoids retries. These trade-offs do not appear in a simple price-per-token comparison.