A test reveals that code agents succeed in 26.4% of tasks

A test based on real user requests reshuffles the cards for coding agents. The best model only fully validates about one task out of four, and the ranking changes depending on the execution environment. Companies are encouraged to evaluate both the model and the framework together.

The ranking varies significantly depending on the execution framework

Researchers executed the same agent models in several environments that allow them to read, write, and execute code. Depending on the framework, Qwen3.8-27B achieves 75% or only 56% on large revisions. The highest-ranked model also changes with the environment. It is recommended to always specify the model-environment pair, and for companies, to test these two components in tandem. A METR study has already documented the gap between performance on test benches and actual adoption, with a code acceptance rate by project managers about 24 percentage points lower than scores on SWE-bench Verified. Environmental defects can also allow a task to be validated without actually solving it. The specific effect of sophisticated frameworks remains uncertain: a recent study found only minimal measurable gains, despite consumption of up to three times more tokens.

Majority of partial successes and the need for human correction

GPT-6 Astra covers on average about 70% of the requirements of a task, but the result is completely correct only in 26.4% of cases. Most of the time, a detail remains to be addressed, necessitating manual corrections. Even in the best configurations, GPT-6 Astra only completes 41% of multimodal tasks and just under 18% of deposit tasks. As soon as a mission combines multiple types of changes—layout, design, interaction, content—performance declines for all models, a situation that is common in real projects. Each of the ten models scores lower when three or four types of modifications are required. Increasing computing power brings little improvement: GLM-5.2 uses almost six times more tokens than GPT-6 Astra for simple adjustments and achieves significantly lower results. In terms of hierarchy, Claude Opus 5 and Qwen3.8-Max follow GPT-6 Astra, while Grok 4.6 brings up the rear. In heavier revisions, the smaller Qwen3.8-27B climbs to second place, without any overall advantage being attributed to it. Overall, these results mean that human intervention remains necessary for nearly three out of four tasks across both benchmark tracks.

Why older tests no longer help differentiate agents

Classic coding benchmarks are considered saturated: once capable of executing, verifying, and correcting their own code, models exceed 99% and the gaps become difficult to measure. On datasets like WebCompass-edit, agents achieve near perfection, while the new protocol leaves more than a 35 percentage point gap. Furthermore, environmental defects can artificially inflate results, and more elaborate frameworks provide little measurable benefit, despite token consumption being tripled.

What Co-Design Bench measures and its limitations

Co-Design Bench evaluates agents on requests derived from real coding sessions. Designers selected cases that required rephrasing or cancellation of changes, to focus the evaluation on difficulties "experienced" by the agents. Tasks range from small tweaks, such as extracting an amount from a receipt and updating all dependent totals, to complete overhauls. One of the multimodal tracks involves retrieving values from receipts, tables, or videos, and then consistently reporting this data in the associated views. However, the protocol maintains precise limits: each task receives only a single initial instruction, with no possibility for follow-up questions or corrections during execution, and judgments are made by AIs that can make mistakes.