Brief IA
›

ARC-AGI-3: Cutting-Edge Models Stay Below 1%

Models & LLM·Tom Levy·

ARC-AGI-3: Cutting-Edge Models Stay Below 1%

ARC-AGI-3: Cutting-Edge Models Stay Below 1%
Key Takeaways
1Leading models evaluated on ARC-AGI-3 achieve less than 1%.
2At Oxford, participants assisted by models reached a maximum of 34.5% correct medical identifications, compared to 47% in the control group.
3According to Scale AI, the best model fully automated 20.8% of projects evaluated as of October 7, 2026.
Why it matters : a high score on one test does not guarantee success on another task or for an assisted user.

On ARC-AGI-3, launched on March 25, 2026, all evaluated state-of-the-art models score below 1%, while humans solve all environments. This assessment focuses on learning, reasoning, and adaptation in the face of the unknown. Other experiments document discrepancies between the success of an isolated model, rule application, and user assistance.

Users Do Not Retrieve Medical Scores

In Oxford, a trial involving 1,298 participants compared several ways to identify medical conditions. Researchers measured a 94.9% correct response rate when large language models worked alone. Participants assisted by these same tools did not exceed 34.5%, compared to 47% for the control group that conducted their own research.

The complete automation of real-world tasks is also subject to separate measurements. According to Scale AI, the top-ranked agents fully completed 2.5% of freelance projects evaluated at the launch of its remote work index. These were mainly sound or visual creations, with some work on data or writing. As of October 7, 2026, the leading model reached 20.8%.

The skills that users can rely on remain difficult to pinpoint and vary with updates. An emerging approach is to examine automatable tasks within a profession separately, rather than considering it as an indivisible whole.

Discrepancies Between Definition, Perception, and Reasoning

Systems designed by Google and OpenAI solved five of the six problems from the International Mathematical Olympiad. A few months later, however, chatbots suggested walking to drive a car to the garage, just 50 meters away. Andrej Karpathy, co-founder of OpenAI and former head of AI at Tesla, referred to these disjointed capabilities as "shattered intelligence" in July 2024. The phenomenon encompasses discrepancies between tasks, difficulties, and evaluation versus usage: one success hardly predicts the next.

An assessment conducted by researchers from MIT, Harvard, and the University of Chicago focused on seven models, incorporating concepts from poetry, psychology, and game theory. They reported about 94% correct definitions, but around 55% failures when it came to classifying cases related to the same concepts. For instance, GPT-4o correctly explained ABAB rhymes before producing a poem that did not adhere to this pattern. Unlike human skills, which tend to progress together, model capabilities do not always exhibit this coherence: knowing how to produce functional code does not guarantee the ability to count the words in a paragraph.

ClockBench measured another dissociation in September 2025: humans correctly read analog clocks 89.1% of the time, compared to 13.3% for Gemini 2.5 Pro, the best-rated model. This reading requires locating the hands, appreciating their angles, and then associating them with numbers. Subsequent calculations were correct when the time had been recognized accurately: the observed failure concerned perception.

La lettre du soirGratuit · 19 h

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

At Apple, researchers varied the difficulty of Tower of Hanoi, River Crossing, and Blocks World. Standard models outperformed reasoning models on easy puzzles; the comparison reversed at medium difficulty. On difficult problems, both categories failed, even with the provided solving algorithm. These results, associated with the Illusion of Thought assessment, are contested.

Several Explanations for Model Weaknesses

Reinforcement learning with verifiable rewards, or RLVR, became an important component of training in 2025. It relies on controllable responses, such as a mathematical result or code passing tests. Karpathy believes this method fosters particularly high capabilities in verifiable domains; in this explanation, progress is slower in literary creation, philosophy, or certain sciences, which lack automatic verifiers. He stated in 2025 that he no longer trusts benchmarks. Their verifiable exercises allow laboratories to train models on closely related tasks, with a possible score increase without equivalent overall gain.

Apple researchers associated with GSM-Symbolic propose another hypothesis: models might reproduce learned sequences rather than logical reasoning, which a small modification could then disrupt. With GSM-NoOp, they introduced unnecessary details into primary-level math problems. Models notably subtracted from the total of kiwis simply described as smaller than average. According to these researchers, adding seemingly relevant information led to a performance drop of up to 65% on top models.

Researchers from MIT, Harvard, and the University of Chicago invoke incompatible internal representations: a correct definition could coexist with an erroneous conception, without a requirement for coherence. They also argue that tests extrapolate understanding from a few correct responses, assuming errors as regular as those of humans, while models could succeed without understanding. Some researchers favor a more radical interpretation, that of separate skills without an underlying general intelligence. None of these explanations is definitive, and several may combine.

Testing Responses and Measuring Results by Category

One possible protocol involves modifying the wording of a real request or adding a detail without utility, then checking the stability of the response when the expected solution remains the same. Another involves applying a rule to new cases, through classification or example generation, rather than simply asking for its definition.

An average precision may conceal degradation on certain requests; thus, measuring results by category constitutes another proposed method. Failing categories can be directed towards an earlier version, with re-examination of this rule during each update. In areas without automatic control, where RLVR cannot apply, the proposed method remains a human verification.

Brief IA — L'actualité IA en français

L'essentiel de l'actualité de l'intelligence artificielle, décrypté et expliqué chaque jour.