ARC-AGI-3: The Failure of AIs in a Humanity Test

⚡

Key Takeaways

1On March 27, 2026, the new version ARC-AGI-3 was released to evaluate agentic AIs.

2AIs, despite their successes elsewhere, fail at ARC-AGI-3, with success rates below 1%.

3The test highlights reasoning abilities, without relying on pre-existing knowledge.

💡Why it matters — This failure underscores the current limitations of AIs in achieving human-like intelligence.

A New Evaluation for AIs

March 27, 2026, marks the release of the latest version of the ARC-AGI benchmark, titled ARC-AGI-3. This test aims to evaluate so-called "agentic" artificial intelligence systems, which are designed to act and learn in interactive environments. Although these models have demonstrated impressive performance on other benchmarks, they largely fail at this specific test.

History of the ARC-AGI Benchmarks

The previous tests, ARC-AGI-1 and ARC-AGI-2, were developed by French researcher François Chollet. They aimed to measure the ability of AI models to abstract and generalize from a few examples. ARC-AGI-1 presented simple puzzles for humans but complex ones for AI models. With ARC-AGI-2, launched in March 2025, the difficulty was increased with more complex tasks. While AI performance has improved over time, this progress now seems to be reaching its limits.

The Disappointing Results of ARC-AGI-3

In ARC-AGI-3, AI models are faced with interactive environments where they must act step by step without explicit instructions. This benchmark is designed so that 100% of the environments are solvable by humans, and untrained volunteers indeed succeed. In contrast, leading AI systems show success rates below 1%. For example, Gemini 3.1 Pro achieves only 0.37%, GPT-5.4 scores 0.26%, Claude Opus 4.6 stands at 0.25%, and Grok-4.20 fails to pass any tests.

Objectives and Methodology of ARC-AGI-3

ARC-AGI-3 consists of abstract mini-games played turn by turn. Agents must observe the state of the environment, choose an action, and evaluate the outcome before deciding on the next move. The test does not rely on world knowledge or language but on basic reasoning abilities, such as detecting patterns, manipulating objects, and anticipating the consequences of their actions.

The Definition of AGI According to the ARC Team

For the ARC team, artificial general intelligence (AGI) is defined as the ability of a system to acquire any human skill with the same efficiency as a human. Intelligence is not merely the sum of skills but resides in the speed and economy of learning new skills.

A Revealing Conclusion

ARC-AGI-3 serves as a thermometer to measure the gap between AI models and human intelligence. Despite technological advancements, all leading AIs fail this test, highlighting the persistent challenges in the field of artificial intelligence. However, it may just be a matter of time before significant progress is made.