Claude from Anthropic Generates 180M Evidence on 123 Libraries

Key Takeaways

1Claude generated 180 million lines of formal proofs on 123 open-source libraries

2Hundreds of bugs were detected, some of which had remained invisible for years

3The proofs are valid under strict conditions and produced at a high cost, with minimal human intervention

4Anthropic and Theorem aim to make this verification as common as standard testing, targeting glibc, OpenSSL, and cURL

Why it matters : AI-driven automation of formal verification could transform trust and security in software development.

Anthropic and Theorem have had Claude produce formal proofs on existing code, covering 123 libraries. This operation uncovered hundreds of bugs and fuels the ambition to mainstream formal verification through AI, despite strict constraints and high computational costs.

Limited Validity and High Computational Costs Announced

Anthropic clarifies that "formally verified" software is not free from errors. The generated proofs are only valid under strict conditions: a given processor architecture, single-threaded programs, and inputs less than 64 MB. In reality, the libraries in question operate on varied hardware and configurations. Each proof requires hundreds of millions of tokens, and Anthropic does not disclose the total cost of the project. Even when the complete proof fails, Anthropic states that the attempt helps uncover real bugs. While verifying a widely used PDF reader, the team identified as many security flaws in one week as in a typical year of reports for that software. In another case, Claude detected a logical error that had gone unnoticed for over ten years.

Results Achieved on 123 Open-Source Libraries

In just a few weeks, Claude generated approximately 180 million lines of proof code covering 123 open-source libraries, relying on test programs from Google’s OSS-Fuzz project. The model helped identify several hundred bugs that had not been discovered by traditional testing methods. During the proof production, Claude had the opportunity to make corrections to the identified defects, many of which only manifest in very specific cases, such as extremely large input files or unusual access modes, according to the researchers.

Le brief IA que lisent les pros

Le brief IA que les pros lisent chaque soir

Les 7 actus IA du jour, décryptées en 5 min. Gratuit.

Inclus dès l'inscription : notre sélection des meilleurs guides & comparatifs IA.

Choisissez votre rythme

Gratuit · Pas de spam · Désabonnement en 1 clic

Method Employed and Claimed Degree of Autonomy

Rather than producing new code, Claude translated crash tests into mathematical problems, analyzed the compiled machine code, and then generated detailed proofs. An independent verifier, Rocq, automatically checks these proofs. According to Anthropic, the procedure requires very little human review, and the researchers describe the proof generation as "fully autonomous." However, the tools and formal foundations were designed by engineers at Theorem. This setup was established by Anthropic's Frontier Red Team in collaboration with Theorem to apply Claude to the mathematical verification of existing code.

A Historically Costly Context for Formal Verification

Exhaustive formal verification has historically been very costly. For example, the proof of the small seL4 kernel required 20 person-years, which has limited this approach to extreme uses such as certain avionics software from Airbus. Anthropic's initiative addresses a central problem of AI-generated software while acknowledging strict practical limits.

Reusability of Proofs, Efficiency Gains, and Next Targets

A more limited test showed that the generated proofs are reusable. The Opus 5.5 model manages to resolve half of the proofs while using about 40% fewer tokens than Opus 5. Anthropic indicates that AI formal verification should soon become as accessible as traditional testing, and Theorem is already targeting critical components such as glibc, OpenSSL, and cURL. Anthropic aims to expand this method, presented as an additional lever for software automation.