COBOL and AI: Promising Migration, Persistent Obstacles

OpenAI announces advancements in translating COBOL to modern languages, while Anthropic highlights its tools. Researchers acknowledge progress but believe that safely replacing critical systems remains out of reach. Between lab results and real-world complexity, the gap persists.

For critical systems, the tolerated gap is often zero

Vadim Zaytsev believes that, in many cases, even very slight differences in behavior between the old and new systems are unacceptable. Experts argue that while AI can accelerate certain processes and reduce costs, partial automation is insufficient for such complex systems. Zaytsev reminds us that rewriting code does not equate to replacing a system, an operation that also involves human and organizational constraints. He emphasizes that these environments often combine multiple languages, data in different formats, technical integrations, and strong expectations from users and institutions. The volume of code can be considerable: he has already observed a system with 250 million lines of COBOL, which is not even the largest existing. He illustrates the risk of unaddressed remnants: even if an AI manages 99% of the code, the remaining 1% could represent a million scattered lines prone to hard-to-detect failures.

OpenAI's announcements: high percentages in controlled tests

OpenAI claims to have significantly improved automatic migration from legacy languages. The company aims to assist in maintaining and modernizing systems by translating COBOL into C++, Java, Kotlin, Python, or Rust, while preserving the behavior of the original software. OpenAI specifies that a migration is only considered successful if all tests are validated. In its evaluations, GPT-6.1 Sol achieves 87% behavior preservation with low reasoning effort and 98% with high effort, while GPT-6 Astra scores around 86% with low effort, GPT-6 Sol 21%, and GPT-5.6 Sol 5%. OpenAI presents these results as progress towards more accurate and affordable migrations and also mentions advancements in creating database software, including in COBOL. However, its spokesperson Oscar Haines acknowledges that legacy code remains far from a solved problem and indicates that the company is working to improve testing and its models' understanding of languages.

Anthropic promotes Claude Code, without published metrics

In February, Anthropic released a step-by-step guide for modernizing code and highlights Claude Code for migrating COBOL to modern languages like Java or Python. The company claims that the economics of COBOL modernization have evolved but has not provided performance indicators on the effectiveness of its agent for these migrations. Following this announcement, IBM's stock price fell. Anthropic did not respond in time for publication.

Researchers' diagnosis: benchmarks too far from the field

Researchers believe that AI tools have made progress in managing legacy code but are still far from enabling the safe replacement of critical systems. Arie van Deursen considers OpenAI's communication superficial and lacking in detail regarding the migration task, arguing that its benchmark evaluates simple, well-defined exercises close to cases encountered during training. He highlights the gap between maintaining and modernizing, with the former being more accessible, and believes that the chosen tasks are very distant from actual modernization.

Testing, documenting, verifying: the minimum conditions

For Alastair Donaldson, robust testing is essential. One method involves running the same workflows on both the old and new systems to verify that behavior remains identical. AI could also generate documentation of its translation to allow for review by specialists in the source language. Jacopo Mauro emphasizes formal verification as a complementary solution to certify code behavior, already used to control mathematical results produced by AI. However, its application to legacy code would require considerable effort. According to him, having automatic verification would be preferable to human oversight, but as long as these tools are not available, human verification remains essential.

Why COBOL still weighs heavily

Designed in 1959, COBOL remains widely present at the heart of many essential systems. According to IBM, it serves as the foundation for over 40% of online banking platforms, supports 80% of in-person credit card payments, and manages 95% of ATM withdrawals. This language is also used by multiple public agencies, including Social Security, the IRS, and most state agencies. An estimate puts the cost of inefficiencies related to COBOL at around $105 billion per year for the U.S. GDP. Replacing COBOL remains very difficult, similar to other legacy languages like Fortran, which is still used in scientific research, meteorology, and sensitive fields. The retirement of many experts and the scarcity of skills exacerbate the situation. While recent advancements exist on the AI side, tools have so far made little progress against these architectures. Alastair Donaldson points out that the difficulty is not limited to rewriting code and that a lack of understanding of how it works is common, as few people still master these languages. He adds that no one would use COBOL to start a new project today. This reality is evident even in agencies like the IRS, where COBOL systems are still in operation.