OpenAI: AI Boosts Code but Not Scientific Accuracy

OpenAI: AI Boosts Code but Not Scientific Accuracy

AI coding agents can modernize research software, but they cannot judge whether the science is correct.

A field report from OpenAI and academic partners shows that coding agents can update and accelerate aging research software. However, much of the work shifts from writing code to verifying results.

Many widely used research tools began as support code for a single paper. Small academic teams often wrote them without the time or resources for proper testing, maintenance, or optimization. The result is fragile software that remains critical for entire fields but requires constant repairs. A field report from OpenAI and academic partners suggests that AI coding agents could help bridge this gap.

The report documents eight case studies, primarily in biology, where research groups used coding agents such as Codex and Claude Code. The projects range from basic maintenance and targeted optimization to complete rewrites in modern programming languages.

Impressive Accelerations

Coding agents enabled accelerations of over 60 times.

One of the simplest projects involved modernizing cyvcf2, a Python library for reading genetic data. GPT-5.5 replaced its outdated build and installation process with a modern one.

The migration of MHCflurry was much more complex. MHCflurry is an immunology model that predicts which targets immune cells will recognize. Claude Code and Codex alternated between developer and reviewer roles while transferring about 10,000 lines of code from TensorFlow to PyTorch.

The rustar-aligner project was more ambitious. It rebuilt STAR from scratch in Rust. STAR maps sequencing reads from cells to corresponding locations in a genome. The original contains over 20,000 lines of C and C++ and is no longer actively maintained, even though it remains an integral part of many research pipelines.

To verify that the rewrite behaved like the original, the team tested both tools on 10,000 short sequencing reads from yeast cells. For single reads, rustar-aligner produced the same result as STAR in 99.815% of cases. For paired reads, the agreement rate was 99.883%.

The comparison covered more than just the mapped location in the genome. It also included several other key fields that both programs produce for each read. Neither tool mapped reads that the other failed to map.

RustQC offered the greatest acceleration by combining 15 distinct quality control tools into a single program. On a large dataset, the runtime dropped from 15 hours and 34 minutes to 14 minutes and 54 seconds, achieving an acceleration of over 60 times.

Another project, HelixForge, replaced a synthetic genomic data generation tool with a version running on GPUs. In a test using data from a donor and a section of genome of ten million base pairs, HelixForge completed the entire pipeline 59.6 times faster than BamSurgeon. The main computational step alone ran 98.6 times faster.

Speed of Code Does Not Guarantee Scientific Quality

Across the case studies, the agents accomplished well-defined tasks quickly but could not reliably judge whether their work was scientifically correct. Even when their code contained errors, the systems often presented it with total confidence.

"With coding agents, it's quite easy to go fast; for now, to go far in science, there is still a need for expert guidance, understanding, taste, and care," writes Brent Pedersen, developer of cyvcf2.

Philip Ewels, who led RustQC, describes the agents as "eloquent, convincing, and certainly wrong in ways that are easy to miss." He never allowed the models to judge the accuracy of their own work and instead built an independent testing framework.

The acceleration came from a series of small code changes rather than a single optimization.

The bayesm case study illustrates how these errors can be difficult to detect. Its rewrite in Rust performed between two and twenty times faster than the original, but the early versions of two advanced methods contained errors that were hard to spot from the output alone.

In one method, the agent reversed a key control parameter, causing the program to use the reciprocal of the expected values. A separate bug affected the calculation itself. The researchers only discovered it after conducting a detailed calibration test against thousands of synthetic datasets with known results.

A statistical calibration allowed them to detect a bug that earlier agreement tests had missed.

Another method, called HART, produced plausible results overall but still contained several flaws. These included unnecessarily costly calculations and a poorly scaled correction factor. Plausible test results alone could not establish that the code was correct.

An earlier attempt to transfer MHCflurry to PyTorch had failed in early 2025. Developer Sergey Feldman now attributes the failure to the models available at the time rather than the coding tools themselves. In his view, only the more recent generations of models have become reliable enough to handle much of this work autonomously.

Humans Define Tests While Agents Write Code

The projects followed a consistent division of labor. Humans defined the goals, success criteria, and validation methods, while the agents managed the implementation.

The hifiasm project shows how this worked in practice. Hifiasm assembles a complete genome from many short fragments. Before asking GPT-5.5 to optimize it, the researcher built a testing framework with separate training and validation datasets. The model then found changes that reduced runtime on real human genomic data by nearly 15%.

HI.SIM, a library for simulating genetic data, required even less human involvement. GPT-5.2 found ways to optimize individual parts of the program in a single pass. A second pass with a newer model found additional improvements. Together, the changes reduced runtime by about 31% without altering the output.

Low-Cost Rewrites Create a Maintenance Problem

The authors also provide rough estimates of potential savings. If agents could resolve between a quarter and half of all installation issues affecting research software, the research time saved on 100 packages would be worth between $600,000 and nearly $5 million. For NumPy alone, the report estimates that agents could save about 650 hours of maintenance work each year.

Long-term maintenance remains a major issue alongside validation and scientific accuracy. Low-cost rewrites could fragment user communities and further extend the already limited time of experienced maintainers.

Teams adopted different approaches to ownership and maintenance. Some changes were directly integrated into the original projects. Since STAR was no longer maintained, rustar-aligner was transferred to the research consortium scverse. The author of FastQC refused to replace the original tool with its rewrite in Rust. Instead, the team added the improvements they found to the original version in Java, which achieved the same tripling of speed.

The field report revisits completed projects and relies on testimonies from those involved. Its authors emphasize that the results do not come from a representative study. They still see the main bottleneck shifting from coding itself to validation, scientific review, and clear accountability for maintenance and future development.

The same pattern appears in software development outside of research. A study by METR found that true project maintainers would reject about half of the solutions that the SWE-bench Verified benchmark evaluates as successful.

A study on developer frustration with AI-generated code found a similar trade-off. The time saved in generating code can instead be spent revising it. The curl project shut down its bug bounty program after AI-generated vulnerability reports consumed maintainers' time without producing useful results.

The field report is part of OpenAI's broader effort in the scientific domain. The company has created a dedicated scientific team led by Kevin Weil, who expects 2026 to be for science what 2025 was for software engineering. In April, OpenAI introduced GPT-Rosalind, a model for life sciences research, and released a free life sciences plugin for Codex, which connects the models to over 50 public databases and biology tools.