OpenAI: Coding Agents Revolutionize Scientific Research

OpenAI and the Optimization of Scientific Software

OpenAI recently published a report detailing the impact of coding agents on eight scientific computing projects. These projects demonstrated how these agents can significantly reduce execution times. The report highlights the use of Codex in five projects, while three others combined Codex with other tools. It is essential to note that this report comes from a provider and is based on case studies provided by the contributors themselves.

Despite this, the report offers valuable insights into the maintenance challenges of research software. Often developed to accompany a single paper, this software is created by small academic teams without the support of dedicated engineers, leading to an accumulation of technical debt that few can afford to repay.

The OpenAI report claims that coding agents can address this technical debt. The eight projects covered in the report span various fields such as genomics, immunology, statistics, and RNA sequencing.

Task Categories of Agents

Coding agents have been involved in three main types of tasks: cleaning up packaging and build systems, optimizing performance on existing code, and complete porting of languages or backends.

• In the case of cyvcf2, a Python library for reading genomic variant files, an outdated build and packaging system was replaced with a more modern and unified process. Brent Pedersen, a contributor, emphasizes that while agents allow for rapid execution, scientific advancement still requires human expertise and careful attention.

• HI.SIM, a DNA sequencing read simulator, benefited from two autonomous optimizations that reduced execution time by 31% without altering the results. Andrew Ho, who does not consider himself a genomics specialist or a C programmer, described the outcome as "magical" from the end-user's perspective, having previously lost time due to performance bugs he could not resolve on his own.

• For Hifiasm, used in genome assembly from PacBio HiFi reads, a 25% reduction was achieved on the optimization target, and about 15% on separate human sequencing data. Suyash Shringarpure, a contributor, noted that the agent could establish its own reference structure and propose candidates independently, although human guidance remains necessary to avoid repeated failure modes.

• MHCflurry, which predicts protein fragments presented to T cells, migrated its backend from TensorFlow/Keras to PyTorch while maintaining compatibility with previously published model weights. Contributors Alex Rubinsteyn, Sergey Feldman, and Timothy O'Donnell described this change as "unglamorous and labor-intensive maintenance" essential for the survival of open-source scientific projects.

• bayesm-rs, a Rust port of the statistical models from the R package bayesm, matched the estimates of the original software within a predefined tolerance and operated 2.3 to 2.7 times faster on a single processor thread, achieving 4.4 to 9.5 times faster on eight threads. Contributors Andrew Bai and Andrew Ho noted that the agents quickly and accurately handled direct references, but extensions requiring statistical judgment necessitated human validation.

Innovations with Rust and GPU

Three other projects, rustar-aligner, svb, and kuva, involved Rust builds carried out with coding agents, including a complete recreation of STAR, a sequence alignment tool that had lost its active maintenance. James M. Ferguson, a contributor, explained that agents change what is achievable: rewriting a 20,000-line aligner by hand is inefficient, but with an agent, it becomes a task that can be completed in a few weeks. However, verification remains a human task, requiring the review of over 900 graphs before publication.

RustQC consolidated 15 RNA sequencing quality control tools into a single program, reducing execution time by 60 times and disk I/O by 25 times. The associated reconstructions FastQC-Rust and Trim Galore operated seven and three times faster, respectively, while preserving the behavior of the original tools. Phil Ewels, a contributor, emphasized that while technology is the easy part, managing behavioral divergences between tools remains a challenge.

HelixForge, a native GPU reconstruction of the mutation simulation tool BAMSurgeon, reduced execution time by approximately 60 times on a benchmark with real human data. Contributors Mamad Ahangari, Varun Goyal, and Hassan Masoudi stated that this produced more accurate mutation frequencies and resolved several bugs from the original tool.

Human Verification, an Imperative

The report reveals that while coding agents are effective at handling well-defined implementation requests, they cannot judge the scientific validity of their own output. Contributors observed that agents express confidence in work containing errors, thereby transferring the responsibility for verification to humans. This involves rigorous acceptance testing: exact matching of outputs, parity checks with existing tools, or validation using simulated data.

Projects generally progressed in stages, with agents producing rapid drafts while humans focused on edge cases and subtle numerical divergences that a simple benchmark would not have detected.

The reduction in engineering costs has ambivalent effects. It enables small teams to undertake projects that would have previously required significant funding, but it also facilitates the creation of incompatible versions of the same tool by different laboratories. Modifications made to MHCflurry and cyvcf2 were reintegrated into their original projects, while rustar-aligner was transferred to new community management due to the abandonment of the original tool.

The OpenAI report highlights the importance of determining ownership of a reconstructed tool and securing that commitment even before the first line of code generated by an agent is written.