CodeRabbit Claims 20x Deliveries and 43,000 Vulnerabilities Detected

The publisher claims to have increased its software shipments by 20 times with the same team, thanks to agents powered by OpenAI models and Codex. It also reports security results with Daybreak and a new product, while describing the technical and process adjustments that support this scaling.

Security: 43,000 vulnerabilities found and $1M in free analyses offered

CodeRabbit reports having identified over 43,000 security vulnerabilities during 13,000 in-depth analyses, all conducted using Codex models. Over the past two weeks, the company has offered $1 million in free analyses to its users, stating that thousands of flaws have already been fixed by its clients. More recently, CodeRabbit utilized OpenAI's Cyber model via Daybreak to test and strengthen the security of its own platform, conducting internal penetration tests to identify intrusion vectors. According to Harjot Gill, this early access allowed for the discovery of several vulnerabilities and advanced attacks. These capabilities have been integrated into the CodeRabbit Secure offering, which combines Daybreak and Codex models to detect and fix vulnerabilities, and CodeRabbit also uses Daybreak to secure both its clients' infrastructure and its own.

Production: 20× deliveries in six months without hiring

The company claims to have shipped 20 times more software with the same team, attributing this result to the use of Codex. Harjot Gill specifies that this volume was achieved over the last six months without hiring engineers, estimating that between 2,500 and 3,000 pull requests were submitted in the 30 days preceding the interview. To achieve this, the team automated the submission of low-risk changes that do not require human intervention, streamlined the publishing process, and reassessed human approval points. The execution of multiple agents highlighted constraints in the development chain, such as memory and power of workstations, testing, compilation and packaging tools, as well as dependencies. Harjot Gill indicates that software delivery has been completely rethought.

Orchestration: generalized agents and multiple sub-agents

CodeRabbit links the generalization of agents to improvements in pricing and caching of OpenAI models, which allow for greater use of AI and the addition of more sub-agents, with a reported benefit for clients. Initially, code navigation relied on one or two sub-agents; cost-effective models like GPT-6 Luna now allow for deploying a larger number on the platform, while GPT-6 Astra manages scheduling and orchestration. Harjot Gill states that agents are now used for previously inaccessible use cases, expanding possible validations and explanations provided to developers who did not write the changes.

OpenAI Models: from prototype to continuous evaluation

CodeRabbit's first product was built with GPT-3.5 Turbo, and according to Harjot Gill, the arrival of GPT-4 has made the tool effective for code review. The company reports having quickly adopted OpenAI o1 and o3. Code reviews are conducted asynchronously in the cloud, with slower models being used when their analysis was deemed more effective. New models are evaluated on an internal benchmark of known issues. At the time of the interview, Harjot Gill estimated that GPT-6 Astra detected 50% of the problems in this test set, placing it ahead of other models, in a context where the nature of bugs and the benchmark evolve with the improvement of agents. Harjot Gill also observes significant advancements in large-scale detection with each new version of models, particularly those focused on reasoning.

Tools and processes: from Slack bot to cloud development

A Slack bot powered by Codex manages the sorting of production incidents, messages, and support tickets, and is capable of creating a pull request for review and merging. Harjot Gill describes cloud development as a solution to reduce limitations related to local workstations and enhance access management and auditing. Beyond engineering, the team drives marketing and operational tasks like code: Google ads are managed through pull requests, and a content studio repository is used to produce presentations and documents for clients via Codex. Harjot Gill indicates a desire to manage the company as much as possible like code. Since 2023, agents have been operating in test environments, cloning repositories and writing code to assist with reviews.

Governance of agentic code: maintaining control at scale

CodeRabbit's platform leverages OpenAI models to govern and validate code generated by agents, aiming to help teams understand, review, and secure changes, then decide what can be shipped. The company claims a base of over 20,000 user organizations and more than 150,000 open-source projects. CodeRabbit indicates it is moving towards large-scale agentic software development, supporting validation, prioritization, explanation, securing, and verification of changes generated by agents. The goal is to help developers maintain control as agents take on an increasing share of the building process, while Harjot Gill anticipates a growing need for assistance in deciding which changes to implement when background agents respond to alerts without human intervention.