OpenAI centralizes Habitat: 70M req/s and 500 Po for ChatGPT

OpenAI has extracted Habitat, its data access layer, into a centralized service to streamline deployments and enhance access controls. The platform now handles over 70 million requests per second and manages more than 500 petabytes, serving a billion weekly users, while temporarily coping with Python's high-throughput limitations.

A Unique Service for Security, Auditing, and Resilience

OpenAI has transformed Habitat into a standalone service to reduce operational dispersion of deployments and provide a single control point for updates and observability. This centralization allows for the enforcement of access control policies, execution of audit logs, and limitation of exposure to storage resources like Azure Cosmos DB. Habitat thus plays a critical role in data protection and the prevention of unauthorized access, whether from external actors, internal users, or agents. This evolution also addresses resilience challenges: the desire to distribute critical data sets across Azure Cosmos DB accounts by region highlighted the burdens of a coordinated deployment behind a feature flag. Between deployments spread over several days, the introduction of shadowing to validate partitioning, successive patches, and the rollback of a team that triggered the dreaded outage, coordination via a client library proved fragile and costly. The centralized service aims to eliminate these organizational failure points.

A Data Layer Scaled for a Billion Users

Habitat processes over 70 million requests per second for products used by more than a billion people each week and operating in nearly 40 regions. The distributed system now manages over 500 petabytes of data. These figures illustrate the scale at which the data access platform operates daily.

Technical Choices Made Around Python

OpenAI chose to launch Habitat as a service without immediately moving away from Python, despite increased network latency and higher CPU and memory costs compared to a local library. The anticipated limits of Python at 100 times the scale made a future rewrite likely, a compromise presented as an accepted technical debt to prioritize platform stability and team agility. Establishing core APIs and a robust infrastructure was deemed a priority, betting on the rapid advancements of its coding models. The gamble that Codex and GPT would facilitate a later migration has proven correct. Even though running Habitat in Python remains suboptimal in performance, the goal was to contain tail latencies, which are crucial when hundreds of database calls can compose a user request, and the slowest call dictates the service perception.

Abstracting Database Complexity for Product Teams

From the outset, Habitat aimed to relieve product engineers from database management: by mid-2024, a small Python library connected to the main ChatGPT server translated a set of operations to Azure Cosmos DB. The layer identified data types, their sources, and verified permissions, while supporting schema discovery, routing, encryption, serialization, request formatting, and connections, regardless of whether the data came from Azure Cosmos DB, caches, or other storage. In products where connection, parameter verification, or chat initiation trigger multiple data queries, this abstraction limits the impact of slowdowns and failures that would otherwise immediately affect the user side. Two years after its inception as a simple library linked to a single database, Habitat has evolved into a distributed architecture.