OpenAI: Habitat Returns to Multi-Tenant on Cosmos DB

OpenAI retraces the journey of its storage platform, Habitat, which transitioned from dedicated container isolation back to a shared model on Azure Cosmos DB. The result is improved global read performance, but also tight trade-offs between reliability, capacity, and neighbor effects, serving over a billion users and more than 500 petabytes of data.

Increased Incidents and Major Outages Avoided Through Isolation

With growth, the pressure on the default container and production incidents increased. OpenAI then provisioned containers for each new service or major feature to spare the shared container from additional load. This strategy helped avoid numerous site-wide outages, limiting the impact to degraded functionalities or services. However, the cost of creating and managing new containers eventually accumulated.

Isolation Costs in Capacity What Multi-Tenant Absorbs

In the old shared container, a use case moving from 0 QPS to 1 million QPS had little effect on the aggregated usage. In an isolated container, the same transition could reach 1,000 times the load of the internal reference. Teams then had to pre-provision the expected throughput before launches, with oversights resulting in SEV cells on the day of the launch to ramp up at the last minute. The advantage of mixed loads inherent in multi-tenant systems disappears with this increased isolation.

Return to a Co-Managed Multi-Tenant Model for Better Global Reads

OpenAI details the evolution of the underlying storage component based on Azure Cosmos DB and a close collaboration with the Azure team. The outcome is a multi-tenant model with co-managed containers, providing excellent read performance at a global scale. Habitat is now a shared database layer for many products, following a journey from multi-tenant to backup single-tenant and then back to default multi-tenant. OpenAI indicates that it is still at the beginning of this cycle and relies on the built-in primitives of Cosmos DB to defer certain investments.

Why Isolation Didn’t Solve Everything

Separating containers partially resolved the limits related to logical keys and noisy neighbors, with each container having its own physical partitions. However, each container requires dedicated capacity planning. OpenAI reserved this approach for critical use cases, supported by a script aligning the total provisioned throughput with the number of physical partitions, which primarily helped avoid issues. Historical constraints, however, left critical cases in the shared container, hence the recommendation to create separate containers for new major services still dependent on common data.

Concrete Limits Encountered with Cosmos DB Initially

Initially, OpenAI relied on the limit of 10K RU/s per logical partition key, with a single default container per account. Exposing the partition key to clients generated collisions on common keys, constraining throughput for the same user. In practice, the ceilings apply to physical partitions: co-located keys share resources and struggle to reach 10,000 RU/s, risking niche workloads disrupting essential flows like login and registration. The distribution of provisioned throughput among partitions, which multiply as data grows, decreases the effective ceiling per partition and can reduce the processing capacity of active loads even at constant query rates.

Target Scale and Engineering Roadmap

All OpenAI products use Habitat to access data in milliseconds and serve over a billion users across 40 regions. The goal combined use case growth, increased reliability, performance, and development speed, which necessitated rethinking the design and operation of online storage. The trajectory followed three stages: reliability for critical traffic, speed for a global audience, and then operation at scale. Habitat is a globally distributed storage platform managing over 500 petabytes. The first phase of the project transitioned Habitat from a Python library to a service, using techniques that pushed the boundaries of Python. On the Cosmos DB side, an account defines containers and replication, each container is provisioned in RU/s, data is sharded by logical key with limits per key, and the service manages the division into physical partitions according to storage and provisioned throughput.