OpenAI has published a detailed post explaining how its Habitat storage platform scaled Azure Cosmos DB to deliver millisecond responses for more than a billion users around the world. Habitat storage was redesigned through multiple engineering phases to balance reliability, performance and operational cost while serving mission-critical product traffic across dozens of regions.
Azure Cosmos DB provided the core primitives—replication, partitioning and throughput controls—that shaped Habitat’s architecture. OpenAI engineers explain that containers in Cosmos DB are provisioned with request units per second (RU/s), logical partition keys map to physical partitions, and physical partitions split as data grows. Those primitives both enabled scale and imposed constraints: per-key throughput limits and the potential for noisy-neighbor effects when many logical partitions shared the same physical shard.
Early in Habitat’s life, exposing logical partition key selection to clients produced frequent key collisions on values such as user_id and workspace_id. Those “hot keys” concentrated traffic on a small set of partitions, capping throughput for individual users and degrading performance for unrelated tenants that happened to share the same physical partition.
OpenAI describes a three-phase evolution to address those issues. Phase I used a single shared container per account, which simplified provisioning but created contention and partition-level throughput bottlenecks. In Phase II the team moved to an isolated container model for critical workloads—single-tenancy—so each workload received its own physical partitions and dedicated throughput. That reduced the blast radius from hot keys but imposed heavy operational overhead: provisioning containers required capacity planning and made aggregate growth unpredictable.
As the organization scaled, OpenAI returned to multi-tenancy as the default, but only after close collaboration with the Azure team to mitigate the original problems. Improvements included hot-key throttling to cap the effect of extremely hot keys, request-tiering quality-of-service to deprioritize offline bulk workloads, and an enhanced serverless mode in Cosmos DB that allowed physical partitions to reach 10K RU/s and to split based on RU usage in addition to storage. Those changes reduced contention while preserving the operational simplicity of multi-tenancy.
At OpenAI’s scale, managed services required co-managed operations. The company now maintains shared tests and dashboards with the Azure Cosmos DB team, direct engineering channels, and production-scale validation accounts. Two regional failovers in late 2024 revealed gaps in failover testing and gateway behavior; investigating those incidents led to operational and software fixes. Today OpenAI jointly monitors both the Habitat and Cosmos DB layers, runs SDK and gateway tests against production-sized non-production accounts, and coordinates best practices with Azure engineers.
Reducing global read latency became a priority as Habitat grew into additional regions. Habitat serves a read-heavy workload, so OpenAI adopted a layered strategy: place data near users where capacity permits, replicate only the smallest latency-sensitive datasets broadly, and use an integrated distributed cache for the majority of reads. The caching layer uses Valkey servers colocated with compute regions and leverages change-data-capture (CDC) from Cosmos DB’s change feed to propagate invalidations across regions within seconds. By invalidating cache entries on updates rather than relying solely on TTLs, the approach improves hit rates while keeping data staleness low. Additionally, OpenAI rewrote the Habitat proxy from Python to Rust to reduce service overhead. Collectively, these measures lowered global read p50 from roughly 120 ms to about 10 ms—a reported 12× improvement.
Change-data-capture is central to more than just caching: it also powers indexing, analytics and data movement. OpenAI’s CDC platform processes millions of changes per second across thousands of containers and feeds hundreds of Kafka topics. A control plane automates container capture, chooses read regions, and balances thousands of workers across regions. To address scale-driven inefficiencies, the team adapted Azure’s CDC SDK to batch checkpoints in memory and flush them periodically, hardened worker lifecycles, and implemented dead-letter handling to avoid head-of-line blocking.
Looking ahead, OpenAI is building an in-house exabyte-scale online storage system optimized for Habitat’s bulky data types. That system will use a RocksDB Cloud core from Rockset with Raft-based replication, implement the same document model as Cosmos DB, and employ disaggregated storage and physical replication to lower flash requirements while accepting a lower peak QPS tradeoff. This work remains in development, and OpenAI says it is hiring engineers for related roles.
OpenAI’s account provides a rare, operationally grounded view of how managed database primitives, close vendor collaboration and targeted engineering optimizations can combine to support global growth. Habitat storage’s trajectory—from multi-tenant simplicity to single-tenant isolation and back to a hardened multi-tenant model—illustrates the tradeoffs organizations face when scaling low-latency, read-heavy systems at planetary scale.
Source: Read the original source
Leave a Reply