OpenAI’s Habitat online storage began as a lightweight Python client intended to make user and product data easily accessible to engineering teams without requiring database expertise. Launched in mid‑2024 and backed by Azure Cosmos DB, Habitat quickly outgrew its original role. Over successive iterations it became a globally distributed service that now handles more than 70 million requests per second, stores over 500 petabytes of data and serves products used by more than 1 billion people each week across nearly 40 regions.
The platform’s evolution from a client library to a centralized service was driven by operational realities. As Habitat’s footprint expanded, coordinating backward‑compatible changes across numerous client services became brittle and risky: protocol changes could demand feature flags, rolling updates and cross‑team coordination that took days and still left room for regressions. To reduce operational fan‑out, OpenAI migrated Habitat from client side code into a standalone service by mid‑2025. Centralization simplified deployments, observability and improvements, and provided a single enforcement point for access controls, audit logging and other security and privacy primitives.
OpenAI initially retained Python for the service layer to move quickly and unblock teams. Running a high‑throughput, multi‑region service in Python introduced trade‑offs: the local library offered lower latency and resource usage, whereas a remote Python service incurred higher network latency and greater CPU and memory costs. Python’s single‑threaded execution model and the Global Interpreter Lock (GIL) also complicated scaling for CPU‑bound work. Habitat depended on asyncio for concurrency, but asyncio does not provide CPU parallelism; many of Habitat’s tasks—routing, compression, encryption, checksums, health checks, shadowing and hedging—are CPU intensive and competed with the event loop.
To confront long tail latencies caused by coroutine scheduling delays, engineers measured event loop scheduling delay by tracking the delta between expected and actual execution times for background tasks. They limited concurrent requests per process and massively increased the number of Python worker processes to reduce contention. CPU profiling revealed specific, actionable causes of spikes. Periodic parsing of a large feature‑flag configuration from Statsig produced synchronized stalls across processes; mitigations included deploying smaller targeted configurations, lengthening refresh intervals and adding jitter to background tasks so work would not align and produce simultaneous stalls.
Load balancing and connection management also required careful tuning. Client‑side connection pools led to metastable failures because Python’s aiohttp TCPConnector defaulted to LIFO connection reuse, concentrating load on slower servers after bursts. Switching to FIFO reuse and leveraging Istio and Envoy for smarter connection pooling and server‑aware balancing helped break that feedback loop. Habitat’s horizontal scaling with many Python processes risked overwhelming downstream systems through connection churn; OpenAI used Envoy to upgrade HTTP/1 to HTTP/2 for connection multiplexing and to centralize rate limiting and circuit breaking, reducing the number of connections that hit downstream services and providing a single place to enforce protections.
Architecturally, Habitat exposes a deliberately constrained NoSQL‑style API that favors predictable, constant‑cost requests over a full query language. This choice prevents clients from issuing unbounded queries or expensive joins that could destabilize the online service. For teams that need richer analytics or search, Habitat streams change data capture (CDC) into isolated Rockset instances that teams manage and scale independently. That separation keeps the online storage optimized for low‑latency operational traffic while still enabling more complex workloads off the critical path.
OpenAI deferred a large rewrite while prioritizing product growth, but by Q2 2026 the team completed a migration to Rust. With two engineers and assistance from Codex and GPT‑5.5, the team rewrote Habitat in Rust; the new implementation now handles roughly 95% of production requests and is planned to replace the Python implementation entirely. OpenAI reports the Rust service is about six times more CPU efficient and fifteen times more memory efficient than the Python version, while also delivering lower average and tail latencies.
This is the first of a two‑part series. The follow‑up will dive into the storage layer and explain how Habitat scales Azure Cosmos DB to support its data footprint, including operational lessons on multi‑tenancy and layered read optimizations. For now, Habitat online storage stands as a case study in pragmatic trade‑offs: start small to move fast, centralize to reduce operational friction, and iterate on implementation when scale and efficiency demand it.
Source: Read the original source
Leave a Reply