OpenAI's infrastructure team just dropped a fascinating deep-dive on scaling Habitat, their online storage platform that now handles over 70 million requests per second for ChatGPT and related products. The post reads like a war story from the trenches of hypergrowth—10x year-over-year scaling for three consecutive years—and it's full of counterintuitive choices that actually worked.
The biggest surprise? They built the service layer in Python. At 70M RPS. Serving 1 billion+ weekly users. And they knew from day one it was technical debt they'd eventually have to pay back.
From library to service: The coordination tax became unbearable
Habitat started life as a simple Python library that product engineers imported directly. The idea was clean: abstract away database operations so developers don't need to think about Azure Cosmos DB, connection pooling, encryption, or routing. Just call the library and get your data.
This worked beautifully until it didn't. The breaking point came when they needed to migrate critical datasets to regionally distributed Cosmos DB accounts to reduce blast radius during outages. What should have been a straightforward infrastructure change turned into a multi-day coordination nightmare across dozens of services.
Every change required rolling out a new client version, waiting for all teams to deploy, then enabling feature flags. Found a bug? Start the whole cycle again. One team rolled back for unrelated reasons and immediately triggered the exact outage scenario the migration was supposed to prevent.
The infrastructure team made the call: pull Habitat into its own standalone service. One deployment path, one control point, immediate rollout to all products. The coordination tax was killing them.
The Python gamble: Strategic technical debt
Here's where it gets interesting. They knew Python was the wrong choice for a high-throughput storage service. The extra network hop adds latency. CPU and memory overhead compared to in-process library calls is substantial. And they explicitly acknowledged that at 100x scale, Python would be unacceptable.
So why do it? Two reasons, both very pragmatic:
First, their primary goal wasn't optimization—it was unblocking product teams and achieving platform stability. A Python service they could ship in weeks beat a perfectly optimized Rust/Go service they'd ship in months.
Second, they made a calculated bet on their own AI capabilities. By the time a full rewrite became necessary, they wagered that Codex and GPT would be sophisticated enough to make migration dramatically easier. That bet, according to the post, "eventually proved correct."
This is such a 2024-era infrastructure decision. Incur technical debt specifically because you expect AI tooling to make payback cheaper. I'm curious whether we'll see this pattern normalize or whether it was unique to OpenAI's position.
Taming Python at scale: The asyncio scheduling problem
The meat of the post is how they actually made Python work at massive scale. The core challenge isn't throughput—it's tail latency. When a user request triggers hundreds of database lookups, the slowest call dominates user experience.
Python's asyncio provides concurrency for I/O-bound work, but it doesn't escape the Global Interpreter Lock (GIL). Only one coroutine executes on the CPU thread at any given time. With Habitat handling CPU-heavy tasks like compression, encryption, checksumming, and request hedging, asyncio scheduling delay became the dominant source of tail latency.
Before tuning, their traces showed requests stalling not because Cosmos DB was slow, but because coroutines were waiting to be rescheduled to parse responses that had already arrived. The CPU was too busy doing other work to process ready data.
Their solution: instrument the hell out of the event loop. They schedule periodic background tasks and measure the delta between expected and actual execution time, giving them real-time empirical data on scheduling jitter. At high utilization, they observed delays up to hundreds of milliseconds, sometimes several seconds in edge cases.
The fix? Counterintuitively, they reduced concurrent requests per process and massively scaled out the number of Python worker processes instead. This keeps the event loop responsive at the cost of horizontal scaling.
Two surprising performance wins
The post documents two delightful debugging wins that every Python service operator should internalize.
Feature flag configs were wrecking tail latency
They were using Statsig for feature flags, configured to poll every 60 seconds with no jitter. The config included every production rule across every service. Each pod ran 8 Python processes to maximize CPU utilization.
Result: every minute, all 8 workers would simultaneously stop processing in-flight requests to parse a giant JSON config file. Live CPU profiling caught it. The fix was straightforward—deploy targeted configs, lengthen refresh intervals, add jitter—but the lesson is universal: periodic background tasks in Python services need careful tuning and jitter.
Connection pooling fought load balancing
Client-side connection pooling created an unexpected problem. A single client process making many concurrent requests would establish only a handful of persistent connections, sending all its load to the same few server processes. Even with perfect request distribution at the load balancer, utilization across backend workers was wildly uneven.
They tuned connection pool settings to establish more connections per client and implemented connection recycling to spread load. Simple fix, but non-obvious consequence of connection pooling at scale.
What's missing: The rewrite and multi-region story
This is explicitly "part one" of a two-part series. The post ends mid-sentence (literally—the source content is truncated). Part two promises details on multi-tenancy reliability, read performance optimization layers, and how they scaled their Azure Cosmos DB partnership.
What I really want to know: did they actually do the rewrite out of Python? The post strongly implies it's coming but doesn't confirm. And if they did, what role did AI coding assistants actually play? That would be the validation or refutation of their strategic debt thesis.
Why this matters
This post is a rare public artifact of what infrastructure decision-making looks like during hypergrowth. Most companies either don't scale this fast or don't write candidly about the tradeoffs.
Three takeaways stand out:
-
Incurring technical debt to buy strategic time is valid, especially when you're confident in future tooling improvements. This is the inverse of "you aren't Google, don't build like Google."
-
Python at scale is possible but requires deep instrumentation. You need visibility into event loop scheduling, not just standard metrics. Treating asyncio delay as a first-class operational metric is critical.
-
Sometimes the bottleneck is organizational, not technical. The library-to-service migration wasn't about performance—it was about deployment coordination tax. The fastest code is useless if you can't ship it.
I'm looking forward to part two, especially the multi-region architecture and whether the Cosmos DB integration hit any interesting scaling cliffs. And honestly, I just want to know if the AI-assisted rewrite gamble paid off.