The Setup: When Your Service Moves Faster Than Your Ops Team
OpenAI has a problem that most engineering orgs would love to have: they ship so fast that traditional performance engineering can't keep up. So they built Performance Factory, an autonomous system that uses Codex agents to find and fix performance regressions without human intervention. The stats are wild: in just over three months, the Regression Workflow tracked more than 1,700 investigations and root caused over 600 cases. That's 17 automated investigations every day.
This isn't a demo or a research project. This is production infrastructure keeping ChatGPT fast while OpenAI's roadmap stays ambitious. And the architecture—task workflows, adversarial review, cloud execution with subscriptions—is worth studying if you're thinking about where agentic systems actually work at scale.
The Six-Stage Evolution: From Manual to Factory
What makes this post fascinating is OpenAI's transparency about how they got here. They didn't jump straight to autonomous agents. They laid out a six-stage progression that feels like a roadmap for any team adopting agentic workflows.
Pre-Codex: Engineers manually examined metrics, compared traces, debugged live machines. Time-consuming, required deep service knowledge, easy to miss signals. They built 30+ specialized performance tools over time.
Codex as partner: Engineers could ask "Why did this get slower?" and work with Codex to investigate. The model could search codebases, find metrics, follow cross-service interactions. But engineers still had to steer—knowing which tools to use, how to guide the investigation.
Reusable skills: They captured their performance team's process in a shared skill called perf-forensics. It tells Codex which tools exist, when to use them, how to work through a slowdown. They tested and refined it with evals based on past regressions.
The human proxy: Engineers started feeling like messengers between alerts and Codex. An alert fires, copy-paste it into Codex with the right skill, paste findings back to Slack. Classic automation smell.
Automated investigations: They moved investigations to the cloud so alerts could trigger Codex sessions automatically. Codex posted findings to Slack. But engineers still had to develop fixes and verify deployments.
The factory: End-to-end automation. The system tracks what each task needs next, assigns agents when steps are ready, maintains state across agents and engineers, and verifies recovery after deployment.
This progression from "partner" to "proxy" to "factory" is the pattern. When you start feeling like middleware between systems that could talk directly, that's your signal to automate.
Architecture: Four Components, Real Production Constraints
Performance Factory has four pieces that handle the messy realities of production work:
Task service with configurable workflows
Engineers define workflows using config files and Markdown prompts. The config lists states a task can move through, allowed transitions, which steps need humans. Separate prompts tell agents what to do at each step. All workflows visible in a UI where engineers can follow progress and share context.
Dispatch loop that assigns work
A loop periodically checks which tasks are ready. It assigns agents by starting a Codex turn with instructions for the task's current state. The agent records progress and moves the task forward if it completes the step.
Codex in the cloud
They built an internal API to invoke Codex turns programmatically. The turn runs on an environment inside their network with access to code repositories, metrics, logs, traces, CPU profiles, memory snapshots—all the observability tools their engineers use. Agents run under a service identity so they can scope permissions and audit actions. Cloud execution means parallel work without depending on engineers' laptops.
Subscriptions for async work
Some work waits for people or deployments. When a task needs to wait, it subscribes to events: review comments, failed CI checks on GitHub PRs. When an event matches, the task becomes ready again and the dispatch loop assigns an agent to continue.
This is good systems design. The subscription model keeps agents from burning tokens in polling loops. The service identity gives them audit trails. The cloud execution decouples work from individual machines.
The Regression Workflow: Alert to Recovery
Their first workflow handles performance regressions end-to-end. When a configured alert fires, the system opens a task that moves through stages: needs investigation → root caused → fix proposed → resolved.
Finding the root cause
The agent starts by figuring out which metrics worsened, when they changed, which requests were affected. The perf-forensics skill provides tools and a playbook. It looks back through data to find when the metric first degraded. It compares request groups: did similar requests actually get slower, or are they just serving more inherently-slow requests? Those findings point to different causes.
Once it knows when and where to look, the agent examines deployments and feature flags, reads relevant code, checks when changes reached affected requests, uses logs and traces to test whether code could explain what happened.
Adversarial review catches lazy diagnoses
Early on, agents would stop at the first plausible explanation. Some diagnoses were wrong. So they added adversarial review: a separate Codex session that tries to disprove the proposed diagnosis. The reviewer rechecks assumptions, tests counterfactuals. If a rollout caused the slowdown, did unexposed requests stay healthy? If the suspected change rolled back, did metrics recover?
In one case, an agent traced failing ChatGPT conversations to image download errors and called it found. The reviewer noticed the file-location service had already recovered while image requests still failed. The diagnosis didn't explain continuing failures, so the reviewer reopened the investigation.
They break larger jobs into focused turns. In a manual review of 100 root cause suggestions, they graded 70% accurate, with 30% needing refinement or not fully establishing causality. Different turns investigate, challenge diagnoses, and propose fixes—checks and balances for accuracy.
Implementation and verification
Once diagnosis passes adversarial review, the workflow asks Codex to implement a fix and open a pull request. An engineer reviews and decides whether to merge. After deployment, another agent picks up the same task to check whether performance returned to baseline.
The Production Lessons They're Learning
OpenAI shares three early lessons that feel honest about where this still needs work.
Domain-specific skills beat general prompting
Early attempts with general reasoning struggled. Building focused skills like perf-forensics with access to specific tools and playbooks worked better. This matches what we're seeing elsewhere: narrow, well-scoped agent capabilities outperform broad instruction-following.
State management matters
Keeping task state outside any single agent's context lets work pass between agents and engineers. Work can resume after code review or deployment. The task becomes the unit of work, not the agent session.
Adversarial review improves accuracy
Separate review sessions that try to disprove findings catch mistakes. This is expensive in token terms but cheaper than deploying bad fixes or missing real causes.
What This Means for Agentic Systems
Performance Factory is interesting because it's production infrastructure solving a real scaling problem, not a demo optimizing for impressiveness. The constraints are different: they need audit trails, permission scoping, async coordination, human-in-the-loop for risky changes.
The progression they describe—from partner to proxy to factory—gives us a pattern for identifying automation opportunities. When you're copying outputs from one system and pasting them into another, you're in the proxy phase. That's your signal.
The architectural choices reveal what matters for production agents: state management separate from agent context, subscription-based wake-ups for async work, adversarial review for accuracy, focused domain skills over general reasoning.
And the honesty about accuracy numbers (70% of root causes graded accurate, 30% needing refinement) is refreshing. This is production AI: good enough to run 17 investigations per day, not good enough to never need human review.
OpenAI is using their own models to maintain their own infrastructure at a pace humans couldn't sustain. That's the actual use case for autonomous agents—not replacing jobs, but keeping up with roadmaps that move faster than traditional ops can scale.