OpenAI just released two new evaluations for low-resource programming languages—COBOL, Fortran, Visual Basic, OCaml, and sixteen others—and the results tell a story about how far reasoning models have really come. We've spent months hearing about o1 and its successors solving competition math and writing proofs, but these benchmarks measure something harder to hype and way more useful: whether models can actually maintain and migrate the code that runs critical infrastructure.
The headline numbers are striking. On MultiLangBench, which asks models to build a SQLite-compatible database engine from scratch in twenty different languages, GPT-6 Astra achieves an average test pass rate of 87.4%, up from 26.8% for GPT-5.6 Sol. On a COBOL-to-modern-language migration task requiring byte-for-byte output parity, GPT-6.1 Sol hits 86.6% accuracy at low reasoning effort and 97.9% at high effort—compared to just 5.4% for the previous model. These aren't incremental gains; they're the kind of jumps that change what you can actually ship.
Why low-resource languages matter (and why they're hard)
COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions, and 95% of ATM swipes. Fortran powers weather forecasting and nuclear reactor safety analysis. These aren't hobbyist projects or legacy curiosities—they're the load-bearing infrastructure of modern civilization.
But they're also languages where models have historically struggled. The training data is sparse compared to Python or JavaScript. Documentation is incomplete. The developers who wrote the original code retired decades ago, and the business logic often exists only in the code itself, undocumented and deeply embedded.
OpenAI frames this work explicitly around feedback from enterprise customers who need to maintain and modernize these systems. That framing matters. This isn't about enabling greenfield COBOL development; it's about migration, preservation, and understanding what existing systems actually do.
MultiLangBench: building SQLite in twenty languages
MultiLangBench is the more ambitious of the two evaluations. It asks models to implement a SQLite-compatible database engine in a specified language, adapted from SQLite's own test suite. The model has to organize a codebase, manage state, handle errors, and add features incrementally while keeping existing functionality intact.
The evaluation uses database files as a shared interface—feed the candidate program an initial database and a SQL query, get back a modified database, then check whether it matches what the reference SQLite implementation would produce. This design lets OpenAI apply identical tests across all twenty languages without worrying about language-specific quirks in how code is structured or run.
GPT-6 Astra scores above 83% in nineteen of the twenty languages tested. COBOL remains the hardest at 72.1%, up from just 10.9% on the previous model. OCaml jumps from 25.6% to 85.8%. Fortran goes from 21.4% to 83.5%. Visual Basic climbs from 27.5% to 89.4%.
GPT-6 Luna offers a cheaper option at roughly 78% lower estimated API cost than GPT-5.6 Sol, hitting 40.3% average pass rate. That's still a meaningful improvement over the prior generation across every language, delivered at a fraction of the cost. For teams that don't need maximum accuracy, Luna makes it feasible to run exploratory passes or generate candidate implementations at scale.
OpenAI is releasing MultiLangBench on GitHub, including the full test suite and per-language results. That's a big deal—public benchmarks for low-resource languages have been nearly nonexistent, and giving developers a way to inspect what's actually being tested makes the evaluation credible in a way closed evals never are.
COBOL migration: the behavior-preservation problem
The second evaluation tests something subtler and arguably harder: migrating COBOL programs to C++, Java, Kotlin, Python, or Rust while preserving their exact behavior. This isn't about rewriting for readability or modernizing architecture—it's about byte-for-byte output parity on hidden test cases.
COBOL's semantics make this treacherous. Fixed-point arithmetic discards excess decimal digits unless rounding is explicitly requested. A naïve rewrite that defaults to rounding can run without errors while producing silently wrong financial results. COBOL programs encode business rules in low-level language behavior—rounding policies, missing-record handling, field alignment—and losing those details breaks the system even if the code compiles.
GPT-6.1 Sol achieves 86.6% accuracy at low reasoning effort, a sixteen-fold improvement over GPT-5.6 Sol's 5.4%, at 39% lower cost. Crank up the reasoning effort and accuracy hits 97.9%, nearly saturating the benchmark. That's not just better—it's crossing the threshold where migration tooling starts to feel plausible for real codebases, assuming expert review and integration testing.
What this means for reasoning models in production
These results confirm what we've suspected since o1: extended reasoning helps most on tasks where correctness is discrete and verification is cheap. MultiLangBench and the COBOL eval both have that structure. You can run tests, get a pass/fail signal, and try again. The model can think longer, explore approaches, catch its own errors.
But they also reveal something about breadth. The jump from GPT-5.6 Sol to GPT-6 Astra isn't just bigger reasoning budgets—it's better base capabilities across languages where training data is thin. COBOL going from 10.9% to 72.1% on MultiLangBench suggests meaningful improvements in how these models generalize from high-resource to low-resource languages, not just how long they think.
The cost dynamics are interesting too. GPT-6 Luna delivers real improvements at dramatically lower cost, which matters for workflows where you're generating dozens of candidate implementations or running speculative migrations. GPT-6.1 Sol at low reasoning effort is cheaper than the previous generation while delivering 16x better accuracy on COBOL migration. That's the kind of price-performance shift that changes procurement conversations.
Open questions and what's still missing
OpenAI is careful to note that behavior-preserving migration "requires expert review and testing against the system's business rules and dependencies." That's correct, but it leaves open how much expert review. If a model hits 98% on an eval, does that translate to 98% of functions migrating cleanly, or does it mean 98% of test cases pass while 30% of functions still need manual fixes? The distribution of failures matters as much as the aggregate score.
The evaluations also don't measure some of the hardest parts of real migration projects: handling platform-specific dependencies, managing data migration alongside code migration, or dealing with systems where the original COBOL is just one layer in a stack of interacting legacy components. MultiLangBench tests greenfield implementation; the COBOL eval tests isolated program migration. Both are useful proxies, but production software migration is messier.
Still, these benchmarks are a step forward for transparency. OpenAI is releasing MultiLangBench publicly and describing the COBOL eval's methodology in enough detail that others can replicate it. For a field that's been criticized for closed, hard-to-verify evals, that's progress.
The bigger bet
What OpenAI is really signaling here is that reasoning models are ready for enterprise software engineering workflows, not just flashy demos. COBOL migration and legacy database implementation aren't sexy, but they're problems with real budgets and urgent timelines attached. If models can preserve behavior across migrations and build working implementations in obscure languages, they can help with the boring, high-stakes work that most software teams actually do.
The question now is whether enterprises will trust these tools enough to integrate them into production workflows—and whether the models can maintain this level of performance when the code gets messier, the dependencies multiply, and the business rules turn out to be undocumented and contradictory. The benchmarks suggest the capability is there. The next test is whether it survives contact with reality.