OpenAI just published a fascinating post-mortem on why GPT-5.6 Sol bombed the ARC-AGI-3 benchmark—and then didn't. The headline number: enabling two API settings tripled the model's score from 13.3% to 38.3% on the public task set, while cutting output tokens by 6x.
But the real story isn't the score bump. It's what this reveals about the gap between how we eval models and how we actually deploy them.
What is ARC-AGI-3?
ARC-AGI-3 is a benchmark of 2D puzzle games designed to measure reasoning and learning in unfamiliar environments. Agents explore games without instructions, inferring rules from observation. It's intentionally stripped down—no special tools, no fancy harness features—to expose model limitations and enable fair comparisons.
The benchmark scores models using Relative Human Action Efficiency (RHAE), comparing agent performance to a human baseline. Based on official gameplay logs, the average human tester scored around 48%.
GPT-5.6 Sol—a model that's solved open problems in mathematics like the cycle double cover conjecture and beaten Pokémon FireRed—scored just 7.8% on ARC-AGI-3. Its predecessor GPT-5.5 managed a paltry 0.4%.
Were 2D puzzle games unusually hard for these models? Or was something else going on?
The problem: the harness was fighting the model
When OpenAI dug into the eval setup, they found two issues with the official harness that were crippling the model's ability to learn.
First: discarded reasoning. After each game action, all private reasoning was thrown away. GPT-5.6 Sol had to re-figure out the game from scratch every turn. It could see a record of past moves and brief notes, but not the plans, insights, or thought processes that led to them.
Imagine trying to solve a puzzle where you're only allowed to remember your actions, not why you took them. That's what was happening here.
Second: rolling truncation. The harness used a 175,000-character rolling window. As the conversation grew, older actions became invisible. So not only was the model unable to remember its past thinking—it was losing memory of its past actions too.
These aren't obscure edge cases. They're fundamental mismatches between how the model was trained and how the eval was running it.
The fix: retain reasoning, enable compaction
OpenAI re-implemented the ARC-AGI-3 harness using their Responses API with two key changes.
Retained reasoning
GPT-5.6 models are trained to think with private reasoning messages before outputting replies or tool calls. These messages are retained as part of conversation history in production—in ChatGPT, in Codex, everywhere.
With the Responses API, passing the previous response ID automatically retains reasoning across tool calls and turns. This mirrors how the model is actually deployed.
The effect was immediate. GPT-5.6 Sol spent less time thinking before each action because it didn't have to reinterpret the game every turn. And when it could remember its past thoughts, it was much better at learning over time and employing coherent strategies.
Compaction instead of truncation
The second change replaced rolling truncation with compaction—a Responses API feature that summarizes conversation history instead of discarding it.
Rolling truncation has two problems: the model loses earlier observations, and it spends much of the task operating with a fuller context window, which can slightly impair performance.
Compaction preserved what GPT-5.6 Sol had learned about each game across longer runs. The model achieved higher scores with fewer output tokens—a win for both performance and efficiency.
The results: 3x score, 6x fewer tokens
With retained reasoning and compaction enabled, GPT-5.6 Sol (max) jumped from 13.3% to 38.3% on the public task set.
That's not a minor bump. That's the difference between looking incompetent and approaching human-level performance (estimated at 48% for the average human tester).
And it used 6x fewer output tokens to get there.
OpenAI's demo video shows the difference beautifully. On one puzzle where no frontier model solved beyond the first level with the official harness, GPT-5.6 Sol solved all six levels with the updated harness.
Why this matters beyond ARC-AGI-3
This isn't the first time OpenAI has been surprised by low benchmark scores, only to discover the eval runner was using a generic harness that dropped reasoning messages.
And it won't be the last.
The lesson here is simple but profound: evals rarely measure models in isolation. They measure a bundle of choices about API settings, harness design, and prompting.
When you see a model score, you're not just seeing how smart the model is. You're seeing how well the eval setup matches the model's capabilities.
This is especially important in the agent era. As models get better at extended reasoning and multi-step planning, the gap between "generic harness" and "production deployment" gets wider.
What this means for developers
If you're building with the OpenAI API and want to maximize performance, OpenAI's recommendations are clear:
- Use the Responses API, not the legacy Chat Completions API
- Retain reasoning across turns
- Use compaction instead of truncation
These aren't optional optimizations. They're the difference between hobbled performance and what the model can actually do.
What this means for eval designers
If you're building benchmarks, think hard about whether your harness matches real-world deployment patterns.
There's value in stripped-down, generic harnesses—they can expose fundamental limitations. But they can also create artificial bottlenecks that tell you more about the harness than the model.
ARC Prize's philosophy was that a simple harness makes model shortcomings more visible and comparisons more fair. That's a reasonable position. But "simple" and "realistic" aren't the same thing.
The most useful evals probably aren't the ones that treat all models identically. They're the ones that let each model use the settings and patterns it was trained for—then measure what it can do in that configuration.
The broader pattern
This ARC-AGI-3 episode is part of a larger trend: the growing importance of inference-time compute and extended reasoning.
Models like GPT-5.6 Sol aren't just bigger or better-trained. They're designed to think longer and remember more. That design shows up in how they use context, how they manage reasoning chains, and how they learn across interactions.
When evals don't accommodate those patterns—when they discard reasoning or truncate context in ways that never happen in production—the scores you get are artifacts of the mismatch, not measurements of the model.
We're going to see more of this. As reasoning models get more sophisticated, the gap between "run it in a minimal harness" and "run it the way it was meant to be run" will keep widening.
The question for the eval community is whether benchmarks will evolve to match how these models actually work—or whether we'll keep measuring them in configurations that guarantee underperformance.
Final thoughts
OpenAI's post ends with gratitude to ARC Prize for their work on AGI evaluation, and for the analysis that inspired this investigation. That's the right note.
This isn't a criticism of ARC-AGI-3. The benchmark is doing what it was designed to do: create a challenging, minimal test of reasoning and learning.
But it's a reminder that when you see a model fail a benchmark, the first question shouldn't be "why is the model bad at this?" It should be "what is the eval actually measuring?"
Sometimes the answer is "the model really can't do this." Sometimes the answer is "the model can do this, but the harness won't let it."
Figuring out which is which matters a lot.