How OpenAI Tripled GPT-5.6 Sol’s ARC-AGI-3 Score by Fixing API Settings

The Benchmark Trap: Why Standard Evaluation Harnesses Fail Frontier Models

When frontier models perform poorly on public benchmarks, industry observers frequently assume the architecture has reached a hard capability wall. However, evaluation frameworks often introduce subtle bottlenecks that severely handicap reasoning models. This was precisely the case with ARC-AGI-3, the third iteration of the benchmark created by the ARC Prize foundation to test how effectively AI agents learn, infer rules, and adapt to unfamiliar interactive 2D grid environments.

Under the benchmark's default public harness, OpenAI's high-tier reasoning model, GPT-5.6 Sol, scored a modest 13.3% on the public task set (and just 7.8% under unoptimized setups), while GPT-5.5 struggled to complete basic game loops, scoring 0.4%. Given that GPT-5.6 Sol had previously solved complex mathematical proofs—such as the longstanding cycle double cover conjecture—and played complex games like Pokémon FireRed and Slay the Spire 2, OpenAI researchers investigated whether the issue lay in the harness rather than model capability.

Their findings highlighted two structural flaws in how the public testing harness communicated with the model API:

• Discarded Chain-of-Thought Memory: After every action, the standard harness stripped away the model’s internal reasoning messages. Consequently, GPT-5.6 Sol had to re-evaluate the entire game state from scratch at every single turn without access to its previous hypothesis testing or tactical planning.
• Naïve Rolling Truncation: When conversation logs exceeded context boundaries (set at 175,000 characters/tokens), the harness discarded older observations. As a game progressed, the model lost memory of both its historical deductions and its earliest environment interactions.

The Technical Solution: Retained Reasoning and Compaction

To align the evaluation setup with real-world production setups used in ChatGPT and Codex, OpenAI modified the benchmark harness to utilize its modern Responses API instead of the legacy Chat Completions protocol.

1. Retained Reasoning across Turns

In modern reasoning models like GPT-5.6, private chain-of-thought tokens allow the system to analyze options, falsify hypotheses, and form multi-step plans before emitting visible tool calls or responses. By passing the previous response identifier across sequential API calls, the model preserves its continuous stream of internal thought. This single change prevented the model from spending vast amounts of compute re-analyzing the basic game rules on every move.

2. Context Compaction Instead of Naïve Truncation

Instead of hard-cutting older context windows when reaching the 175,000 token limit, OpenAI implemented context compaction. This system dynamically summarizes key historical milestones, environmental feedback, and game mechanics while discarding redundant step-by-step logs. The model maintains long-term tactical memory without filling its context window with raw frame descriptions.

The combination of these two settings allowed GPT-5.6 Sol (max context configuration) to jump from 13.3% to 38.3% in Relative Human Action Efficiency (RHAE) on the public set. For comparison, human test baselines average around 48% on these interactive tasks. Crucially, because the model retained its previous chain-of-thought, it generated 600% fewer output tokens per completed task.

Performance & Cost Efficiency Benchmark Comparison

To demonstrate the practical impact of harness engineering, we compiled a comparison table reflecting model scores, context retention methods, and approximate API execution costs across top-tier models evaluated on the ARC-AGI-3 public harness.

European Angle: API Economics, EU AI Act, and System Design

For AI engineers and enterprise software architects in the European Union, OpenAI's findings carry major commercial and regulatory implications that go far beyond benchmark bragging rights.

1. Financial Savings for EU Businesses

The 6x drop in output token usage translates directly into operational savings. With current enterprise rates for reasoning-heavy frontier APIs hovering around $15.00 / €13.80 per million input tokens and $60.00 / €55.20 per million output tokens, failing to preserve reasoning history forces agents to re-think context from scratch. For complex agentic workflows—such as automated software testing or complex SAP/ERP data integration—enabling retained reasoning and context compaction cuts API billing drastically.

2. EU AI Act and Reasoning Transparency (Article 50 Compliance)

Under the EU AI Act, providers and deployers of high-risk AI applications face mandatory transparency obligations regarding model decision-making processes. Retaining internal reasoning tokens inside private API state managers allows developers to log complete execution chains for auditability. However, European firms must balance private reasoning retention with GDPR data minimization principles. Compaction pipelines must ensure that personal identifiable information (PII) is sanitized or hashed before long-term state summaries are stored in memory buffers.

3. Availability in Europe

The Responses API features—including retained reasoning identifiers and compaction endpoints—are fully available to European developers via standard OpenAI API endpoints as well as Microsoft Azure OpenAI Service in European data centers (including Frankfurt, Dublin, and Sweden). European open-weight alternatives, such as those from Mistral AI, are actively adopting similar stateful API paradigms to optimize agent interactions across our local AI coverage and analysis workflows.

Key Takeaways for Developers and AI Evaluators

As autonomous agents transition from basic text generation to interactive problem solving, evaluation methodology must evolve. Relying on legacy stateless completions APIs introduces artificial bottlenecks that distort real-world model utility.

• Deprecate Legacy Chat Completions: For multi-turn reasoning tasks, transition from legacy Chat Completions endpoints to stateful message handlers (such as the Responses API).
• Preserve Internal State: Ensure your system preserves intermediate chain-of-thought output across turns rather than clearing context buffers.
• Implement Intelligent Compaction: Avoid brute-force context trimming. Use semantic compaction or structured memory summarization to retain critical initial instructions while staying within token budgets.