OpenAI published a July 29 engineering note on ARC-AGI-3 that is easy to misread as a simple benchmark win. The same GPT-5.6 Sol model scored 13.3% on the public set with the official harness, then 38.3% when OpenAI retained reasoning and enabled compaction. Output tokens fell by about 6x.
The gain sits in memory
ARC-AGI-3 asks agents to explore unfamiliar 2D games without instructions, infer goals and act efficiently. OpenAI’s explanation is that the official harness discarded private reasoning after each action and used rolling truncation once context exceeded 175,000 characters. That forced the model to repeatedly rebuild its understanding.
OpenAI wrote: “Benchmarks rarely measure AI models in isolation.” In plain terms, the benchmark also reflects API settings, context management and harness design. Retained reasoning keeps the model’s earlier plans across tool calls, while compaction preserves older observations as the run grows long.
The numbers need labels
The 13.3%, 38.3% and estimated 48% human score are all RHAE figures on ARC-AGI-3 public tasks. They are not private-leaderboard results and not accuracy scores. The ARC paper said humans solved 100% of its environments at release while frontier systems were below 1%, and it warned that purpose-built harnesses can blur what is being measured.
For developers, the practical lesson is direct: long-running agents should be evaluated with the same memory and context behavior they will use in production. Otherwise, a test may measure forced amnesia instead of model capability.
Agent evals are becoming systems work
The result does not prove GPT-5.6 Sol has solved ARC-AGI-3. It also does not invalidate ARC. It shows that agent benchmarks now include the surrounding system: memory retention, tool feedback, failure recovery and context compression. The next checks are whether ARC adjusts comparison rules, whether other labs reproduce similar gains, and whether private-set scores move in the same direction.
Sources: OpenAI research, ARC Prize Foundation paper, ARC documentation, CocoLoop, Hugging Face gameplay logs; verified ARC-AGI-3 RHAE scope, GPT-5.6 Sol official-harness 13.3% and Responses API harness 38.3%, public-task human estimate of 48%, roughly 1/6 output-token use and 175k context-window setting.