On ARC-AGI-3, the Harness Can Outweigh the Model
A harness comparison ARC Prize published this month moved GPT-6 Astra's ARC-AGI-3 score from 62.7 to 99.9 percent, and a separate NVIDIA demonstration took Claude Opus 5 from 30.2 to 100 percent — evidence that on this benchmark, the software wrapped around a model can matter as much as the model itself.
On September 3, 2026, ARC Prize published its own breakdown of how OpenAI's GPT-6 Astra performs on ARC-AGI-3, the organization's interactive reasoning benchmark. The same model, tested on the same semi-private set of environments, scored anywhere from 17.5 percent to 99.9 percent depending on nothing but which piece of software carried its actions between turns. At maximum reasoning effort, Astra scored 62.7 percent through ARC Prize's own standard evaluation harness and 98.6 percent through OpenAI's own Provider Adapter harness, a 36 point gap for identical model weights on identical puzzles.
A harness, in this context, is the code that sits between a model and a benchmark environment: it decides how much of the model's prior reasoning survives from one action to the next, how conversation history gets trimmed as it grows, and how retries and errors are handled. ARC Prize's standard harness discards a model's private reasoning after each action and truncates older turns out of its context window, so the model effectively re-learns the rules of each game with every move. OpenAI's Provider Adapter preserves that reasoning state across the run and compresses history instead of cutting it, letting the model build on what it already worked out. ARC Prize reported the Provider Adapter runs were about 3.66 times faster and used 49 percent fewer tokens than the standard harness runs, on top of scoring far higher.
At high reasoning effort rather than maximum, the gap widened further: 54.8 percent on the standard harness against 99.9 percent, Astra's best recorded score anywhere on ARC-AGI-3, through the Provider Adapter, at a cost of 18817 dollars for the full run. The standard harness runs, despite scoring lower, cost more: 40705 dollars at high effort and 26098 dollars at maximum effort, since a model rediscovering a game from scratch on every move needs more tokens to get there.
Three weeks earlier, on August 21, NVIDIA published a parallel demonstration using a different model. ARC Prize's own leaderboard credits a bare Claude Opus 5, at high reasoning effort, with 30.2 percent on ARC-AGI-3's public set, itself a record when ARC Prize announced it on July 24, nearly four times the previous best of 7.8 percent set by GPT-5.6 Sol. Wrapped in NVIDIA's AVO agent architecture, which adds persistent memory and a supervisor layer on top of the same Claude Opus 5, the same model cleared all 183 levels across the public set's 25 environments for a perfect score, using 6624 environment actions, 12 percent fewer than the previous agent harness leader, VISTA, which needed 7542 actions running the same model.
NVIDIA was explicit that the comparison is not a controlled experiment: AVO and the bare model differ in agent backend, observation format, memory and context management, not just in whether a wrapper is present, so the jump from 30 percent to 100 percent should not be read as measuring AVO's contribution in isolation. The 100 percent figure also covers only the public set of environments; ARC-AGI-3's semi-private and private held out sets, which are harder to game because they are not published, remain untested by AVO.
Taken together, the two disclosures point at the same structural fact about benchmarks built around interactive, multi-step tasks rather than single questions: a leaderboard entry increasingly names a model plus an unspecified amount of surrounding engineering, and the two are not easy to separate from a single number. A vendor's own adapter or agent framework can move a score by tens of points without changing a single model weight.
ARC Prize's decision to publish the standard harness figure alongside the vendor supplied one, rather than only the higher number a lab would prefer to headline, is what makes the Astra comparison possible to make at all. It gives the leaderboard a fixed reference point that does not change when a vendor's own tooling improves, which is the same function an official results page serves in any competition where the equipment matters as much as the competitor.