What is ARC-AGI and why it matters

ARC-AGI (Abstraction and Reasoning Corpus for Artificial General Intelligence) was created in 2019 by François Chollet, the creator of the Keras framework and one of the most influential thinkers in AI. Unlike most benchmarks, it doesn't measure how much data a model "absorbed" during training. It measures the ability for abstract reasoning on unfamiliar tasks — precisely the skill we expect from AGI.

Each task in ARC looks like a grid composed of colored squares. The model receives 2–3 example input-output pairs and must deduce a transformation rule, which it then applies to the test input. No learned patterns — each time it's a different type of transformation. To give you an idea: the average human scores roughly 80% on ARC-AGI-1, while standard language models without special prompting barely surpass 15%.

The ARC Prize Foundation, which runs the benchmark, calls it the "north star of AGI" — and major figures like Sam Altman, Demis Hassabis, and Elon Musk have repeatedly cited ARC as a key indicator of progress toward general intelligence.

Two traditional paths — and a third that no one tried

Previous successful approaches on ARC-AGI-1 fell into two camps:

First path: expensive models + evolutionary search. You take a top-tier model like GPT-4o or Grok-4, have it generate candidate solutions, and then improve them evolutionarily. It works — Ryan Berman achieved 79.6% with Grok-4 this way in 2025 — but it costs $8.42 per task. Other systems ran as high as around $400 per task.

Second path: specialized models trained directly on ARC data. Small networks learned from scratch or fine-tuned on the ARC distribution. The downside is obvious: the model learns the benchmark, not abstract reasoning. And as subsequent analysis showed, the performance of these systems often depends on test-time augmentations and special task identifiers — without them, accuracy drops to zero.

Kabir Moghe and Peter Chin of Dartmouth College took a third path: What if you don't need an expensive model, nor do you need to specially train it? What if a substantial portion of the performance is achievable purely through the architecture of the agent system that wraps around the model?

Explorer-Definer Pipeline: Divide and conquer

The core of their approach is a two-phase agent architecture called the Explorer-Definer Pipeline. Instead of the model immediately guessing the correct transformation, the researchers split the process into two separate steps:

Phase 1 — Explorers: Five parallel-running agents independently study the training input-output pairs. Each takes notes, tests hypotheses, and ultimately formulates a description of the pattern it sees in the data. This isn't voting — each explorer brings its own perspective.

Phase 2 — Definer: One agent receives all five pattern descriptions and, based on them, writes a Python function that should perform the transformation. The function is then run on the training data — if it matches, it moves forward. If not, the definer receives feedback and tries again.

The result? The pipeline achieved 57.50% pass@2 at a cost of $0.25 per task — roughly 6 CZK. For comparison: the same model without the agent architecture (simple one-shot) scored only 15.50%.

Reflective Orchestrator: When the first attempt isn't enough

The pipeline, however, hit a ceiling. The authors conducted an important diagnosis: they found that the system is not selection-bound but generation-bound. In other words — even if you had a perfect mechanism for selecting the correct transformation, you wouldn't find it, because the system never generated it in the first place.

This insight led to the creation of the Reflective Orchestrator — an extension of the pipeline that adds to the definer phase the ability to "spawn" new explorers right in the middle of the process. When the definer discovers that its current hypothesis fails on the training data, instead of repeatedly refining a flawed assumption, it can kick off an entirely new exploration phase, this time targeted at the specific failure.

The architecture thus gains the ability for adaptive re-exploration — and this translated into results: 67.25% pass@2 at a cost of $0.62 per task (about 15 CZK). The total jump from the basic one-shot model to the orchestrator amounts to 52 percentage points — all on the same model, without any fine-tuning whatsoever.

Think tool as the secret weapon

An interesting side discovery is the role of the so-called think tool — a mechanism that allows the agent to "think privately" between individual steps, similar to what Anthropic's Claude does. When the researchers removed the think tool from the definer, the pipeline's accuracy dropped by 5.75 percentage points. Conversely, removing the feedback mechanism from the training data had only minimal impact (−0.75 pp).

This suggests that the key value doesn't come from iterative error correction, but from the quality of the initial reasoning — and that's exactly what the think tool serves.

What this means in practice — and for the Czech Republic

The Dartmouth research has several practical implications worth noting:

1. Agent architecture as a performance multiplier. Even a relatively cheap model (DeepSeek V3.2 costs roughly $0.30 per million input tokens via AtlasCloud) can compete with much more expensive systems given a good architecture. For Czech startups and developers who don't have the budget to run GPT-5.5 or Claude Opus 4, this is a crucial message.

2. DeepSeek V3.2 is open-weight. You can download it and run it locally, without API dependency. This is key for companies that want to keep their data under control — and in the context of the European AI Act and GDPR, local model deployment is often the preferred path. DeepSeek also supports Czech at a very good level, making it a practical choice for Czech applications.

3. The code is publicly available. All prompt-engineering templates, agent loop implementations, and analytical scripts can be found on the project's GitHub. For Czech AI researchers and students, this is excellent study material — the architecture is well-documented and reproducible.

Limitations that are fair to talk about

The authors are honestly critical of their results. The results are not from the official ARC Prize leaderboard — that requires testing on a private (semi-private) set of tasks, to which the researchers did not have access. They used the public evaluation set of 400 tasks, which means the numbers may not be directly comparable with officially verified results.

Another limitation: the architecture was primarily tested on a single model (DeepSeek V3.2), although preliminary tests on Qwen3-235B show a similar improvement pattern. And last but not least — the results apply only to ARC-AGI-1, not to the newer and harder versions ARC-AGI-2 and ARC-AGI-3.

It's not about the size of the model, but how you use it

The main message of the Dartmouth research is clear and has far-reaching consequences: between a "dumb" one-shot query and expensive evolutionary search, there is a vast space that can be filled with thoughtful architecture. The authors showed that 52 percentage points of performance can be gained purely through how you use the model — not by which model you have, or how much you invested in it.

For the developer and researcher community in the Czech Republic and Europe, this is encouraging: you don't need to wait for the next generation of models from OpenAI or Google. With what's available today — and with smart architecture — you can achieve results that a year ago required hundred-million-dollar investments.