ARC-AGI Living Survey: Benchmarks & Insights
- ARC-AGI Living Survey is a dynamic benchmark ecosystem that tracks the evolution of AI reasoning, compositional learning, and interactive task solving.
- It documents cross-generational methods, from static grid puzzles to interactive environments, highlighting shifts in evaluation rigor and strategy.
- The survey reveals a persistent gap between high synthetic benchmark performance and real-world adaptive intelligence, emphasizing trade-offs in methodology.
Searching arXiv for the cited ARC-AGI survey and benchmark papers to ground the article. arxiv_search query="ARC-AGI Living Survey benchmark ARC-AGI-2 ARC-AGI-3" max_results=10 arxiv_search query="(Pfister et al., 13 Jan 2025, Chollet et al., 17 May 2025, Vahdati et al., 9 Mar 2026, Chollet et al., 15 Jan 2026) ARC-AGI" max_results=10 arxiv_search query="(Liao et al., 5 Dec 2025) CompressARC ARC-AGI Without Pretraining" max_results=5 The “ARC-AGI Living Survey” denotes the evolving survey effort around the Abstraction and Reasoning Corpus benchmark family, treating ARC-AGI as a benchmark ecosystem rather than a single static dataset. In this literature, ARC-AGI is positioned as a measure of few-shot generalization on novel tasks, with successive benchmark generations used to probe abstraction, compositional reasoning, and efficient skill acquisition under minimal prior knowledge. The survey perspective is explicitly cross-generational: “The ARC of Progress towards AGI” presents a first cross-generation analysis of 82 approaches across ARC-AGI-1, ARC-AGI-2, ARC-AGI-3, and the ARC Prize 2024–2025 competitions, while the ARC Prize technical reports document annual shifts in method families, leaderboard dynamics, and benchmark design (Vahdati et al., 9 Mar 2026, Chollet et al., 2024, Chollet et al., 15 Jan 2026).
1. Benchmark lineage and survey scope
ARC-AGI-1 was introduced as a benchmark for “general, human-like fluid intelligence” through input-output grid transformations that require inferring a hidden rule from a few demonstrations and applying it to held-out test inputs. The tasks use discrete colored cells, grids up to , and up to 10 distinct colors, and were designed around three core properties: resistance to memorization and overfitting, minimal prior knowledge, and human solvability. The original dataset contains 400 training tasks and 400 evaluation tasks, with additional hidden sets of 100 Private Evaluation tasks and 100 Semi-Private Evaluation tasks (Chollet et al., 17 May 2025).
ARC-AGI-2 preserves the same input-output pair format but introduces a newly curated and expanded set of tasks aimed at “a more granular signal to assess abstract reasoning and problem-solving abilities at higher levels of fluid intelligence.” Its task design emphasizes more unique tasks, often larger grids, more objects and more concepts per task, and more frequent multi-rule and multi-step compositional reasoning. Human testing is a major part of its validation: the reported study includes 407 unique participants, 515 sessions, 1,848 unique task test pairs attempted, 13,405 total test-pair attempts, and 8,277 successful test-pair solves, or 62% of attempts. The final task partitions were calibrated so that mean human accuracy differed by at most 1 percentage point across Public, Semi-Private, and Private subsets (Chollet et al., 17 May 2025).
ARC-AGI-3 changes the problem class from static puzzle solving to interactive reasoning. In one account it is described as interactive, level-based, turn-based, with no explicit natural-language instructions and action costs that make blind exploration undesirable. In another, the Preview Challenge is framed as a suite of game-like environments where agents must discover mechanics through interaction, with 6 games total in the preview setting: public games ft09, ls20, vc33, and private games sp80, lp85, as66. This shift expands the survey’s scope from rule induction over static grids to exploration, memory, planning, and adaptation over trajectories (Rodionov, 6 May 2026, Rudakov et al., 30 Dec 2025).
2. Intelligence, skill, and the interpretation of ARC-AGI scores
A central theme of the living-survey literature is the distinction between benchmark skill and intelligence. One explicit formulation defines skill as “the ability to achieve a specific goal under specific known conditions” and intelligence as “the ability to create new skills that allow goals to be achieved under previously unknown conditions.” On this view, intelligence is a meta-skill, and the paper “Understanding and Benchmarking Artificial Intelligence: OpenAI’s o3 Is Not AGI” sharpens the criterion to: “An agent is the more intelligent, the more efficiently it can achieve the more diverse goals in the more diverse worlds with the less knowledge” (Pfister et al., 13 Jan 2025).
This distinction is used to interpret headline ARC results conservatively. The same paper treats OpenAI’s o3 score of 87.5% on the semi-private ARC-AGI test set as impressive but not evidence of AGI, emphasizing an estimated cost of about USD 346,000, or roughly USD 3,460 per task, and arguing that the benchmark’s format permits “massive trialling of combinations of predefined operations.” The criticism is structural: ARC tasks are small colored-grid transformation problems whose candidate rules can be repeatedly tested before submission, whereas “for most problems in the physical world and in the human domain, solutions cannot be tested in advance and predefined operations are not available” (Pfister et al., 13 Jan 2025).
The survey literature generalizes this concern into a broader diagnosis of current systems. The ARC Prize 2025 technical report argues that frontier AI reasoning performance remains “fundamentally constrained to knowledge coverage,” creating new forms of benchmark contamination, while the cross-generation survey argues that ARC-AGI is valuable precisely because it isolates skill-acquisition efficiency from mere prepared skill. The same survey’s central empirical finding is that program synthesis, neuro-symbolic, and neural approaches all exhibit 2–3x drops from ARC-AGI-1 to ARC-AGI-2, indicating persistent limitations in compositional generalization even as ARC-AGI-1 scores rise sharply (Chollet et al., 15 Jan 2026, Vahdati et al., 9 Mar 2026).
3. Method families on static ARC-AGI tasks
The living survey documents a broad methodological spectrum rather than a single dominant paradigm. Technical reports for ARC Prize 2024 and 2025 identify deep learning-guided program synthesis, test-time training, zero-pretraining deep learning, evolutionary program synthesis, and application-layer refinement loops as major families. A recurring pattern is that top systems no longer rely on one-shot prediction alone; instead they combine search, verification, adaptation, and candidate selection under strict pass@2 evaluation (Chollet et al., 2024, Chollet et al., 15 Jan 2026).
Several representative systems illustrate this landscape:
| System | Core mechanism | Reported result |
|---|---|---|
| Product-of-Experts with LLMs | augmentation-heavy generation and scoring with DFS | 71.6% on the public ARC-AGI evaluation set |
| SOAR | LLM-guided evolutionary search with hindsight finetuning | 52.00% ARC-test solved |
| CompressARC | inference-time MDL optimization, no pretraining | 20% accuracy on ARC-AGI-1 evaluation puzzles under pass@2 |
| ABPR | Prolog repair via declarative traces and APD | Pass@2 = 56.67% on ARC-AGI-2 public evaluation |
| Modality-Driven Search with Holistic Trace Judging | text/image/code search plus long-context judging | 72.9% on ARC-AGI-2 semi-private, 76.11% on public eval |
The product-of-experts solver fine-tunes a single LLM, generates candidates with a probability-thresholded DFS search, and re-scores them across ARC-valid augmentations using a product-of-experts aggregation. Its reported score is 71.6% two-guess accuracy, equivalent to 286.5/400 public evaluation tasks solved, with an average cost of about \$0.02 per ARC task assuming \$0.36/hour for an Nvidia 4090 (Franzen et al., 8 May 2025).
SOAR reframes ARC as a self-improving program-synthesis loop. It allocates 6k total program attempts per task—3k initial samples and 3k refinements—and then converts sampled and refined traces into supervised finetuning data via hindsight relabeling. After self-improvement, the reported ARC-test scores reach 36.25% for Qwen-2.5-Coder 7B, 42.75% for 14B, 44.37% for 32B, 44.87% for Qwen-2.5-72B, and 45.50% for Mistral-Large-2, with a final ensemble score of 52.00% ARC-test solved and 57.25% oracle performance (Pourcel et al., 10 Jul 2025).
CompressARC demonstrates a radically different regime: a 76K-parameter equivariant neural network with no pretraining and no training on the ARC-AGI training split. It optimizes a minimum-description-length surrogate directly on each target puzzle, starting from random initialization. The reported result is 20% accuracy on ARC-AGI-1 evaluation puzzles under the official pass@2 setting, with 34.75% on the training split after 2000 inference-time optimization steps, at a cost of about 20 minutes per puzzle for a single run (Liao et al., 5 Dec 2025).
ARC-AGI-2 has intensified the role of repair and selection. ABPR, or Abduction-Based Procedural Refinement, couples an LLM with a Prolog meta-interpreter that materializes execution into declarative tree-structured traces following Udi Shapiro’s Algorithmic Program Debugging. Using Gemini-3-Flash-Preview, it reports Pass@2 = 56.67%, compared with 34.03% for no correction and 49.17% for conversational self-correction on the same model family (Qiu et al., 20 Mar 2026). Land’s modality-driven solver instead treats text, image, and code as search operators and uses a long-context judge to compare full reasoning traces jointly; it reports 72.9% on the ARC Prize semi-private evaluation set at \$38.99 per task and 76.11% on the public evaluation set at \$19.69 per task (Land, 30 Jun 2026).
A distinct line of analysis studies what apparently strong ARC performance is actually measuring. The TRM technical note shows that for the ARC Prize TRM checkpoint, test-time augmentation and majority-vote ensembling raise Pass@1 from 29.25% under single-pass canonical inference to 40.00% in the 1000-sample voting pipeline, while replacing the correct puzzle ID with a blank or random token yields 0.00% accuracy. Its recursion-trajectory analysis reports 38.25% at step 1, 40.38% at step 2, 40.13% at step 3, and 40.50% at step 4, suggesting shallow effective recursion rather than deep iterative reasoning (Roye-Azar et al., 4 Dec 2025).
4. Representation, modality, and refinement as design variables
The living survey increasingly treats representation not as a preprocessing choice but as a central determinant of solvability. “How Modality Shapes Perception and Reasoning” explicitly separates perception from reasoning across nine modalities—row_only, col_only, ascii, json, image_14x14, image_15x15, image_16x16, image_17x17, and image_768x768—and reports that structured text yields precise coordinates on sparse features, images preserve 2D shape structure but are resolution-sensitive, and combining text with image improves downstream execution by about 8 perception points and about 0.20 median similarity. The best multi-modal combination, row_col_json_image, reaches a median execution similarity of about 0.69, compared with around 0.51 for text-only baselines (Wen et al., 11 Nov 2025).
This concern with representation aligns with augmentation-heavy search systems. The product-of-experts approach uses ARC-valid symmetries, color permutations, and example-order permutations throughout training, generation, and scoring, effectively turning one LLM into an ensemble over equivalent task views. Its DFS procedure enumerates high-probability candidates under a threshold , with the final reported public ARC result using and 16 augmentations for scoring (Franzen et al., 8 May 2025).
ARC-AGI-2 methods have also reframed refinement itself. ABPR argues that code repair should be driven by semantic bug localization over declarative execution traces rather than by conversational “plausible reasoning,” and its strongest ablation result is that declarative execution traces are the main driver of improvement (Qiu et al., 20 Mar 2026). Land’s solver arrives at a complementary conclusion from the opposite direction: its strongest negative results show that step-by-step templates, forced reasoning stages, domain-specific heuristics in the prompt, strict structured outputs, and iterative prompt refinement all reduce performance by narrowing the hypothesis space and degrading diversity. In that system, holistic judging yields +7 solved instances over majority voting on the public eval run, all seven being minority recoveries, and judge synthesis adds +1 additional solved instance (Land, 30 Jun 2026).
A plausible implication is that ARC-AGI-2 has made candidate selection and hypothesis diversity at least as important as candidate generation. This interpretation is strongly supported by methods that succeed either by comparative judging across heterogeneous traces or by tightly constrained clause-level repair guided by execution semantics.
5. Benchmark contamination, evaluation rigor, and resampleable task families
The survey literature is unusually explicit about evaluation failure modes. ARC Prize 2024 identifies several ARC-AGI-1 limitations: a private evaluation set of only 100 tasks, repeated reuse of the private set across competitions, anecdotal evidence of inconsistent difficulty across splits, and strong dependence of search-based results on compute budget. ARC Prize 2025 extends this diagnosis by arguing that contemporary systems can exploit “knowledge coverage” when the task domain is sufficiently represented in pretraining corpora and a verifiable feedback signal is available (Chollet et al., 2024, Chollet et al., 15 Jan 2026).
ARC-AGI-2 was explicitly designed to counter some of these problems. Its motivation includes non-generalizable solutions via brute-force or exhaustive program search, lack of reliable first-party human baselines, saturation below the full human range, and leakage due to repeated reuse of hidden tasks. Competition evaluation therefore requires solving 240 previously unseen ARC-AGI-2 tasks—120 Semi-Private and 120 Private—within a 12-hour wall-clock window, offline, with no internet access, on four NVIDIA L4 GPUs, and the grand prize threshold is at least 85% accuracy on the hidden Private Evaluation set (Chollet et al., 17 May 2025).
A more structural response is ARC-TGI, the ARC Task Generators Inventory. ARC-TGI treats an ARC task as a task family rather than a frozen episode, implemented by compact Python generators with create_input, transform_input, and create_grids. Its defining feature is task-level constraints that ensure training examples collectively expose the variation needed to infer the underlying rule and avoid test-only cues or degenerate shortcuts. Each sampled task is paired with natural-language input and transformation reasoning chains and partially evaluated Python code implementing sampling, transformation, and episode construction. The released suite contains 461 generators covering 180 ARC-Mini tasks, 215 ARC-AGI-1 tasks, and 66 ARC-AGI-2 tasks (Lehmann et al., 5 Mar 2026).
The cross-generation survey makes evaluation rigor itself a central survey variable. It reports that, after filtering for evaluation rigor, only 59 of 82 surveyed approaches remain for quantitative comparison, and it repeatedly warns that small task subsets can inflate apparent progress. This suggests that the living-survey project is not only cataloguing methods; it is also standardizing what kinds of evidence count as credible progress on ARC-AGI (Vahdati et al., 9 Mar 2026).
6. Interactive reasoning in ARC-AGI-3 and future directions
ARC-AGI-3 extends the benchmark philosophy from static abstraction to interactive reasoning. The ARC Prize 2025 technical report previews it as introducing “interactive reasoning challenges that require exploration, planning, memory, goal acquisition, and alignment capabilities,” and the cross-generation survey identifies ARC-AGI-3 as the clearest evidence that current systems do not yet support robust exploration, persistent memory, goal inference, or world-model induction (Chollet et al., 15 Jan 2026, Vahdati et al., 9 Mar 2026).
Two early baselines illustrate the emerging method space. “Graph-Based Exploration for ARC-AGI-3 Interactive Reasoning Tasks” is training-free and maintains a directed graph of explored states and action-induced transitions, using image segmentation, status-bar masking, salience-based action grouping, state hashing, and shortest-path navigation to frontier states. On the ARC-AGI-3 Preview Challenge, the full method solves 19 total levels under a 4,000 interactions/game cap, achieves a median of 16 private-game levels and 14 public-game levels solved across 5 runs in the full 8-hour setting, and ranks 3rd on the private leaderboard with 12 private-game levels solved in the official challenge evaluation (Rudakov et al., 30 Dec 2025).
“Executable World Models for ARC-AGI-3 in the Era of Coding Agents” instead uses a coding agent to maintain an executable Python world model verified against previous observations and refactored toward simplicity as a practical proxy for an MDL-like bias. Evaluated on all 25 public ARC-AGI-3 games with fresh-agent playthroughs, it reports a mean per-game RHAE of 32.58%, a median per-game RHAE of 14.65%, 7 of 25 games fully solved, 6 of 25 games with RHAE greater than 75%, and 106 of 209 attempted levels solved across 29 recorded runs (Rodionov, 6 May 2026).
These ARC-AGI-3 results remain far below human performance, and the survey literature treats that gap as substantive rather than incidental. The cross-generation survey reports that ARC-AGI-3 best AI performance is 13%, while humans maintain near-perfect accuracy across versions, and interprets the interactive gap as evidence that compositional reasoning and interactive learning remain unsolved (Vahdati et al., 9 Mar 2026).
A separate future direction comes from the critique of ARC-AGI as an AGI benchmark. The proposal is to shift from benchmarking skills on a fixed dataset to benchmarking intelligence as adaptive performance in unfamiliar environments: many different worlds, initially unknown, each with some regularities but not human-familiar, different kinds of goals, limited prior knowledge, efficiency-sensitive scoring, and reduced opportunity for massive trialling of candidate solutions. The examples given include a Mars simulation, a four-dimensional alien gas planet with chemical production tasks, a digital strategy game against human players, and a world where the agent must predict future states without direct manipulation (Pfister et al., 13 Jan 2025).
In this sense, the ARC-AGI Living Survey documents both progress and boundary conditions. It records sharp improvements in ARC-AGI-1, substantial but still fragile gains on ARC-AGI-2, and a major competence gap on ARC-AGI-3. It also shows that the field has converged on several recurring design patterns—test-time adaptation, refinement loops, explicit verification, modality-aware representation, and cost-sensitive evaluation—while leaving open the deeper question of whether these patterns amount to efficient creation of new skills in diverse unknown worlds.