ARC-AGI-2: Advanced Abstract Reasoning Benchmark

• ARC-AGI-2 is a benchmark with rigorously designed, human-calibrated tasks for evaluating abstract reasoning, compositional generalization, and problem solving in AI.
• It employs a multi-stage task construction and empirical difficulty calibration, using extensive human screening to ensure robust evaluation.
• Performance drops in state-of-the-art AI models highlight intrinsic challenges in multi-rule and multi-step compositional reasoning across various paradigms.

ARC-AGI-2 is an upgraded benchmark in the Abstraction and Reasoning Corpus series, specifically designed to evaluate and drive progress in abstract reasoning, compositional generalization, and problem-solving abilities of artificial intelligence systems. Building directly on the task pair format and "easy for humans, hard for AI" principle of its predecessor ARC-AGI-1, ARC-AGI-2 introduces a rigorously constructed, human-calibrated, and significantly more challenging task suite whose granularity and complexity expose multiple bottlenecks in current frontier AI approaches (Chollet et al., 17 May 2025).

1. Origins, Motivation, and Design Principles

ARC-AGI-2 was created in response to well-identified limitations of ARC-AGI-1. While ARC-AGI-1 (2019–2024) utilized small (≤30×30) colored-grid puzzles with a handful of I/O examples and demanded only “Core Knowledge” (objectness, counting, basic geometry), over time it became increasingly tractable to massive compute-based brute-force strategies and susceptible to information leakage from repeated test set use. Human benchmarks were also fragmentary and inconsistent (Chollet et al., 17 May 2025).

ARC-AGI-2 preserves the input–output pair format and core evaluation philosophy but expands to deliver:

1. Lower brute-force susceptibility (through deeper compositionality, context, and symbol grounding).
2. Extensive human calibration (407 participants, 1,848 test pairs, over 13,000 attempts).
3. Careful difficulty stratification (subsets sampled to match human solve-rate distributions).
4. A broader difficulty spectrum, providing finer-grained measurement of AI progress.

Underlying construction was a multi-stage process: excess generation, human screening (with a defined full/partial credit regime), human-in-the-loop inclusion thresholds, empirical difficulty calibration (mean accuracy differences ≤1 percentage point between evaluation sets), redundancy pruning, and end-to-end validation by external and internal solvers. The resulting evaluation test pairs averaged a 75% solve rate per task for humans, with average individual accuracy at 66% (Chollet et al., 17 May 2025).

2. Cognitive and Computational Demands of ARC-AGI-2 Tasks

ARC-AGI-2 advances the state of AI benchmarking by emphasizing task categories that require:

• Multi-Rule Compositional Reasoning: Simultaneous application of two or more independent rules (e.g., cropping, rescaling, and object matching in a single task).
• Multi-Step Compositional Reasoning: Tasks where each transformation step depends intricately on prior step output, precluding end-to-end shortcut guesses.
• Contextual Rule Application: Rule execution that is modulated by context-specific gating conditions (akin to conditional control flow).
• In-Context Symbol Definition: On-the-fly symbol grounding, where objects within a given instance are assigned task-specific semantics only discoverable from demonstrations.

This categorical expansion sharply reduces the feasibility of exhaustive enumeration, disrupts simple local-pattern fitting strategies, and significantly increases the cognitive and representational demands on both symbolic and neural solvers (Chollet et al., 17 May 2025).

3. Evaluation Protocols, Human Baselines, and AI Performance

ARC-AGI-2 enforces strict parity in evaluation practices:

• Humans: Extensive live testing yielded a 62% full-solve rate on test pairs, a median solve time of 2.2 minutes, and no significant demographic performance variance. Every evaluation task was solved by at least two participants within their first two attempts.
• AI Systems: The main metric is full-task accuracy—models must be correct on all test pairs to score a solve. Public, semi-private, and private evaluation splits are calibrated to the same human difficulty distribution (≤1 pp mean solve-rate difference).

State-of-the-art models at the time of release exhibited a severe drop when transitioning from ARC-AGI-1 to ARC-AGI-2, with no system exceeding a 5% accuracy "noise threshold" on the semi-private set. For example:

Model	ARC-AGI-1	ARC-AGI-2
o3 (Medium)	53.0%	3.0%
ARChitects (’24 winner)	56.0%	2.5%
Claude 3.7 (8 K)	21.2%	0.9%

This 2–3× "compositional collapse" generalizes across paradigms: program synthesis, neuro-symbolic, and neural approaches all show analogous degradation when moving from ARC-AGI-1 to ARC-AGI-2, evidencing fundamental limitations in current approaches to compositional generalization (Vahdati et al., 9 Mar 2026).

4. Task Construction Methodology and Dataset Partitioning

Task construction in ARC-AGI-2 followed a multi-stage, human-guided protocol to ensure robustness and rule diversity:

• Excess generation (mixture of new and legacy, unused ARC-AGI-1 candidate tasks).
• Stringent human screening for inclusion based on solve rate and number of unique solvers.
• Empirical, quantile-based difficulty matching across public, semi-private, and private subsets; mean accuracy differences strictly limited to ≤1 pp.
• Redundancy elimination using consensus-based reviews.
• Manual correction of minor artifacts and logic errors following end-to-end solution verification by internal and external validators.

The benchmark thus achieves representativeness and difficulty-spectrum continuity, and preferentially allocates new tasks to the private split to minimize prior knowledge (Chollet et al., 17 May 2025).

5. Key Empirical Findings and Current Methodological Landscape

ARC-AGI-2 has substantiated several core insights:

• Tasks requiring deliberate, multi-rule, and multi-stage reasoning yield a marked increase in difficulty relative to one-shot pattern matching.
• Rigorously measured, first-party human baselines provide reliable calibration for both researchers and competition participants.
• Symbolic and neural approaches alike—when pushed to multi-step compositionality and symbol grounding—exhibit acute deficits in generalization, even at trillion-scale parameter counts and thousands of synthetic demonstration examples (Vahdati et al., 9 Mar 2026).
• Winning approaches in the ARC Prize 2025 converged on refinement loop paradigms: iterative "generate → verify → correct" cycles at the level of program synthesis, model weights, or both. Such loops included evolutionary program synthesis, application-layer harnesses around commercial LLMs, and gradient-based zero-pretraining pipelines (e.g., TRM, CompressARC) (Chollet et al., 15 Jan 2026). Nevertheless, no entrant demonstrated meaningful advances in abstraction independent of knowledge coverage and scale.

6. Impact, Core Bottlenecks, and Future Directions

ARC-AGI-2's design and empirical outcomes expose multiple, paradigm-agnostic research bottlenecks:

• Compositional Search Explosion: The hypothesis space grows exponentially with transformation chain depth ($O(n^d)$ for $n$ primitives, depth $d$), overwhelming both heuristics and beam/discrete search methods (Vahdati et al., 9 Mar 2026).
• Symbol Grounding and Knowledge-Bound Reasoning: Leading systems rely on synthetic data, large corpora, and prior demonstration coverage. They lack mechanisms to discover grounded, semantically reusable primitives autonomously.
• Efficiency Limitation: State-of-the-art models require orders of magnitude more compute per task than humans (e.g., $>$ a thousand times higher for comparable accuracy).
• Input Representation and Multimodal Reasoning: Systematic analysis confirms that perception and execution bottlenecks can arise from ill-matched encodings; hybrid vision–language approaches and cross-modal validation pipelines yield usable performance gains but fall short of robust generalization (Zhang et al., 19 Nov 2025, Wen et al., 11 Nov 2025).

The research community has identified high-priority directions:

• Hierarchical decomposition architectures to reduce search complexity.
• Meta-learning of refinement strategies, moving beyond hand-coded search curricula.
• Compression-guided objectives (e.g., Minimum Description Length) to favor simple, reusable rules over memorization.
• Explicit interactive environments, as planned for ARC-AGI-3, to eliminate the brute-force "testing loophole" and enable direct measurement of agent efficiency, alignment, and exploration (Pfister et al., 13 Jan 2025, Chollet et al., 15 Jan 2026).

7. Benchmark Structure and Broader Relevance

ARC-AGI-2 is the centerpiece of the ongoing ARC Prize and industry evaluation, with its private leaderboards and rigorously enforced splits forming the new standard for open and closed-source model assessment. Out-of-distribution robustness, alignment with human-level generalization, and reproducibility are explicitly foregrounded. The transition to ARC-AGI-2 signals a shift in the community towards benchmarks that expose the limits of pattern recognition and brute-force exploitation, steering research toward genuinely human-like, flexible reasoning (Chollet et al., 17 May 2025, Chollet et al., 15 Jan 2026).

Table: ARC-AGI-2 Benchmark Overview

Dimension	Detail
Input format	2–5 I/O grid pairs per task, up to 30×30, 10 fixed colors
Task types	Multi-rule composition, sequential reasoning, context, symbol definition
Evaluation splits	Public, semi-private, strict private (calibrated ≤1 pp in human accuracy)
Human solve rate	Task avg. 75%; individual avg. ≈66%
SOTA AI accuracy	None >5% on 2025 release; top Kaggle 2025 ≈24% (private set)
Core principles	“Easy for humans, hard for AI”; resists brute-force; no prior knowledge

ARC-AGI-2 thus operationalizes a new bar for "general fluid intelligence" benchmarks: strictly human-calibrated, compositionally demanding, and explicitly resistant to brute-force and scale-centric AI approaches, directing the field toward open, reproducible, and meaning-based reasoning progress (Chollet et al., 17 May 2025, Vahdati et al., 9 Mar 2026, Chollet et al., 15 Jan 2026).