ARC-BENCH: AI Benchmark for Compositional Reasoning

• ARC-BENCH is an AI benchmark that assesses general intelligence with grid transformation tasks, using a few input-output examples to infer novel rules.
• It challenges systems by requiring analogical reasoning, program induction, and spatial abstraction, with tasks featuring complex grid patterns and up to 10 distinct symbols.
• Recent methodological advances, including deep-guided program synthesis and hybrid ensembles, have pushed performance to over 55% accuracy in the ARC Prize 2024 competition.

ARC-BENCH is an artificial intelligence benchmark introduced to measure generalization and compositional reasoning in novel task domains, without explicit task-specific training. Designed and maintained by François Chollet and further documented in the context of the ARC Prize 2024 competition, ARC-BENCH (also referred to as ARC-AGI) centers on assessing "general intelligence"—the capacity of a system to solve tasks it has never encountered, based solely on a few input–output demonstrations. Unlike benchmarks that test pattern recognition or language modeling, ARC-BENCH presents unique challenges in analogical reasoning, program induction, and spatial abstraction. It has become a prominent, unsolved benchmark in the field of artificial general intelligence (AGI), actively stimulating innovation in learning algorithms, program synthesis, and hybrid neuro-symbolic frameworks (Chollet et al., 2024).

1. Formal Definition and Structure

ARC-BENCH tasks are defined as input–output grid transformations, with each task $T$ consisting of a small set of demonstration pairs $D = \{(x_1, y_1), \ldots, (x_k, y_k)\}$ (typically $k=3$), where $x_i$ and $y_i$ are discrete-valued $(r \times c)$ grids, and a set of test inputs $\{x_t\}$. The fundamental objective is to induce a function $f_T$ from $D$, such that $f_T(x_t)$ yields the correct test output $D = \{(x_1, y_1), \ldots, (x_k, y_k)\}$0. Each task is constructed to demand reasoning based on "Core Knowledge" priors—objectness, topology, relational rules, spatial transformations, or simple arithmetic—but deliberately excludes real-world or linguistic background knowledge.

No two tasks in ARC-AGI-1 share the same generative rule. Task grids can be up to $D = \{(x_1, y_1), \ldots, (x_k, y_k)\}$1, with each cell taking one of 10 distinct values (colors/symbols). The evaluation split comprises $D = \{(x_1, y_1), \ldots, (x_k, y_k)\}$2 tasks (e.g., $D = \{(x_1, y_1), \ldots, (x_k, y_k)\}$3 in the private set), where the performance metric is strict accuracy: $D = \{(x_1, y_1), \ldots, (x_k, y_k)\}$4 with each test input permitting up to two guesses. Human performance on this metric is approximately $D = \{(x_1, y_1), \ldots, (x_k, y_k)\}$5–$D = \{(x_1, y_1), \ldots, (x_k, y_k)\}$6 on the private eval set (Chollet et al., 2024).

2. Dataset Composition and Evaluation Protocol

The canonical ARC-AGI-1 benchmark includes 1,000 tasks divided into:

Split	#Tasks	Description
Public Train	400	"easy", full public access
Public Eval	400	"hard", full public access
Semi-private Eval	100	"hard", partial API access
Private Eval	100	"hard", fully withheld

Each task is unique in specification and solution, with the private-eval split reserved for leaderboard rankings and to mitigate overfitting.

Submissions must return outputs for each test input, given only the demonstration set $D = \{(x_1, y_1), \ldots, (x_k, y_k)\}$7 for that task. There is no standardized loss function; successes and failures are determined by full-match accuracy. To discourage leaderboard hacking and overfitting, the principal benchmark scores are reported exclusively on the private-eval split (Chollet et al., 2024).

3. Methodological Advances and ARC Prize 2024

Between 2019 and early 2024, pure deep-learning approaches (e.g., direct LLM prompting) achieved 0–1% accuracy, while brute-force domain-specific language (DSL) search methods reached up to 20.6%. The "ARCathon" iterations incrementally improved results to just above 30%. The initiation of the ARC Prize 2024 competition catalyzed breakthroughs, pushing top open-source solutions to 53.5% (ARChitects on Kaggle private-eval), and privately to 55.5% (Chollet et al., 2024).

Notable methodological drivers included:

• Deep Learning–Guided Program Synthesis: LLMs (e.g., GPT-4o, Code Llama) are used to generate, rank, and debug candidate programs, typically in Python or a purpose-built DSL. Programs are evaluated for consistency with demonstrations, and LLMs may be prompted iteratively (including optional "debug" prompts) to achieve high demo accuracy. This hybridizes symbolic and neural paradigms.
• Test-Time Training (TTT)/Transduction: On-task fine-tuning or adaptation of networks using demonstration pairs. This may entail LoRA or full weight updates at inference time, directly optimizing predictions for the specific demonstrations. Approaches leverage ARC-like synthetic data (BARC, RE-ARC), 2D-aware transformer architectures, and extremely rapid adaptation.
• Hybrid Ensembles (Induction + Transduction): Ensembles select between program synthesis and TTT subsystems based on confidence, exploiting their complementary strengths and eliminating single-method performance plateaus.

The close tracking of semi-private (ARC-AGI-Pub) and private (Kaggle) leaderboard progressions demonstrates that technique advancements, not computation scale, led to increased SOTA (Chollet et al., 2024).

4. Open-Source Tooling and Ecosystem

ARC-BENCH has stimulated the development of a robust ecosystem of open implementations and auxiliary resources:

• ARC-DSL (Michael Hodel): Minimal DSL and baseline solver (arc-dsl), enabling canonical programmatic reasoning.
• RE-ARC: Infinite procedurally generated ARC-like tasks for augmentation.
• BARC: Large-scale (400K) synthetic task generators aiding in pretraining and robustification.
• ConceptARC: Diagnostic suite targeting specific abstract concepts.
• arcsolver: Claude-based, object-centric solver.
• ARC Gym: RL- and search-oriented experimentation framework.
• Arckit: Python/CLI utilities for exploration and evaluation.

In addition, numerous interactive web visualizers and hundreds of publicly available Kaggle notebooks facilitate broad participation and methodological transparency (Chollet et al., 2024).

5. Leaderboard Performance and State-of-the-Art

Significant progress has occurred since 2020, as summarized in the following key milestones:

Year	SOTA Private Eval (%)	Key Methodologies
2020	20.6	DSL brute-force search
2022	28.5	Hybrid DPSL, expanded heuristics
2023	30.45	ARCathon2, program induction variants
2024	34.4 (pre-Prize)	Hybridizing search and learned models
2024	55.5 (Prize)	DL-guided synthesis, TTT, ensembles

Best open-source solutions in the ARC Prize 2024 included ARChitects (53.5% private-eval), while the best reported but non-open-sourced method (MindsAI) reached 55.5%. Performance on public and semi-private sets kept pace, implying minimal impact from tailored meta-learning or secret compute scaling (Chollet et al., 2024).

Notably, state-of-the-art LLMs (GPT-4o, Claude 3.5 Sonnet) perform poorly (pass@1: 5–14% semi-private eval), reinforcing the necessity of explicit induction and algorithmic composition to achieve competitive results.

6. Limitations and Prospective Directions

Principal limitations of the current ARC-BENCH (ARC-AGI-1):

• Eval Set Saturation: The private-eval (100 tasks) is now partially compromised due to score leakage; brute-force DSL search already solves about 49% of tasks.
• Discriminative Ceiling: Many tasks no longer serve as discriminators for frontier AGI models, being solvable by earlier methods.
• Difficulty Calibration: Inconsistent sampling of task difficulty across splits complicates rigorous comparison of public, semi-private, and private performances.

In response, development of ARC-AGI-2 is underway, aiming for larger, better-balanced semi-private and private splits (~300–400 tasks), new difficulty stratification, and curated resistance to brute-force solution strategies. Methodologically, leading proposals include:

• Specialist deep models for guiding structured search (e.g., AlphaProof-like policies).
• Latent Program Networks for more efficient adaptation.
• Hybrid neuro-symbolic architectures combining abstraction learning, constraint satisfaction, and dynamic program induction.

The overarching ARC Prize initiative is structured to incentivize both open innovation and industrial participation, maintaining its "north star" of 85% private-eval attainment and annual competition cycles until this threshold is reached with a fully open solution (Chollet et al., 2024).

7. Impact and Research Significance

ARC-BENCH has established itself as a central challenge for the AI, ML, and AGI research communities, redirecting focus from memorization and imitation to compositional, algorithmic generalization. Its formulation, evaluation protocol, and competition infrastructure jointly motivate research into inductive biases, program synthesis, self-adaptation, and truly general-purpose learning architectures. It provides a reproducible, nontrivial testbed for open benchmarking and empirical progress in intelligence science (Chollet et al., 2024).