---
title: ARC-AGI-1 Benchmark for AGI Evaluation
url: https://www.emergentmind.com/topics/arc-agi-1-benchmark
type: topic
---

# ARC-AGI-1 Benchmark for AGI Evaluation

The ARC-AGI-1 Benchmark is a rigorous testbed for evaluating artificial general intelligence (AGI) systems’ ability to perform abstract reasoning and rule induction from minimal examples. It comprises diverse, hand-crafted grid-based tasks designed to mirror “learning how to learn” in humans, specifically focusing on few-shot conceptual rule induction and transfer to novel problems. ARC-AGI-1 tasks have become central to measuring out-of-distribution generalization, compositional reasoning, and cognitive plausibility for frontier foundation models [2511.15703].

## 1. Formal Definition and Objectives

ARC-AGI-1 presents each solver with a set of tasks $\{\tau_j\}$, each defined by $K$ input–output training pairs and a hidden test input:

- Each grid $m \in \mathbb{Z}^{H \times W}$ for $H,W \leq 30$, with cell values $0 \ldots 9$.
- Task $\tau$ consists of $\{(m_i^{in}, m_i^{out}) | i = 1 \ldots K\}$ and one $m_\text{test}^{in}$.
- The underlying transformation is always a deterministic rule $r: \mathbb{Z}^{H \times W} \to \mathbb{Z}^{H \times W}$, which must be inferred from a handful of examples.
- Modalities: 
    - Textual encoding: $t = \mathcal{T}(m)$
    - Visual encoding: $i = \mathcal{V}(m)$
    - Both $\mathcal{T}$ and $\mathcal{V}$ are invertible.

The principal evaluation metric is exact match (Pass@1) accuracy over a holdout set:

$$
\mathrm{Acc} = \frac{1}{N} \sum_{j=1}^N \mathbf{1}\left[m_\text{pred}^{(j)} = m_\text{gt}^{(j)}\right]
$$

This assesses whether the model's predicted output grid $m_\text{pred}$ matches the ground truth $m_\text{gt}$ for each test input, over $N$ tasks.

## 2. Benchmark Structure, Modalities, and Metrics

ARC-AGI-1 contains 1,000 total tasks with 400 public “train,” 400 public “eval,” and 200 private tasks, each requiring solvers to generalize across transformations such as symmetries, counting, object manipulation, and spatial relations [2412.04604].

- Each task only shares the high-level structure—no two have the same transformation logic.
- Input/output grids are either serialized as text sequences or rendered as color-coded images.
- Base metrics:
    - Pass@1 (single guess per test)
    - Two-guess accuracy (for competitions): proportion of tasks solved in up to two guesses.

Table: Task Split and Metrics

| Split        | # Tasks | Main Metric    |
|--------------|--------:|---------------|
| Train        |     400 | Pass@1        |
| Public Eval  |     400 | Pass@1, 2-guess |
| Private Eval |     200 | Pass@1           |


Significance: ARC-AGI-1 is specifically designed to require generalization on previously unseen rules, precluding brute-force memorization and stressing abstract reasoning [2501.07458].

## 3. Algorithmic Paradigms and Synergistic Strategies

ARC-AGI-1 has catalyzed the development of several advanced solver paradigms.

### Program Synthesis and Evolutionary Search

- Deep learning-guided program synthesis utilizes code LLMs to generate candidate programs in small DSLs, executed and filtered against training examples [2412.04604, 2507.14172].
- SOAR (Self-Improving Operators for Automated program Refinements) iteratively fine-tunes LLMs via hindsight learning from search traces, enabling dramatic gains in solve rates (up to 52% with ensembles) [2507.14172].

### Vision-Language Synergy and Self-Correction

Vision-Language Synergy Reasoning (VLSR) and Modality-Switch Self-Correction (MSSC) are two synergistic inference strategies [2511.15703]:

- VLSR: Decomposes reasoning into Phase 1 (visual rule summarization) and Phase 2 (textual rule application), using vision for holistic pattern abstraction and language for element-wise rule execution.
- MSSC: Cross-modal error correction loop, critiquing text outputs via vision to overcome confirmation bias, yielding monotonic accuracy improvements.

Pipeline pseudocode:

```python
# Phase 1: Visual rule summarization
r_pred = LVLM_vision("Summarize rule", images_in, images_out)

# Phase 2: Textual application of rule
t_pred = LVLM_text("Apply rule", r_pred, text_grids, test_input_text)

# MSSC loop for self-correction
for r in range(N_max):
    is_valid = LVLM_vision("Judge output", test_input_image, predicted_image)
    if is_valid: break
    feedback = "Your output breaks [spatial pattern X]. Revise."
    t_pred = LVLM_text("Apply rule with feedback", feedback, ...)
```
- VLSR + MSSC: Average gain of 4.33% over text-only baselines across major models [2511.15703].

## 4. Empirical Results and Comparative Performance

Extensive benchmarking has established state-of-the-art and baseline scores:

- Frontier language models (GPT-4o, Gemini, Qwen3-VL, o4-mini): Text-only accuracy ranges from 8.25% to 42.25% [2511.15703].
- VLSR: +3.02% improvement on average.
- MSSC: +1.82% improvement.
- Combined VLSR+MSSC: Up to 4.33% over baseline.
- SOAR ensemble: 52% tasks solved [2507.14172].
- Product-of-Experts with LLMs and systematic data augmentations reaches 71.6% at $0.02 per task—surpassing average human baseline [2505.07859].
- Best private eval scores (ARC Prize 2024): ~55.5% [2412.04604].

Table: Pass@1 Accuracy for Main Approaches

| Model         | Text-Only | VLSR+MSSC | SOAR Ensemble | PoE+DFS |
|---------------|----------:|----------:|--------------:|--------:|
| GPT-4o        |     8.25% |    14.50% |         —     |    —    |
| Gemini-2.5-Pro|    35.00% |    42.25% |         —     |    —    |
| o4-mini       |    42.25% |    46.75% |         —     |    —    |
| Qwen3-VL      |    20.25% |    22.25% |         —     |    —    |
| SOAR Ensemble |     —     |     —     |      52.00%   |    —    |
| PoE+DFS       |     —     |     —     |        —      | 71.6%   |


## 5. Cognitive Modelling and Neurosymbolic Methods

Solutions informed by human cognition and symbolic reasoning have been explored:

- Vector Symbolic Algebra-based systems combine System 1 perceptual heuristics (object segmentation, fast similarity) and System 2 symbolic program inference (minimum-hitting set search, parameter induction) [2511.08747].
- Task composition leverages symbolic operations (e.g., Extract, Recolour, Grow), employing HRRs and SSPs for object and position representations.
- Sample efficiency and interpretability are high, but benchmark-level generalization remains limited (e.g., 10.8% on ARC-AGI-1 Train, 3.0% on Eval).

Neural Cellular Automata (NCA) variants (including EngramNCA) constitute developmental, self-organizing meta-models. These iterate local update rules across grids, showing promise in generalizing to novel grid sizes and patterns at low compute cost [2505.08778, 2506.15746].

## 6. Human Trajectories and Alignment

ARCTraj augments ARC-AGI-1 by recording and analyzing human object-level action trajectories on 400 training tasks [2511.11079]:

- Formalizes ARC as a finite-horizon Markov Decision Process with states as grid-object compositions and actions as symbolic transformation triplets.
- Provides a human baseline for explainability and aligns reinforcement learning, sequence modeling, and generative methods to real human reasoning.
- Shows that common selection biases (compact regions), color-attribution logic, and strategy grammars provide rich inductive signals for AGI model alignment.

## 7. Limitations, Open Questions, and Future Directions

Several limitations persist:

- Modal switching overhead for vision–language pipelines; visualization resolution bottlenecks for large grids.
- ARC-AGI-1’s tractability for brute-force program search with fixed DSL primitives diminishes its capacity to measure true abstraction (skills vs. intelligence) [2501.07458].
- Dataset overfitting risk due to limited private evaluation set size; index 49% of tasks are brute-force solvable [2412.04604].
- ARC-AGI-2 proposals: expand pools, rebalance difficulty, introduce procedural novelty metrics, track concept-specific progress.

Open future directions include:

- Joint multimodal training for vision–language models beyond zeroshot inference [2511.15703].
- LLM–NCA hybrid pipelines, adaptive search via neural priors, enhanced representation learning for symbolic compositionality [2505.08778].
- Universal benchmarks based on diversity-weighted, skill-generation metrics rather than fixed puzzle sets [2501.07458].
- Human-aligned learning strategies and intention prediction modules derived from ARCTraj [2511.11079].
  
In sum, the ARC-AGI-1 Benchmark remains the canonical AGI testbed for few-shot abstraction, requiring systems that synergize vision and language, symbol and neural computation, and human-like reasoning trajectories. Recent advances in evolutionary program synthesis, vision–language decomposition, and cognitive modelling have significantly shifted the frontier, though full AGI-level abstraction and transfer remain unsolved [2511.15703, 2505.07859, 2511.08747, 2505.08778, 2507.14172, 2412.04604, 2501.07458].

Source: https://www.emergentmind.com/topics/arc-agi-1-benchmark