---
title: 'ARC-Bench: Benchmark for ARC-AGI Tasks'
url: https://www.emergentmind.com/topics/arc-bench-55520b29-3c00-411c-8929-ecb0afea2c65
type: topic
---

# ARC-Bench: Benchmark for ARC-AGI Tasks

ARC-Bench denotes a benchmark construction paradigm for ARC-AGI-style abstraction and reasoning tasks in which evaluation is performed on resampleable task families rather than on a static set of hand-authored puzzles. In the ARC-TGI framework, ARC-Bench is defined as a rigorous, resampleable ARC benchmarking suite built from human-validated task generators, each of which samples diverse train/test episodes that preserve a latent rule while varying nuisance factors such as grid size, palette, and distractors [2603.05099]. The benchmark is motivated by longstanding limitations of static ARC collections—overfitting, dataset leakage, memorization of specific instances, and poor support for controlled generalization studies—and remains anchored in the original ARC-AGI problem setting, where a system must infer a transformation from a small set of demonstrations and produce exact-match outputs on novel test inputs [2603.05099] [2412.04604].

## 1. ARC-AGI context and the motivation for ARC-Bench

ARC-Bench is intelligible only in relation to ARC-AGI, the Abstraction and Reasoning Corpus benchmark for generalization on novel visual grid tasks. In ARC-AGI, each task provides a set of demonstration pairs $D=\{(x_i,y_i)\}$, with a median of three examples, together with one or more test inputs $X_{\text{test}}=\{x_j^\*\}$. Each grid is a rectangular array $G \in C^{H \times W}$ with $H,W \le 30$ and $C=\{0,1,\ldots,9\}$. The objective is to infer a transformation $f$ such that $f(x_i)=y_i$ for all demonstrations and then apply $f$ to each test input; a task is counted as solved only when every predicted test output exactly matches the ground truth [2412.04604]. In this literature, ARC-AGI is explicitly distinguished from AI2’s unrelated ARC science-exam dataset [2412.04604].

The original benchmark design emphasizes novelty: each ARC-AGI task follows different logic, the tasks are human-authored for diversity, and the intended priors are restricted to human Core Knowledge such as objectness, basic topology, and elementary arithmetic [2412.04604]. Human performance is correspondingly high: two individuals originally scored $97\%$ and $98\%$ on the private set and together solved all 100 tasks, while a 2024 NYU study reported that $99\%$ of public evaluation tasks were solved by at least one Mechanical Turk worker when 10 workers were assigned per task [2412.04604].

ARC-Bench arises from the observation that static ARC(-AGI) collections make progress hard to measure. The ARC-TGI paper identifies three recurring issues: overfitting and dataset leakage on small, fixed puzzles; memorization of specific instances instead of inductive rule learning; and the inability to perform controlled experiments that vary one nuisance factor while preserving the underlying rule [2603.05099]. A central premise of ARC-Bench is therefore that an ARC task should be treated not as a single puzzle instance but as a task family: a distribution over solvable episodes sharing a latent rule.

## 2. Generator-based formulation in ARC-TGI

The technical substrate of ARC-Bench is ARC-TGI, the ARC Task Generators Inventory. ARC-TGI turns each ARC(-AGI) task into a compact Python generator that subclasses an abstract `ARCTaskGenerator` and implements three fixed-signature methods: `create_input(self, taskvars, gridvars) → np.ndarray`, which samples an input grid while randomizing nuisance factors; `transform_input(self, grid, taskvars) → np.ndarray`, which deterministically applies the latent rule to produce the output; and `create_grids(self) → (taskvars: Dict, train_test_data)`, which assembles a complete episode and enforces task-level constraints across examples [2603.05099].

Each sampled episode is exported as a solver-facing bundle. That bundle contains natural-language “input reasoning chain” and “transformation reasoning chain” templates instantiated from `taskvars` and `gridvars`; partially evaluated Python code for input sampling, transformation, and episode construction, with sampled variables inlined; and the train/test grids in ARC-JSON form [2603.05099]. The framework also includes optional helper libraries for consistent input construction, such as connected components, coloring, and densities, and solver-facing transformation primitives such as `GridObject(s)`, `find_connected_objects`, and geometric operations [2603.05099].

The formalization used for ARC-Bench makes the task-family perspective explicit. A generator is written as $G(\theta)$, where $\theta \sim p(\theta;\eta)$ encodes task-level parameters controlled by hyperparameters $\eta$, such as palette-size ranges, grid-size ranges, and allowed symmetries. A latent rule $r$ determines a deterministic mapping $f_r: X \to Y$ between input and output grid spaces. An episode sampled from $G$ is
$$
E=\{(x_i,y_i)\}_{i=1}^{n},
$$
with task-level constraints $C$ ensuring that training examples collectively expose the variations needed to infer $r$ and that test examples do not introduce unseen features. Formally, $y_i=f_r(x_i)$ for all $i$, and $C$ couples examples across the episode to guarantee solvability [2603.05099].

The canonical constraint formulation prioritizes coverage of salient attributes and train–test consistency:
$$
C:\ \text{diversity}(\{\phi(x_i)\}_{i=1}^n) \ge \delta\quad\text{and}\quad \forall i,\, y_i = f_r(x_i),
$$
where $\phi$ extracts salient attributes such as colors, shapes, positions, and sizes, and $\text{diversity}(\cdot)\ge\delta$ enforces variation sufficient to disambiguate the rule [2603.05099]. Reproducibility is handled by RNG seeding $s \in \mathbb{N}$ in the generator call, with $\theta \sim p(\theta;\eta,s)$ and grid-level randomness similarly conditioned on $s$; the framework’s `create_task` wrapper captures $\theta$ and $s$ into the exported witness program, enabling deterministic regeneration and verification [2603.05099].

## 3. Episode-level constraints, human refinement, and local verification

A defining feature of ARC-Bench is its treatment of ARC episodes as designed sets of examples rather than as independent samples. The ARC-TGI paper argues that independent per-example sampling often fails because it can omit critical variation, include test-only cues, or degenerate to trivial shortcuts such as identity or constant outputs [2603.05099]. ARC-Bench therefore elevates `create_grids` to a first-class stage for cross-example coupling.

The enforced constraint types are explicit. They include train–test consistency, such as “no unseen colors or shape classes at test”; input-construction constraints, such as requiring objects to be connected under 4-/8-connectivity and bounding boxes to fit target placements; disambiguating coverage, such as requiring at least two sizes or multiple positions or orientations across training pairs; and rejection sampling until $C$ is satisfied, supplemented by framework-level invariants and shortcut screening [2603.05099]. This design directly addresses a common misconception that a large number of randomly generated ARC-like instances is sufficient for faithful benchmarking; the ARC-TGI formulation instead treats cross-example disambiguation as a benchmark requirement.

Human refinement is another central element. Generator authoring is described as human-in-the-loop: contributors analyze each task, identify `taskvars` and `gridvars`, author natural-language reasoning templates and episode-level constraints, draft generator code either with LLM assistance or manually, and iteratively resample and visualize outputs until grids and reasoning traces remain correct and natural under variation [2603.05099]. The stated purpose is to avoid subtle rule violations and misaligned explanations.

The resulting exports are self-verifying. The inlined transformation program must reproduce stored train/test outputs exactly; invariant checks verify well-formed grids and declared restrictions such as “no unseen colors at test”; and optional shortcut screening filters identity or constant-output degeneracies unless they are intended by the latent rule [2603.05099].

The paper’s concrete examples illustrate why these mechanisms matter. In the within-family generalization example for stacked colored segments (“Taskbeb8660c”), `taskvars` are fixed per episode and include vertical stacking direction $\text{dir} \in \{\text{top},\text{bottom}\}$, horizontal alignment $\text{align} \in \{\text{left},\text{right}\}$, a palette, and object size roles; `gridvars` vary per example and include the number of segments, segment lengths, colors, and positions before transformation. The transformation chain specifies stacking all segments along the chosen edge, aligning the stack to the specified side, and preserving segment colors, lengths, and orientation. The rationale for the constraint is explicit: if all training segments share the same length or color, the alignment/stacking rule is ambiguous, so $C$ enforces diversity over lengths and colors and forbids unseen colors at test [2603.05099]. A second example (“Task3befdf3e”) enforces that training include at least one $3 \times 3$ object and one $5 \times 5$ object when the transformation differs by size; otherwise the test episode could become unsolvable from the demonstrations [2603.05099].

## 4. Benchmark construction protocols, splits, and evaluation metrics

ARC-Bench is not a single immutable dataset but a benchmark construction protocol over a set of generators. The recommended procedure begins with family selection: define a suite $\mathcal{F} \subseteq$ generators aligned to target splits, for example ARC-AGI-1 train families for in-distribution evaluation and ARC-AGI-2 families for out-of-distribution generalization. To avoid leakage in generalization studies, train and test splits should use disjoint family sets [2603.05099].

The recommended sampling budget is $k$ episodes per family, with the paper’s practical default set to $k=50$ after a small sweep. Three split regimes are then described. In the ID split, for each $F \in \mathcal{F}_{\text{train}}$, one samples $k$ episodes and partitions them $50/50$ into train/test per family. In the OOD split, one holds out $\mathcal{F}_{\text{test}}$ as disjoint generators and samples $k$ test episodes per held-out family. In the cross-benchmark split, one evaluates on public ARC-AGI-1 eval after training on ARC-TGI families [2603.05099]. Practical guidance names two canonical suite constructions: **ARC-Bench-ID**, defined by selecting $\mathcal{F}_{\text{train}}$ from ARC-AGI-1 train families, sampling $k=50$ episodes per family, splitting $50/50$ train/test, and reporting `MacroAcc`, per-family `Acc_F`, and sample-efficiency curves $A(n)$; and **ARC-Bench-OOD**, defined by fine-tuning on ARC-TGI $\mathcal{F}_{\text{train}}$ and evaluating on held-out $\mathcal{F}_{\text{test}}$ from ARC-AGI-2 together with ARC-AGI-1 eval [2603.05099].

Exact reproducibility is treated as part of the benchmark definition. The protocol fixes a suite seed $S$, per-family seeds $s_F$, and per-episode seeds $s_{F,j}$ so that resampling is deterministic across implementations. The reporting recommendations further require publication of $S$, $\{s_F\}$, $\{s_{F,j}\}$, code versions, and any filtering criteria such as token-context limits [2603.05099].

The primary evaluation metrics are family-wise exact-match accuracy and macro-averaging across families. For a family $F$,
$$
\text{Acc}_F=\frac{1}{|E_F^{\text{test}}|}\sum_{(x,y)\in E_F^{\text{test}}}\mathbf{1}[\hat{f}(x)=y].
$$
The macro-average is
$$
\text{MacroAcc}=\frac{1}{|\mathcal{F}|}\sum_{F\in\mathcal{F}}\text{Acc}_F.
$$
Generalization to held-out families is measured by
$$
\text{Acc}_{\text{held-out}}=\frac{1}{|\mathcal{F}_{\text{test}}|}\sum_{F\in\mathcal{F}_{\text{test}}}\text{Acc}_F,
$$
with $\mathcal{F}_{\text{test}}$ disjoint from $\mathcal{F}_{\text{train}}$. Sample efficiency is reported through curves $A_F(n)$ versus the number $n$ of training demonstrations per episode, aggregated across families to study few-shot induction [2603.05099]. The paper also describes an optional reasoning-chain consistency metric, defined as an alignment rate between generated chains and solver predictions, for example the fraction of steps in the transformation chain whose predicates or object references are validated by the solver’s predicted program or inferred intermediate states [2603.05099].

Leakage control is formalized rather than left implicit. When measuring across-family generalization, episodes from the same generator should not be mixed across train and test. When measuring within-family generalization, the study should be labeled separately and use disjoint episode seeds. In all cases, generators’ constraints and invariants should be verified to prevent test-only features [2603.05099].

## 5. Coverage, baseline behavior, and empirical properties

The ARC-TGI release underlying ARC-Bench contains 461 generators spanning three sources: 180 ARC-Mini tasks, 215 ARC-AGI-1 tasks, and 66 ARC-AGI-2 tasks. The ARC-AGI-1 coverage is further partitioned into 200 train and 15 eval/test tasks, while the ARC-AGI-2 coverage is partitioned into 55 train and 11 eval/test tasks [2603.05099]. Because each generator defines a distribution rather than a single puzzle instance, the suite supports scalable sampling; the ARC-TGI-50N setting samples 50 episodes per family [2603.05099].

Distributional analyses in the ARC-TGI paper indicate that the generated families preserve dominant size modes of original tasks while expanding coverage through within-family variation. The paper also reports generator-by-model heatmaps that show stable difficulty ordering across models, with long-tail hard families preserved under resampling [2603.05099]. This is significant because it suggests that the benchmark is not merely generating interchangeable easy variants but is retaining family-specific structure that remains discriminative across systems.

Representative baseline numbers are reported directly on ARC-TGI-50N. Qwen3-30B reaches $21\%$ exact-match, while Claude Sonnet 4.5 averages approximately $50\%$, and the difficulty matrices exhibit stable, family-specific behavior under resampling [2603.05099]. The recommended reporting protocol therefore includes `MacroAcc`, `Acc_held-out`, generator-by-model heatmaps, and per-family bar plots, since aggregate accuracy alone can obscure the difficulty structure of the suite [2603.05099].

The paper also specifies a fine-tuning protocol for benchmarking transfer. The described LoRA configuration uses $r=64$, $\alpha=16$, dropout $=0.05$, 10 epochs, AdamW with learning rate $2\times 10^{-5}$, warmup $0.03$, weight decay $0$, and maximum context 14k tokens. Under this protocol, the paper reports significant ID gains for Phi-4 and Llama-3.1-8B, limited transfer to ARC-AGI-1 eval, and generator-specific improvement and decline patterns [2603.05099]. A plausible implication is that ARC-Bench exposes not only absolute performance but also heterogeneity in how models respond to synthetic resampling and family-level fine-tuning.

## 6. Positioning, alternate usages, and open issues

Relative to original ARC-AGI and ARC-Mini, ARC-Bench preserves the open-ended train/test format but adds resampling, solver-facing reasoning templates, partial-evaluation witnesses, and first-class episode-level constraints. This makes possible robustness sweeps and matched-distribution studies such as within-family versus across-family generalization that are infeasible on one-off puzzles [2603.05099]. Relative to generator frameworks such as ARC-DSL, ReARC, and ARC-GEN, the ARC-TGI formulation specifically adds step-by-step natural-language reasoning aligned to each sampled instance and a `create_grids` stage that emphasizes episode-level constraints, together with human validation and self-verifying exports [2603.05099].

The benchmark is also positioned in relation to newer synthetic reasoning suites. CellARC is explicitly presented as a complement and extension, not a replacement: it isolates local-rule induction in multicolor one-dimensional cellular automata, offers unlimited sampling and explicit difficulty knobs such as alphabet size, radius, Langton’s $\lambda$, coverage, and cell entropy, and is intended for use alongside ARC-Bench to disentangle generalization due to human priors from formal local-rule induction [2511.07908]. This suggests a broader research program in which ARC-Bench provides human-authored, object-centric abstraction tasks while CellARC provides tightly controlled studies of local-rule inference under reproducible complexity controls.

The term “ARC-Bench” is not fully standardized across the literature. In the GLM-4.5 paper, “ARC-Bench” does not refer to the ARC-TGI suite at all; it denotes the authors’ comprehensive evaluation suite for Agentic, Reasoning, and Coding capabilities, aggregating 12 benchmarks: TAU-Bench, BFCL v3, BrowseComp, MMLU-Pro, AIME 24, MATH-500, SciCode, GPQA, HLE, LiveCodeBench, SWE-bench Verified, and Terminal-Bench. In that usage, the aggregate ARC ranking is the model’s average across those 12 tasks, and representative scores reported for GLM-4.5 are $70.1\%$ on TAU-Bench, $91.0\%$ on AIME 24, and $64.2\%$ on SWE-bench Verified [2508.06471]. In encyclopedia usage, the surrounding citation context is therefore essential for disambiguation.

Several limitations remain open in the ARC-TGI-based conception of ARC-Bench. The paper notes coverage gaps despite the 461 released families, persistent long-tail difficulty, the need for stronger automatic checks for degeneracy and ambiguity, opportunities for reusable disambiguation templates and richer symmetry and compositional constraints, and unresolved questions about community-standardized budgets, reporting conventions, and human-solvability audits such as time-to-solve and inter-annotator agreement [2603.05099]. This suggests that ARC-Bench is best understood not as a finished benchmark artifact but as a procedural, constraint-aware framework for evaluating abstraction and reasoning under matched distributions with controlled nuisance variation.

Source: https://www.emergentmind.com/topics/arc-bench-55520b29-3c00-411c-8929-ecb0afea2c65