ARC-Style Benchmarks Overview

• ARC-Style Benchmarks are evaluation tools designed to measure abstract reasoning, fluid intelligence, and systematic generalization without relying on domain-specific priors.
• They employ a minimal input–output paradigm with diverse tasks like geometric transformation, pattern completion, and multi-rule compositional reasoning.
• Recent studies highlight that evaluation protocols and perceptual bottlenecks significantly affect reported performance gaps between AI systems and human baselines.

ARC-style benchmarks are a class of evaluation tools designed to quantitatively assess abstract reasoning, fluid intelligence, and systematic generalization capabilities in both humans and artificial intelligence systems. Inspired by grade-school science examinations, puzzle design, and fundamental cognitive skills, these benchmarks construct novel tasks in input–output formats explicitly avoiding domain or linguistic priors, enabling rigorous testing of on-the-fly reasoning. The most established instantiations include the Abstraction and Reasoning Corpus (ARC), ARC Challenge, ARC-AGI, ARC-AGI-2, CellARC, and several synthetic or variant datasets. Recent work analyzes both their core design principles and exposes latent confounds in evaluation, especially the joint roles of perceptual and reasoning bottlenecks, and the impact of protocol choices on reported model difficulty.

1. Foundational Principles and Task Structures

ARC-style benchmarks are externally defined by three main characteristics: a minimal input–output paradigm, exclusion of domain knowledge, and reliance on novel, handcrafted task construction (Chollet et al., 17 May 2025, Borchmann, 2024). Each task $T$ comprises a small set of demonstration pairs $\{(x_j, y_j)\}$—typically grids up to $30 \times 30$ cells for ARC—with discrete symbols (colors), and one or more test inputs whose outputs must be inferred. Solvers must induce the transformation rule underlying the demos, then correctly generalize to applications on unseen test inputs.

Task diversity is achieved by mixing abstract reasoning categories:

• Pattern completion
• Geometric transformation (translation, rotation, scaling)
• Object counting or mapping
• Multi-rule compositional reasoning
• Sequential reasoning
• Contextual or conditional rule application
• In-context symbol definition

These tasks explicitly avoid overlap with training sets and external data, guaranteeing that the reasoning challenge is novel for any system. No external priors, domain annotation, or linguistic labeling are assumed.

2. Benchmark Variants and Dataset Construction

Classic ARC splits into two primary subsets for evaluation: ARC Easy and ARC Challenge (Borchmann, 2024). Easy tasks admit surface-level solution strategies—retrieval, pattern matching—while Challenge tasks are constructed to defeat IR-based and naïve neural baselines, instead demanding multi-step reasoning, explicit comparison, or exclusion. The delineation is empirical, based on initial baseline performance and manual task curation.

ARC-AGI and its successor ARC-AGI-2 expand the paradigm by rigorously calibrating difficulty levels via large-scale human testing ($N \sim 400$ participants, $>13,000$ task attempts) and filtering tasks so that all splits—Public, Semi-Private, Private—yield tightly matched human solve rates (difference $\leq 1\%$) (Chollet et al., 17 May 2025). ARC-AGI-2 introduces additional reasoning categories, including multi-rule, sequential, and symbol-in-context mappings.

CellARC extends this design to a fully synthetic domain: 1D multicolor cellular automata (CA), where each episode consists of five support input–output pairs and one masked query. Task difficulty is explicitly controlled via alphabet size $k$, neighborhood radius $r$, rule family, Langton’s $\lambda$, query support coverage, and cell entropy; enabling both interpolation (in-distribution) and extrapolation (out-of-distribution) test splits. Unlike ARC, CellARC supports arbitrarily large episodic datasets ($>95k$ tasks), synthetic regeneration, and fine-grained difficulty logging by metadata (Lžičař, 11 Nov 2025).

3. Evaluation Protocols and Quantitative Impact

Evaluation protocols for ARC-style benchmarks critically shape reported difficulty. Two main protocols are now recognized (Borchmann, 2024):

• Separation (Isolated Scoring): Each answer option is scored independently, and scores are normalized by token count or reference context. This setup is ill-suited for comparative or exclusionary reasoning, and artificially deflates reported accuracy for complex tasks.
• Options (Joint Scoring): Question plus all candidates are presented simultaneously, as in natural multiple-choice exams. Model selects the most likely answer label or candidate via log-probability, typically omitting normalization.

Empirical analysis shows that the protocol choice can induce a performance gap $$\Delta_S = \text{Acc}_{\text{options}}(S) - \text{Acc}_{\text{separation}}(S)$$ as large as 29 points on ARC Challenge for Llama 3.1 70B (64% $\to$ 93%). Similar dramatic gains are observed on SIQA (67% $\to$ 91%), OpenBookQA (48% $\to$ 89%), and ARC Easy (85% $\to$ 95%) (Borchmann, 2024). These improvements render protocol artifacts the dominant factor in perceived benchmark difficulty, rather than underlying task complexity.

Evaluation guidelines distilled from recent work recommend: (1) always prefer options protocol for any test requiring relative judgment; (2) reserve separation only for independent plausibility tasks (cloze); (3) constrain scoring to provided candidates; (4) randomize or rotate choice order; (5) explicitly report protocols to avoid unintentional artifacts in comparative studies.

4. Human vs. AI Baselines and Performance Characteristics

ARC-style evaluation robustly quantifies the gap between human and machine reasoning performance. On ARC-AGI-2, aggregate human accuracy averages $\approx 75\%$ while top AI systems (e.g., o3, ARChitects, Claude 3.7) remain below $5\%$ on Semi-Private splits (Chollet et al., 17 May 2025):

Model	ARC-AGI-1	ARC-AGI-2
o3-mini (High)	34.5%	3.0%
ARChitects	56.0%	2.5%
Claude 3.7 (8K)	21.2%	0.9%
Icecuber	17.0%	1.6%

In CellARC, a 10M-parameter Transformer achieves 58%/32% per-token accuracy (interpolation/extrapolation), outperforming recursive neural and symbolic baselines, while GPT-5 High reaches 62%/48% on curated 100-task subsets (Lžičař, 11 Nov 2025). Ensemble strategies (oracle selection per episode) further raise accuracy to 65%/36%, indicating neuro-symbolic complementarity.

VARC, a vision-centric pipeline treating ARC as image-to-image translation on learnable canvases, achieves 60.4% pass@2 accuracy on ARC-1 (ViT-18M + U-Net-55M), matching the average human rate (60.2%), with far fewer parameters than large LLMs (Hu et al., 18 Nov 2025).

5. Perceptual Bottlenecks and Reasoning Separation

Recent experiments reveal that the human–AI gap in ARC-style benchmarks mostly reflects visual perception, not just reasoning limitations (Wang et al., 24 Dec 2025). By employing a two-stage pipeline—per-image natural-language description followed by rule induction and application—absolute accuracy gains of 11–13 points are obtained across Mini-ARC, Bongard-LOGO, and ACRE. Manual error inspection attributes ≈ 80% of failures to misperception.

Dataset	One-Stage Accuracy	Two-Stage Accuracy	Δ (pp)
Mini-ARC	8.05%	20.13%	+12.08
Bongard-LOGO	62.00%	73.00%	+11.00
ACRE	22.00%	34.50%	+12.50

Error attribution (Mini-ARC) under one-stage: Perception (Demo/Test) 88.7%, Reasoning (Inductive/Deductive) 11.4%; under two-stage: Perception 64.9%, Reasoning 35.1%. This indicates that observed deficits largely reflect perception bottlenecks and task confounds.

Recommended evaluation principles now include explicit separation of perception and reasoning, prevention of cross-item leakage during perception, fine-grained error categorization, auxiliary annotation for benchmarking perception modules, and reporting of both end-to-end and stage-separated accuracies.

6. Design Enhancements and Future Directions

ARC-AGI-2 introduces several enhancements over ARC-AGI-1: increased resistance to brute-force program search (higher grid complexity, greater compositional depth), large-scale human calibration for difficulty labeling, wider accuracy spectrum, and balanced task splits (Chollet et al., 17 May 2025). New challenge types such as multi-rule, contextual, and symbol-in-grid demands promote advanced compositional generalization.

CellARC contributes a tunable, fully synthetic framework enabling reproducible generalization experiments and difficulty manipulation via metadata. Vision-centric approaches such as VARC demonstrate that visual priors—2D locality, translation and scale invariance—increase accuracy by over 27 points relative to naïve baselines (Hu et al., 18 Nov 2025). ARC-style benchmarks will benefit from more physically grounded tasks (e.g., 3D, occlusion, continuous color), increased focus on multi-modal reasoning (vision + language), and more rigorous benchmarking for inductive generalization beyond domain-anchored cognition.

7. Guidelines and Controversies in Evaluation

ARC-style benchmarks, while highly influential, have surfaced critical controversies. Protocol-dependent scoring may exaggerate reasoning gaps, task design may confound perception with abstract reasoning, and AI progress may be misreported if evaluative artifacts are not controlled (Borchmann, 2024, Wang et al., 24 Dec 2025). Guidelines developed across recent literature include:

1. Prefer options protocol for comparative or reasoning tasks
2. Isolate perception from reasoning to avoid bottleneck attribution errors
3. Stratify difficulty and task variety to prevent leaderboard overfitting
4. Provide auxiliary annotation and metadata for reproducibility and analysis
5. Explicitly report protocol, error sources, and baselines.

A plausible implication is that future ARC-style benchmarks should iteratively refine both task construction and scoring to cleanly measure core inductive reasoning capabilities, enabling more interpretable and actionable comparisons across cognitive architectures.