ARC Is a Vision Problem! (2511.14761v1)

Abstract: The Abstraction and Reasoning Corpus (ARC) is designed to promote research on abstract reasoning, a fundamental aspect of human intelligence. Common approaches to ARC treat it as a language-oriented problem, addressed by LLMs or recurrent reasoning models. However, although the puzzle-like tasks in ARC are inherently visual, existing research has rarely approached the problem from a vision-centric perspective. In this work, we formulate ARC within a vision paradigm, framing it as an image-to-image translation problem. To incorporate visual priors, we represent the inputs on a "canvas" that can be processed like natural images. It is then natural for us to apply standard vision architectures, such as a vanilla Vision Transformer (ViT), to perform image-to-image mapping. Our model is trained from scratch solely on ARC data and generalizes to unseen tasks through test-time training. Our framework, termed Vision ARC (VARC), achieves 60.4% accuracy on the ARC-1 benchmark, substantially outperforming existing methods that are also trained from scratch. Our results are competitive with those of leading LLMs and close the gap to average human performance.

• The paper demonstrates that reframing ARC as an image-to-image translation task using Vision Transformers yields state-of-the-art results.
• The paper leverages data augmentations, a canvas abstraction, and 2D positional encoding to enforce spatial invariance and enhance visual reasoning.
• Employing a two-stage learning process with offline training and test-time adaptation significantly improves meta-generalization on diverse ARC tasks.

ARC as an Image-to-Image Translation Problem: Vision-Centric Reasoning with VARC

Introduction and Motivation

The Abstraction and Reasoning Corpus (ARC) is a canonical challenge for few-shot rule induction and abstract reasoning, originally designed to measure machine intelligence via human-interpretable visual puzzles. Historically, ARC has been approached from a language-centric perspective, leveraging LLMs to process the problem as a form of inductive program synthesis over tokenized grids and symbolic representations. The paper "ARC Is a Vision Problem!" (2511.14761) disputes the naturalness of the language-centric paradigm for ARC and proposes that ARC should instead be treated as a fundamentally visual, not linguistic, problem. This reframing enables the use of established vision architectures—specifically, Vision Transformers (ViTs)—casting ARC as an image-to-image translation task where the model learns to map input grids to output grids directly in pixel space.

ARC Reformulation: Problem Structure

The ARC benchmark consists of hundreds of disparate few-shot reasoning tasks. Each task presents a small number of (input, output) grid examples, with the requirement to infer the transformation rule and apply it to unseen inputs from the same task. The inputs and outputs are 2D grids, typically up to $30 \times 30$ in size, with categorical color indices. Each task contains only 2–4 demonstration pairs.

This formulation is depicted as a multi-task, meta-generalization scenario: the model must learn a meta-prior over a collection of tasks (training set) and rapidly adapt or infer correct behaviors in unseen tasks with minimal data (test set).

Figure 1: The ARC benchmark contains diverse reasoning tasks, each with a handful of examples. VARC formalizes ARC as image-to-image translation, where task rules reflect visual concepts such as symmetry and gravity.

VARC: Vision ARC Framework and Visual Priors

VARC proposes a significant paradigm shift by recasting ARC as a vision problem, leveraging architectural priors from CV. The method utilizes a "canvas" abstraction—a fixed-size, high-resolution grid (e.g., $64 \times 64$)—onto which each input is placed after random scale and translation augmentations. This setup enables explicit modeling of translation and scale invariance, mirroring common practices in vision tasks.

Figure 3: Inputs undergo random scaling and translation and are placed on a large canvas, supporting data diversity and visual invariance.

Canonical vision backbones, notably ViTs and U-Nets, are deployed for image-to-image mapping, treating each task as per-pixel classification (akin to semantic segmentation). Task identity is incorporated via learnable task tokens or embeddings, which condition the model on the current reasoning rule.

Critical design aspects include:

• Patchification: The canvas is split into patches (e.g., $2\times2$), yielding combinatorial expressive capacity and encouraging spatial reasoning beyond what is attainable with tokenized grid values alone.
• 2D Positional Encoding: Positions are encoded in two dimensions, preserving essential spatial structure absent in language-model sequences.
• Augmentations: Translation, scaling, rotation, flipping, and color permutation augmentations enforce invariances and provide a richer training distribution.
• Task-Conditional Tokenization: Each task is associated with a unique embedding, facilitating meta-learning and supporting efficient adaptation during test-time training.

  Figure 5: The VARC architecture is a standard ViT, processing inputs on a canvas with a dedicated task token.

Two-Stage Learning: Meta-Generalization via Test-Time Training

Learning occurs in two distinct phases:

1. Offline Training: VARC is trained from scratch on a meta-dataset of ARC training tasks, optimizing a per-pixel cross-entropy loss. The offline model learns reusable visual feature extractors and parametric inductive biases.
2. Test-Time Training (TTT): Upon encountering a new test task, the model is fine-tuned exclusively on the provided demonstrations using data augmentations. TTT is realized by adjusting task token embeddings and optionally all parameters, quickly adapting to specific rules underlying each test task.

   Figure 2: Predictions improve progressively through test-time training as the model adapts to the new task with few demonstrations.

Empirical Results and Ablations

Comprehensive ablation studies highlight the significance of vision-centric modeling choices:

• Vision priors (canvas, patchification, 2D positional encoding, scale and translation augmentations) produce up to a 27.7 percentage point gain on the ARC-1 benchmark.
• ViT models outperform comparable U-Net variants, but both architectures benefit from the canvas-based representation.
• Test-time training is indispensable for generalization to unseen tasks, especially when augmented with auxiliary task formation through label permutations and geometric transformations.

  Figure 4: Ablation showing cumulative accuracy gains from incorporating visual priors (canvas, 2D positional encoding, patchification, augmentations).

Scalability experiments demonstrate that increased model capacity (width/depth) improves test-set performance up to an overfitting threshold, constrained by the limited diversity of ARC training tasks.

Figure 6: Accuracy scales with model size and depth, but excessive capacity without sufficient task diversity can lead to overfitting.

The VARC ensemble (ViT + U-Net) achieves 60.4% pass@2 accuracy on ARC-1, surpassing the best recurrent-from-scratch models and matching average human performance, while being competitive with several LLM-based methods—remarkably, using only 18–73M parameters and no external pretraining data.

Attentional Analysis and Visual Reasoning Dynamics

Attention analyses reveal that VARC develops structured, hierarchical visual reasoning capacities:

• Pixel-to-pixel attention maps show that deep layers focus on nonlocal reasoning (e.g., symmetry, copying patterns, extrapolating "rays"), while early layers capture local correspondences.
• t-SNE visualization of learned task embeddings indicates that the network discovers meaningful task clusters, aligning similar strategies or visual rules in embedding space.

  Figure 7: Layer-wise attention maps show mid- and high-level layers focusing on distinct semantic regions relevant to each ARC task.

  

  Figure 8: t-SNE plots of task embeddings reveal unsupervised clustering by semantic similarity, such as coloring and logical operations.

Case studies of success and failure illustrate that VARC can capture complex rule induction (e.g., object renaming, symmetry, spatial composition), but challenges remain in ambiguous or subtle tasks.

Figure 9: Example tasks from ARC-1 that VARC solves or fails, demonstrating competency and current system limitations.

Implications and Future Directions

The shift to a vision-centric ARC protocol has significant ramifications:

Practical Implications

• Feasibility of Vision-Only Generalization: VARC demonstrates that feedforward architectures trained solely on visual data can induce complex abstract rules, matching human performance in a nonlinguistic few-shot regime.
• Model Efficiency: Vision-centered approaches achieve competitive (and sometimes superior) results relative to LLMs, with orders-of-magnitude fewer parameters and no reliance on internet-scale pretraining.
• Transferability: The framework is extensible to other few-shot, many-task domains that retain a spatial or structural component, supporting broader meta-learning applications.

Theoretical Implications

• Role of Architectural Priors: The performance gains from incorporating spatial and scale invariance, as well as 2D positional encoding, suggest that architectural bias can be as important as (if not more so than) pretraining data scale for abstract reasoning in highly structured domains.
• Visual Abstraction: The emergence of task embedding clusters and sophisticated attention behaviors indicates that vision models can autonomously discover task similarities and hierarchies relevant to higher-order reasoning.

Future Research

Prospects include scaling up image-centric pretraining, hybrid integration of symbolic/linguistic reasoning with visual models, exploration of richer visual priors (e.g., object-centric representation), and cross-modal meta-learning protocols blending vision with language supervision.

Conclusion

VARC reframes ARC as a core vision challenge, dispelling the assumption that few-shot abstract reasoning necessarily requires language-based architectures or program induction. By leveraging vision transformer models, heavy data augmentation, 2D positional encoding, and a dedicated test-time adaptation regime, VARC achieves state-of-the-art performance among models trained from scratch, closing the gap with human averages and LLM-based systems. This opens a new direction for abstract reasoning research, foregrounding the potential of vision priors and visual meta-learning in tasks previously dominated by language approaches. The paper establishes a rigorous baseline for future vision-based reasoning systems on ARC and comparable few-shot abstraction benchmarks.