ARCLE: RL Benchmark for Visual Abstraction

• ARCLE is a Gymnasium-compatible MDP designed to challenge RL agents with visual abstraction tasks from the Abstraction and Reasoning Corpus.
• It employs a PPO-based actor-critic framework with auxiliary losses to overcome sparse rewards and manage large, structured action spaces.
• Empirical results show that sequential and color-equivariant policy parameterizations significantly improve learning and adaptability on both random and ARC-specific grid tasks.

ARCLE—the Abstraction and Reasoning Corpus Learning Environment for Reinforcement Learning—constitutes a Gymnasium-compatible Markov decision process (MDP) designed to expose reinforcement learning (RL) agents to the full spectrum of challenges presented by the Abstraction and Reasoning Corpus (ARC). ARCLE provides a principled environment for investigating RL-based approaches to inductive reasoning, contending with combinatorially large action spaces, extremely sparse reward signals, and a diverse array of task patterns. It serves as both a benchmark and a controlled setting for developing RL algorithms that can approach human-level systematic reasoning in visual abstraction domains (Lee et al., 2024).

1. Formal Environment Specification

ARCLE defines its environment as an MDP $(\mathcal{S}, \mathcal{A}, P, R)$, in which the state $s \in \mathcal{S}$ captures the agent’s entire context for solving an ARC task. Each episode presents the agent with a static input grid of size $H \times W$, a mutable working grid, a clipboard register ("clip") for copy/paste operations, and, for object-oriented actions, a decomposition into "object" and "background" layers augmented with metadata such as selection mask, positional attributes, dimensions, rotation parity, and an activity flag.

The action space is structured as $a = (\mathsf{op}, \mathsf{sel})$, where $\mathsf{op} \in \{0, \dots, 34\}$ encodes primitives: ten color-painting operations, ten flood-fill operations, move, rotate, flip, copy/paste, and environment manipulations (CopyInput, ResetGrid, ResizeGrid, Submit), while $\mathsf{sel}$ is either a binary mask or bounding-box coordinates marking targeted pixels. The two-part action encoding enables rich edit sequences while supporting autoregressive selection conditioned on operation.

Transition dynamics are deterministic: the built-in function $\mathrm{step}(s_t, a_t)$ maps uniquely to the next state $s_{t+1}$. Object operations use a two-layer implementation, separating transformed selections from a static background to prevent information loss during iterative moves.

The default reward is sparse: $R(s_t, a_t) = 1$ if the submitted grid matches the target ("answer") and 0 otherwise. To mitigate the sparse-reward problem, a dense auxiliary reward proportional to the fraction of incorrect pixels after each edit is introduced:

$$r(s_t, a_t) = -\frac{\#\{\text{pixels where } grid_{t+1} \neq answer\}}{H\,W}$$

2. Reinforcement Learning Architectures and Optimization

ARCLE implements an on-policy actor-critic method, Proximal Policy Optimization (PPO), as the central RL baseline, employing a shared Transformer encoder for state representations. Grid and answer tensors are tokenized (flat $s \in \mathcal{S}$0), and each token embedding comprises positional, color-value, and type (grid vs. answer) vectors. Non-editable (masked) pixels are zeroed out.

The architecture predicts both value ($s \in \mathcal{S}$1 via a "value token") and action distributions (via policy tokens). The PPO objective function draws directly from standard formulations, optimizing for clipped surrogate loss and mean-squared value error:

$s \in \mathcal{S}$2

$s \in \mathcal{S}$3

with $s \in \mathcal{S}$4, $s \in \mathcal{S}$5, and $s \in \mathcal{S}$6 the stop-gradient operator.

3. Auxiliary Losses and Policy Parameterizations

To improve data efficiency and representation learning, three auxiliary heads are trained in tandem with the PPO objective:

1. Previous reward prediction ($s \in \mathcal{S}$7): regresses the previous reward from $s \in \mathcal{S}$8.
2. Current reward prediction ($s \in \mathcal{S}$9): regresses the current reward given $H \times W$0.
3. Next-state prediction ($H \times W$1): outputs pixel-wise predictions of the next state.

These losses enhance gradient quality and bias the architecture towards capturing task-relevant structure. The first loss ties directly to the value token, while the others require additional state-action encodings.

A key methodological contribution is the comparison of different policy parameterizations:

• Fully factorized: $H \times W$2,
• Sequential: $H \times W$3 followed by $H \times W$4,
• Color-equivariant: parameter-sharing across color operations exploits the underlying color-permutation symmetry present in ARC.

Empirically, sequential and color-equivariant parameterizations outperform the fully factorized approach in both learning speed and asymptotic performance. This confirms the necessity of modeling strong dependencies between operation type and pixel mask—especially to manage the $H \times W$5 complexity of the selection space.

4. Empirical Evaluation and Analysis

ARCLE’s empirical validation uses two principal settings:

• Random-Grid: Random $H \times W$6 input/output grid pairs require the agent to learn arbitrary grid transformations.
• ARC-Grid: Genuine ARC tasks restricted to small sizes ($H \times W$7), offering a diverse but challenging benchmark.

Without auxiliary losses, PPO fails to escape zero reward regimes. Synergistically incorporating all three auxiliary heads yields over $H \times W$8 success on random grids (in three out of four training seeds after $H \times W$9 environment steps) and significant improvements on ARC-Grid (raising success rates from about $a = (\mathsf{op}, \mathsf{sel})$0 to $a = (\mathsf{op}, \mathsf{sel})$1–$a = (\mathsf{op}, \mathsf{sel})$2 across runs).

A continual learning curriculum, incrementally increasing the number of available colors $a = (\mathsf{op}, \mathsf{sel})$3, reveals that color-equivariant policies robustly adapt to changing task distribution, while sequential policies suffer irrecoverable performance drops beyond two color transitions. This underscores the importance of symmetries and appropriate architectural inductive biases for generalization (Lee et al., 2024).

5. Future Research Directions

Upon demonstrating that a PPO agent with auxiliary and non-factorial heads can master small-scale ARCLE tasks, the framework proposes three primary research extensions toward full ARC generalization:

• Meta-Reinforcement Learning (Meta-RL): Leveraging Model-Agnostic Meta-Learning (MAML), policies would be meta-trained across distributions of ARC tasks. The inner loop fine-tunes on a small number of demonstrations per task, and the outer loop aggregates gradient information for transfer to unseen tasks, aiming at few-shot generalization.
• Generative Flow Networks (GFlowNets): By representing grid-editing trajectories as paths in a directed acyclic graph, a flow-matching objective encourages policies to sample diverse solution paths with probability proportional to final reward. Restricting moves to single-pixel edits preserves the acyclic property and supports the discovery of multiple high-reward, human-like strategies.
• World Models: Inspired by DreamerV3, this direction calls for building latent-space generative models that capture object structures and topologies from the raw grid, then learn actor–critic policies directly in the latent domain. Vision modules infer grid abstractions; memory models extract spatial and relational priors; controllers generate candidate edit sequences.

These research avenues collectively aim at solving the full ARC benchmark—marked by large grids, no access to target answers, and maximally diverse visual reasoning tasks—by leveraging meta-learning, structured exploration, and model-based RL architectures.

6. Significance for Reinforcement Learning and Program Induction

ARCLE bridges the gap between the discrete, compositional reasoning required by ARC and the continuous optimization techniques prevalent in RL. It surfaces unique challenges for RL agents, such as managing large, structured action spaces, handling extremely sparse rewards, and transferring inductive reasoning across diverse visual patterns. Its state, action, and reward design reflect a direct mapping from the nature of ARC tasks to RL formalism, enabling systematic investigation of architectural, algorithmic, and auxiliary-signal innovations in the context of human-like abstraction and reasoning. Results suggest that leveraging appropriate auxiliary supervision and symmetry-aware policy heads can make intractable environments amenable to practical RL training regimes (Lee et al., 2024).

A plausible implication is that progress on ARCLE will inform broad directions in RL-based program induction, especially in settings requiring few-shot generalization, modularity, and compositional reasoning over discrete symbolic domains.