Perceptual Abstraction & Reasoning Language (PeARL)

• PeARL is a domain-specific language designed to improve general visual reasoning, particularly for the ARC benchmark tasks.
• It integrates a functional DSL for program synthesis and interfaces novelly with LLMs for object-level decomposition.
• PeARL's hybrid framework enhances task-solving efficacy through symbolic and neural method synergy.

The Perceptual Abstraction & Reasoning Language (PeARL) is a domain-specific language and representational framework introduced to address highly generalizable visual reasoning, specifically targeting the Abstraction and Reasoning Corpus (ARC) benchmark. PeARL serves as both a functional DSL for use by neurosymbolic program synthesis engines and as a structured interface between vision modules, LLMs, and program synthesis systems in ARC task solvers. It facilitates the decomposition of pixel-level reasoning into object-level and program-level representations, which are tractable by existing neural and symbolic models (Camposampiero et al., 2023, Bober-Irizar et al., 2024).

1. Formal Language Design and Representations

PeARL is realized in two major forms: (1) a purely functional, statically-typed DSL for use in program synthesis, and (2) a structured natural language representation suitable for LLM prompting.

DSL Formulation: Every PeARL program is a pure function mapping an input grid to an output grid—formally, its type is $grid \rightarrow grid$. The DSL is automatically constructed from a set of Pythonic primitives with explicit type signatures. Each primitive $f$ with signature $$f : \tau_1 \rightarrow \cdots \rightarrow \tau_n \rightarrow \tau_{out}$$ is a production in the grammar. Programs are constructed by lambda-abstraction and nested application: $f$1 This sample program transposes a grid, fills its interior, and mirrors it horizontally.

Grammar Syntax:

• <Program> ::= λ$0. <Expr>
• <Expr> ::= <Prim> (<Expr>)* | $0
• Representative primitives include: rigid transforms (rot90, flipx), cropping, color filters, object decomposition (split8), and morphological drawing (fillobj, ic_makeborder).

Types:

• grid: 2D integer array (colors 0–9), with position and size metadata.
• colour: Integer $1–9$ ($0$ is background).
• size ≡ (w, h)$,pos ≡ (x, y),count`.

Natural-Language Representation:

No formal BNF exists for the structured language prompt, but its grammar follows a regular pattern:

• Each image is described as a paragraph with object listings, shapes, symmetries, positions, and colors, organized by unique shape identifiers and sorted by size.
• Tasks are represented as sequences of “Input k:”/“Output k:” pairs plus a “Test Input” and a blank “Output T: <to be generated>” slot for the LLM to fill.
• Semantically, each object segment corresponds to a single connected region of non-background color, labeled by geometry, symmetry, and placement (Camposampiero et al., 2023).

2. Vision Encoding and Object Extraction

The mapping of raw ARC task grids into PeARL representations is accomplished by a vision module employing the following heuristic pipeline:

• Background detection: Small CNN predicts the grid’s background color.
• Object segmentation: Connected-component analysis over non-background pixels identifies objects.
• Shape categorization: Each object is matched to a finite catalog (rectangles, lines, crosses, or classified as “complex”).
• Symmetry detection: Bilateral symmetry along the axes is tested.
• Object annotation: Each object in a grid receives a unique label (consistent shape ID across images), upper-left position, bounding size, monochromatic color, and symmetry flags.
• Generated paragraphs group objects and order them by size.

This approach allows the pipeline to robustly parametrize ARC scenarios in terms of candidate objects and transformations, supporting both symbolic (program synthesis) and neural (LLM) solvers (Camposampiero et al., 2023).

3. Program Synthesis and Recognition-Model Integration

Within neurosymbolic frameworks such as DreamCoder, PeARL is used to explicitly enumerate candidate programs that solve ARC tasks:

• Program enumeration: Programs are ordered by increasing minimum description length (MDL), calculated as the sum of negative log-probabilities for each grammar production, i.e., $\sum_{r \in p} -\log P(r)$.
• Recognition model: A neural perception module computes a 256-dimensional feature embedding $\phi(T)$ for each ARC task $T$, based on residual CNNs with dilated convolutions applied to input-output grid pairs.
• Contextual grammar: From $\phi(T)$, a small MLP (“GrammarNet”) predicts a PCFG-style weight tensor $Q$ over grammar productions and their argument positions.
• Self-supervised learning: Lacking ground-truth programs, the system “dreams” synthetic tasks by sampling existing programs. Loss is the sum of (i) an entropy term over used rules and (ii) binary cross-entropy for primitive inclusion; $f$0.
• Neurosymbolic loop: Alternating “wake” (MDL-guided enumeration), “abstraction sleep” (subtree compression), and “dreaming sleep” (recognition model training) phases yield 10–30x speedup over context-free search; no end-to-end loss for pixel-level accuracy is specified (Bober-Irizar et al., 2024).

4. LLM Augmentation and Prompt Engineering

PeARL supports direct LLM-based reasoning via prompt-engineered transformations:

• Grids are serialized as ASCII digit matrices using model-specific tokenization (e.g., single-token-per-digit for LLaMA).
• LLMs are prompted with a game-format template, typically showing multiple (input, output) ARC training pairs and the test input with the output slot blank; the LLM is expected to generate the missing grid.
• Augmentation: Each ARC instance is converted into three parallel prompts (raw, transposed, and rotated), allowing the LLM three attempts per test input; solve rate approximately doubles with this “3-shot” augmentation.

LLMs trialed include LLaMA (7B/33B), GPT-3, GPT-3.5-turbo, and GPT-4. The best scores on 400 ARC tasks with augmentation are: GPT-4 achieves 21% (ARC-Easy), 8% (ARC-Hard); size-matching accuracy approaches 567/800 for GPT-4 (Bober-Irizar et al., 2024).

Model	ARC-Easy (aug.)	ARC-Hard (aug.)	Size-Acc (/800)
LLaMA-7B	13	1	201
LLaMA-33B	24	9	429
GPT-3.5-turbo	40	13	538
GPT-4	85	32	567

5. Compositional Ensembles and Performance Evaluation

The system-level architecture combines several complementary solvers:

• Symbolic DSL search (“IceCuber”): Handcrafted DSL-based program search.
• DreamCoder + PeARL: MDL-guided synthesis with recognition-model acceleration.
• LLM solvers: Zero-shot generation with “3-shot” augmentation.

Each system proposes candidate grids. An ensemble method implements weighted voting (DreamCoder: 20 votes per solution; GPT-4: 3 votes; IceCuber: 8/4/4 for top guesses), returning the 3 highest-scoring candidate solutions.

System	ARC-Easy	ARC-Hard
IceCuber (DSL search)	209	160
DreamCoder + PeARL	70	18
GPT-4	85	32
IceCuber + DC + GPT-4 ensemble	228	161

The ensemble consistently outperforms the best individual method by 10–15%, demonstrating the complementarity between program synthesis, vision-language, and LLM-based approaches (Bober-Irizar et al., 2024).

6. Software Infrastructure and Accessibility

The arckit Python library standardizes code interfaces, data loaders, evaluation, and visualization for the ARC challenge:

• arckit.load_data() yields ARC-Easy and ARC-Hard sets of 400 tasks each; tasks can be indexed by integer or id string.
• All grid data is stored as Numpy arrays accessible via .train_inputs, .train_outputs, and .test_inputs.
• Evaluation functions compute task-level solution scores, rate, and visualization options.
• The command-line interface (arctask) provides a REPL for interactive evaluation and visualization.

All PeARL integration and ensemble pipelines are published open-source as part of arckit for reproducibility and continued ARC research (Bober-Irizar et al., 2024).

7. Empirical Insights and Limitations

PeARL-based frameworks solve 10–12% (zero-shot) of ARC tasks using LLMs, including several that defeated all top handcoded program search solvers. Performance increases monotonically with LLM model scale (log-linear with parameter count). Notable strengths include leveraging LLMs’ implicit compositional priors and handling multi-object scenarios without hand-tuned heuristics. However, failures arise in pixel-level effects (e.g., color filling, interpolations) and complex scenes that exceed prompt length, and current approaches function solely zero-shot—no end-to-end fine-tuning over the structured language pipeline has been performed. A plausible implication is that PeARL marks a decisive point in using language as a bridge between vision, program synthesis, and neural reasoning but remains constrained by the expressiveness of its object-level representations and the breadth of LLMs’ learned priors (Camposampiero et al., 2023, Bober-Irizar et al., 2024).