Abstract Visual Representation

• Abstract visual representation is a structured encoding of visual content that preserves key spatial, geometric, and relational invariants for abstract reasoning.
• It encompasses paradigms such as symbolic encoding, object-centric relational models, and discrete conditional generation to facilitate rule induction.
• Establishing the proper representational bottleneck is critical, as experiments show that structured representations significantly enhance performance on abstract visual tasks.

Abstract visual representation denotes a structured encoding of visual content that preserves the relations, invariances, and compositional regularities needed for abstract inference while suppressing irrelevant surface detail. In recent machine-learning work, the concept is most tightly coupled to abstract visual reasoning: the solver must recover rules over shapes, quantities, positions, movement, geometry, or analogy structure, rather than identify named objects or exploit external world knowledge. Recent diagnostic studies argue that performance on such tasks depends critically on the representational interface between perception and reasoning; when that interface is symbolic, object-centric, or otherwise structure-preserving, downstream reasoning improves sharply, whereas raw-pixel pipelines often fail near chance (Vaishnav et al., 23 Apr 2026).

1. Conceptual foundations

Within the recent literature, abstract visual representation is not a single standardized artifact but a family of representations organized around abstraction. In the most explicit formulation, it is a symbolic or structured encoding of a visual scene that preserves the geometry, topology, and compositional structure relevant to reasoning, rather than the scene’s raw pixels. More generally, visual abstraction has been formalized as a process that transforms a source thing into a less concrete sign thing under a task-dependent point-of-view; visual abstraction is the case in which the sign thing is visual and the transformation intentionally disregards certain aspects of the data representation (Viola et al., 2019). This formulation makes representation purposive rather than merely compressive: what is omitted and what is preserved are both task-conditioned.

This perspective aligns naturally with abstract visual reasoning benchmarks. The relevant representation must expose latent regularities such as progression, arithmetic, repetition constraints, spatial relations, or compositional analogy. A plausible implication is that abstract visual representation is best understood not as a particular modality or architecture, but as a functional property: it is adequate when it retains the structure required for rule induction and candidate verification.

A broader, category-level version of the same idea appears in work on neural “shared visual abstractions.” There, a perception-driven drawing system generates highly simplified ink-and-paper forms that strongly activate target classes across multiple CNN architectures and are also recognizable to human observers. The generated drawings can elicit stronger responses than real validation images, and a survey of 12 students found that all participants matched all drawings to the correct ImageNet category. That result supports the claim that abstraction can preserve a category’s “essence” across both human and machine perception (White, 2019).

2. Formalization inside abstract visual reasoning

Most operational uses of abstract visual representation arise inside abstract visual reasoning tasks. These tasks are typically posed as panel-based problems in which a model must infer a hidden rule from context panels and apply it to a missing or queried panel. In MARVEL, each puzzle contains context panels $p_1, p_2, \dots, p_n, p_b$, where $p_b$ is blank, and four candidates $c_i$. The objective is to choose $c_c$ such that the pattern function is preserved: $P(p_1,p_2,\dots,p_n) = P(p_1,p_2,\dots,p_n,c_c).$ This same family includes Raven’s Progressive Matrices, visual analogy problems, Odd-One-Out tasks, and SVRT-style discrimination problems, all of which require reasoning over hidden relations rather than object labels (Jiang et al., 2024).

Surveys of the field describe AVR tasks along multiple axes: input shapes, hidden rules, target task, cognitive function, and specific challenge. The same benchmark family therefore contains geometric-shape tasks, abstract-shape tasks, completion problems, discrimination problems, generation problems, and tasks emphasizing domain transfer, extrapolation, or arithmetic. This taxonomy is important because it shows that abstract visual representation must accommodate more than one formal regime; a representation that works for fixed-vocabulary RPM panels need not work for Bongard-style concept learning or real-world visual analogies (Małkiński et al., 2022).

Recent unified formulations make this point explicit. UCGS recasts several AVR tasks as conditional predictability estimation. For RPM and VAP, the correct answer is the candidate with highest conditional probability under the context, $x^\star=\arg\max_{x\in I^s} p(x\mid I^p_{\neg N})$. In this view, a representation is useful insofar as it supports reliable estimation of the predictability of one image from the rest of the panel (Shi et al., 15 Jul 2025). A different unification strategy renders an entire AVR instance as a single image, with no a priori assumptions about the number of panels, their locations, or their roles. That unified view treats the problem as ordinary image recognition over a structured composite image, shifting representational responsibility from task-specific panel parsing to the model itself (Małkiński et al., 2024).

3. Representation as the primary bottleneck

A central empirical result of the recent literature is that failures on abstract visual reasoning benchmarks often originate in representation and grounding rather than in downstream logical machinery alone. MARVEL makes this diagnosis with unusual clarity. The benchmark contains 770 puzzles spanning six core knowledge patterns, two shape types, and five task configurations, and it augments the final AVR question with coarse-grained and fine-grained perception questions. Across nine representative MLLMs, all models show near-random performance on the main AVR question. Human performance is $68.86\% \pm 9.74$, whereas models cluster in the mid-20% range, with the best model around $28.83\%$. Coarse-grained perception is often below 50%, fine-grained perception is near-random, and panel counting is frequently below 45%. When accurate text descriptions are supplied, performance jumps substantially—for some closed models, from around $11.57\%$ to $44.21\%$—and GPT-4V reaches human-like performance in that controlled setting. The paper’s conclusion is that the bottleneck is basic visual grounding: models often cannot reliably count panels, identify spatial relations, or extract relevant shape attributes (Jiang et al., 2024).

VisuRiddles reinforces the same diagnosis at larger scale. The benchmark contains 1,000 questions and 1,100 images, spanning five perceptual dimensions—Numerosity, Attribute, Style, Position, and Spatiality—plus RAVEN reasoning and Sudoku reasoning. The accompanying Perceptual Riddle Synthesizer generates fine-grained perceptual descriptions and intermediate supervision. The reported gains are large: the baseline Qwen2.5-VL-7B rises from an average of 20.9 to 43.7 after PRS training, with Sudoku improving from 0.0 to 60.0 and RAVEN from 14.0 to 95.0. Replacing raw visuals $p_b$0 with perceptual descriptions $p_b$1 also sharply improves performance, for example raising Qwen2.5-VL-72B from 30.9 average on raw visuals to 73.6 on perceptual descriptions (Yan et al., 3 Jun 2025).

Symbolic-grounding experiments on Bongard-LOGO sharpen the argument further by holding the reasoning backend largely fixed while changing only the representation. A strong visual baseline, Gemini-2.5-Flash, remains near chance on raw images, with 50.2 on Free-form, 49.8 on BD, and 50.1 on HD. Under the Componential–Grammatical paradigm, symbolic inputs derived from ground-truth programs raise performance to 78.1/68.8/61.0 for action programs and 79.3/72.0/59.1 for action descriptions, and Phi-4-Reasoning reaches 96.2% on Free-form. Ablations show that input format, explicit concept prompts, and minimal visual grounding matter much less than the shift from pixels to symbolic structure (Vaishnav et al., 23 Apr 2026).

The diagnosis is not completely uniform across benchmarks. MultiStAR reports that existing MLLMs perform adequately on basic perception tasks but struggle in more complex rule-detection stages, and its manual error analysis finds reasoning error to be the dominant failure mode, followed by perception error. Injecting prior information reduces perception errors but does not solve reasoning failures (Jiang et al., 28 May 2025). MaRs-VQA arrives at a closely related conclusion from a visual-cognition perspective: the main weaknesses are visual pattern extraction, visual working memory, and integration across multiple images, not language-only inference (Cao et al., 2024). Taken together, these results suggest that “representation bottleneck” is a family diagnosis rather than a single failure type: in some settings the decisive problem is fine-grained perception, in others it is maintaining and composing already-perceived structure.

4. Representational paradigms

One major paradigm is explicit symbolic or program-like encoding. In the Componential–Grammatical framework, Bongard-LOGO images are represented through a hierarchy such as BongardImage $p_b$2 OneStrokeShape $p_b$3 BasicAction, with serialized forms like line_TYPE_LENGTH-TURNANGLE and arc_TYPE_ARCANGLE_ARCRADIUS-TURNANGLE. This representation is componential because it is built from interpretable parts, and grammatical because those parts compose according to explicit structural rules. It is not a caption-like summary; it preserves ordered procedural structure, quantized geometric parameters, and latent generating programs (Vaishnav et al., 23 Apr 2026).

A second paradigm is object-centric relational representation. Slot Abstractors combine Slot Attention with Abstractors in a two-stage pipeline: unsupervised object discovery yields slots, and relational abstraction operates over those slots rather than over raw pixels. The model factorizes feature embeddings $p_b$4 from position embeddings $p_b$5, then updates relational states by relational cross-attention. This produces a relational bottleneck in which downstream computation accesses relations rather than raw appearance. The design scales as $p_b$6, compared with OCRA’s effective $p_b$7, and supports scenes with more than 100 objects, including PGM problems with up to 144 objects across 9 panels (Mondal et al., 2024).

A third paradigm is discrete conditional generation. UCGS-T first learns a VQ-VAE-style tokenizer, then represents each image as discrete patch codes, aggregates higher-level concepts through class-slot tokens, and predicts the missing image autoregressively from the context. This supports both selective and generative AVR tasks under one probabilistic interface. The approach treats the answer image not merely as a label to classify but as a structured object whose predictability reflects whether the latent rule has been captured (Shi et al., 15 Jul 2025).

A fourth paradigm is task-general structured reasoning with dynamic or grouped computation. SCAR introduces a Structure-Aware dynamic Layer whose effective weights adapt to task structure, allowing the model to solve RPMs, VAPs, and Odd One Out without fixed assumptions about panel count or arrangement (Małkiński et al., 2023). UMAVR instead uses a unified rendering of the full problem as a single image and a hybrid local-perception/global-reasoning architecture to process that rendered structure (Małkiński et al., 2024). PoNG emphasizes grouped and normalized pathway computation: its reasoner contains pointwise, ordinary, group, and group-pair 1D convolutions, while TCN is described as preserving relations within task-relevant groups and discarding absolute magnitude information, which the authors argue improves out-of-distribution extrapolation (Małkiński et al., 19 May 2025).

A fifth line of work attempts to improve representations by post-training rather than by redesigning the core architecture. LLaVA-AVR-7B uses synthetic regular puzzles, semi-automatic non-regular puzzle annotation, visual elicitation, template-based chain-of-thought, process-level supervision, and conditional multi-task learning. The resulting model reaches 82.7% overall on RAVEN, compared with 10.2 for the base LLaVA-NeXT-7B, and obtains 75.5 perception accuracy and 35.7 reasoning accuracy on MARVEL while keeping general multimodal benchmarks close to baseline. This indicates that representational quality can be substantially altered through staged data design and post-training, even without introducing a new backbone (Zhu et al., 2 Apr 2025).

5. Benchmarks, evaluation regimes, and generalization

The benchmark literature increasingly treats abstract visual representation as something that must be diagnosed, not merely assumed. Earlier surveys already emphasized that AVR research was fragmented across task-specific datasets and argued for taxonomies and mixed-task evaluation, noting that human IQ tests combine multiple problem types while machine-learning work usually studies one task in isolation (Małkiński et al., 2022).

Several recent benchmarks operationalize different failure modes and representational demands.

Benchmark or framework	Representational emphasis	Key structure
MARVEL	Perception-grounded multidimensional AVR	770 puzzles; 6 patterns; 2 shape types; 5 configurations
VisuRiddles + PRS	Fine-grained perceptual supervision	1,000 questions; 5 perceptual dimensions; RAVEN and Sudoku subsets
MultiStAR + MSEval	Multi-stage rule induction	Direct Answer levels and Logical Chain
MaRs-VQA	Visual cognition and working memory	1,440 instances from 18 questionnaires
A-I-RAVEN / I-RAVEN-Mesh	Held-out-attribute generalization and transfer	70,000 instances per regime

MARVEL broadens the usual RPM-style design along three dimensions: six core knowledge patterns derived from core knowledge theory, two classes of input shapes, and five task configurations including sequence, two-row, matrix, group, and reassembling (Jiang et al., 2024). VisuRiddles broadens in a different direction by combining low-level perceptual categories with higher-level RAVEN and Sudoku reasoning, and by pairing benchmark construction with a data-generation engine that exposes intermediate perceptual states (Yan et al., 3 Jun 2025). MultiStAR makes the reasoning chain explicit through six Direct Answer complexity levels and a Logical Chain of dependent subproblems, while MSEval scores intermediate correctness instead of only the final endpoint (Jiang et al., 28 May 2025). MaRs-VQA draws from the Matrix Reasoning Item Bank and adds step-wise reasoning annotations for Qwen2-VCog, producing a benchmark tied more closely to the literature on human visual cognition (Cao et al., 2024).

Generalization-focused benchmarks examine whether representations transfer beyond the training distribution. A-I-RAVEN defines four primary held-out-attribute regimes—A/Color, A/Size, A/Type, and A/Position, with Position and Number coupled—so that the model must apply non-Constant rules to an attribute that was Constant during training and validation. I-RAVEN-Mesh adds a mesh component with Number and Position rules and is used for progressive knowledge-acquisition experiments. These datasets reveal that many strong RPM solvers remain weak at rule transfer across attributes and at low-data transfer to new structural components (Małkiński et al., 2024). PoNG’s strong results on these regimes show that some structured inductive biases improve generalization, but the same experiments also confirm that held-out attribute-pair regimes remain difficult even for the best models (Małkiński et al., 19 May 2025).

This benchmark progression suggests a broader shift in the field: final-answer accuracy is no longer treated as a sufficient proxy for abstract visual competence. Perception questions, step-wise chains, perceptual annotations, symbolic upper bounds, held-out attributes, and transfer settings all attempt to measure whether the learned representation actually encodes the structure that the task demands.

6. Broader notions and acronym ambiguity

Abstract visual representation also intersects with broader theories of visual abstraction. In visualization theory, abstraction is defined by information suppression under a purpose-driven point-of-view, and meaningful visual abstraction is the case in which this suppression preserves what matters for cognition while reducing cost (Viola et al., 2019). In computer vision, “amplification through simplification” appears in the finding that sparse abstract drawings can preserve category identity across architectures and even across human and machine observers (White, 2019). These perspectives do not describe benchmark solvers directly, but they provide a conceptual backdrop: abstraction is useful not because it is sparse, but because it selectively preserves structure.

The acronym “AVR” is also polysemous in recent arXiv literature. In one paper it denotes an audio-visual humor detection system that fuses VideoMAE and AST embeddings without ASR, reaching 56.70% accuracy with a CNN downstream model (Sharma et al., 2024). In another it denotes Attention based Salient Visual Relationship Detection, which combines local-context attention with a heterogeneous-graph prior for visual relationship detection and reports gains of up to 87.5% in terms of recall (Lv et al., 2020). In a third it denotes Audio-Visual Recordings used by AVR-Eval and AVR-Agent to compare and improve generated games and animations (Jolicoeur-Martineau, 1 Aug 2025). These usages are unrelated to abstract visual reasoning or visual abstraction, and they make explicit acronym disambiguation necessary in bibliographic and technical discussion.

Across the abstract-reasoning literature proper, the dominant conclusion is consistent. Raw pixels are often an inadequate substrate for high-level rule induction; better performance arises when the model is given, learns, or is trained toward a representation that is symbolic, object-centric, relational, discrete-compositional, or otherwise structured around the latent invariants of the task. This suggests that progress in abstract visual reasoning depends at least as much on representational design and supervision as on generic increases in model scale.