Percept-V: Multimodal Visual Perception Benchmark

• Percept-V is a multimodal perception benchmark designed to isolate and test basic visual perception tasks using generated images following the TVPS-4 framework.
• It systematically scales visual complexity across 30 domains and 20 task sizes, thereby evaluating model performance under increasing distractor load.
• Empirical findings reveal that models excel on simple tasks but sharply decline in accuracy as task complexity increases, highlighting limits in current MLLMs.

Percept-V is a multimodal perception benchmark introduced to test whether multimodal LLMs can solve basic visual perception problems on uncontaminated, program-generated images containing basic shapes and structures. Its central premise is that strong performance on broad multimodal reasoning, mathematics, or science tasks does not by itself establish competence in foundational visual perception. Percept-V therefore isolates elementary but systematically scalable perceptual tasks, grounds them in the TVPS-4 framework from cognitive science, and evaluates model accuracy as visual complexity increases across 30 domains and 20 task sizes (Ghosh et al., 28 Aug 2025).

1. Terminological scope and usage

Within the 2025 multimodal-evaluation literature, Percept-V denotes a dataset-centered benchmark for controlled testing of basic visual perception in MLLMs. The benchmark is defined by program-generated images, explicit task rules, and a complexity axis that increases without changing the underlying task principle (Ghosh et al., 28 Aug 2025).

The label is not fully uniform across adjacent research areas. In one vision-science context, a “PerceptNet/Percept-V-style autoencoder” refers to a bio-inspired image-reconstruction model in which perceptual properties emerge from self-supervised reconstruction rather than from human perceptual labels (Hernández-Cámara et al., 14 Aug 2025). In an assistive-systems context, Percept-V is also presented as an LVLM-based environment-perception system for visually impaired people that combines a wearable RGB-D sensor, a touchscreen smartphone, semantic segmentation, and a cloud backend (Chen et al., 25 Apr 2025). This suggests that the term is best interpreted through its immediate paper context: benchmarking basic visual perception, modeling emergent perceptual structure, or deploying perception-oriented assistive interaction.

2. Cognitive framework and dataset construction

Percept-V is motivated by two design goals: avoiding data contamination and testing basic perception rather than domain knowledge. The first goal addresses the possibility that models may have encountered benchmark material during training; the second addresses the tendency of existing multimodal benchmarks to conflate perception with specialized knowledge or sophisticated reasoning. Percept-V therefore uses generated images with basic shapes and structures, and emphasizes puzzle-like domains with controlled compositional difficulty (Ghosh et al., 28 Aug 2025).

The benchmark is grounded in the TVPS-4 (Test of Visual Perceptual Skills) framework, which decomposes visual perception into eight skills.

Skill	Definition
Visual Attention	focus on relevant visual information and ignore distractors
Visual Discrimination	detect differences/similarities in shape, color, size, etc.
Visual Memory	recall visual traits of an object or form
Visual Sequential Memory	recall an ordered sequence of objects
Visual Spatial Relationships	understand spatial relations among objects
Visual Figure Ground	find an item in a cluttered background
Visual Form Constancy	recognize a shape despite scale/orientation changes
Visual Closure	recognize incomplete or partially occluded forms

The dataset contains 7,200 total images, 30 categories/domains, 200 experiments/questions per domain, 10 instances per problem size, and 20 problem sizes from size 1 to size 20. Each domain corresponds to one question type, and each domain is tested at increasing task size. “Problem size” serves as a proxy for complexity and usually means the number of objects, the number of distractors, or the number of sub-tasks involved. The underlying task remains fixed while the image becomes more crowded or structurally complex (Ghosh et al., 28 Aug 2025).

3. Domains, skill composition, and complexity scaling

Percept-V comprises 30 domains. They include change detection, counting, matching, spatial localization, sorting, maze solving, graph interpretation, layered-shape analysis, and mirror or water-image reasoning. The full domain set is: Change Colour; Circle Boxes; Circle Location; Circle Right Triangle; Colours Present; Comparing Size; Count Coloured Circles; Counting Circles; Counting Locations; Counting Shapes; Cross and Knots; Graph Counting; Identifying Shapes; Inside Circles; Layered Colours; Layered Shapes; List Colours; List Shapes; Locate Circles Colour; Locate Circles Shape; Match Outline; Match Shadow; Maze Solving; Mirror Image; Numbered Shapes; Sort Circles; Sort Lines; Vanishing Objects; Water Image; and grid-based or multi-step spatial tasks such as maze solving, matching, sorting, and locating objects (Ghosh et al., 28 Aug 2025).

The skill-category mapping is explicitly combinatorial rather than one-domain-one-skill. For example, Graph Counting, Counting Circles, and Maze Solving are associated with Visual Attention; Change Colour, Comparing Size, Sort Circles, Sort Lines, and Vanishing Objects with Visual Discrimination; Circle Location, Circle Right Triangle, Counting Locations, Cross and Knots, and Maze Solving with Visual Spatial Relationships; and Layered Colours, Layered Shapes, Match Outline, and Match Shadow with Visual Closure. Some domains test only one skill, whereas others combine up to four skills, making them more cognitively demanding (Ghosh et al., 28 Aug 2025).

The benchmark’s structural logic can be summarized by indexing domain as $d$ and task size as $s \in \{1,\dots,20\}$, with 10 instances for each $(d,s)$. The crucial empirical axis is therefore $\text{accuracy}(d,s)$ as $s$ increases. Because the task rule is held fixed while visual load increases, degradation with larger $s$ is interpreted as a failure of scalable perception rather than a failure to learn a new reasoning procedure (Ghosh et al., 28 Aug 2025).

4. Evaluation protocol and answer formats

Percept-V is evaluated on GPT-4o, Gemini 2.0 Flash, Claude 3.5 Sonnet v2, OpenAI o4-mini, and DeepSeek R1. The evaluation uses zero-shot prompting, no human-in-the-loop, temperature $= 0$, and a token budget capped at 1000 tokens. Chain-of-thought prompting was not used, because it was found unnecessary and prior evidence was noted to support only modest or narrow gains from CoT in related task types (Ghosh et al., 28 Aug 2025).

Each prompt has three parts: Input, Rules, and Output. Input describes the image type, Rules explains how to solve the task, and Output specifies the exact output format. This strict structure is operationally important because model outputs are automatically parsed using regex into JSON for evaluation (Ghosh et al., 28 Aug 2025).

The benchmark defines four response formats. A Single Answer requires one value and is correct only if it matches exactly. Paired Answers require a tuple of values, both of which must be correct. A List Answer requires all values to be correct in the correct order. A Set Answer requires all values to be present, but order does not matter. For each task size in each domain, 10 candidate problems are evaluated and model accuracy is averaged across those 10 instances (Ghosh et al., 28 Aug 2025).

5. Empirical findings and interpretive significance

The principal empirical finding is that accuracy drops sharply as task size increases across all domains and all models. On small tasks, models often achieve near-perfect accuracy; on larger tasks, performance does not scale. Because the core rule of a domain remains unchanged as size grows, the reported decline is interpreted as evidence that the tested models are not mastering the underlying perception task, but are instead handling only the simplest cases (Ghosh et al., 28 Aug 2025).

A second finding is that performance trends are consistent across models. For a given perceptual skill, the tested MLLMs exhibit similar accuracy trends across problem size. The paper interprets this as evidence of similar training biases across models, limited adaptation to unfamiliar visual structures, and shared weaknesses in basic perception under compositional load. The failure mode is therefore presented as systematic rather than idiosyncratic (Ghosh et al., 28 Aug 2025).

A third finding is that some perceptual skills are easier than others. Gemini performs best on Visual Figure Ground and Visual Form Constancy, Claude performs best on Visual Closure, and Gemini tends to outperform the others in a majority of domains. The paper does not claim a universal hierarchy of skills, but it does argue that the benchmark exposes differential robustness across perceptual abilities (Ghosh et al., 28 Aug 2025).

A common misconception directly addressed by these results is that strong multimodal reasoning on prominent benchmarks implies strong visual perception. Percept-V argues the opposite: good performance on easy perception problems does not imply robust visual perception. Complexity reveals hidden weakness, especially when the number of objects, distractors, or sub-steps increases while the logical form of the task stays constant (Ghosh et al., 28 Aug 2025).

6. Relation to broader perception research

Percept-V belongs to a broader research movement that treats perception as a distinct object of study rather than as a by-product of end-task reasoning. In image representation learning, a related line of work shows that perceptual properties can emerge from optimizing a biologically inspired vision model for image reconstruction without human perceptual labels; in that setting, the strongest alignment with human perceptual judgments appears in a V1-like encoder stage, and the best correlations occur under moderate noise, blur, and sparsity rather than extreme regularization (Hernández-Cámara et al., 14 Aug 2025). Although the modality and objective differ, both lines of work emphasize that perceptual structure should be examined directly, not inferred from downstream task success alone.

In assistive multimodal systems, a complementary theme is that reliable perception requires explicit grounding. An LVLM-based environment-perception system for visually impaired people uses semantic segmentation as external knowledge, injecting segmented object categories into the prompt to reduce hallucination during global or local scene description. That work couples wearable sensing, segmentation, and language generation, and reports improved grounding on POPE, MME, and LLaVA-QA90 relative to Qwen-VL-Chat (Chen et al., 25 Apr 2025). In that context, perception is operationalized not as benchmark accuracy on elementary tasks, but as trustworthy scene understanding under real-world interaction constraints.

A plausible implication is that Percept-V strengthens the case for separating perceptual competence from reasoning competence in VLM development. Later work on two-stage reinforcement learning explicitly argues that VLMs must first accurately perceive and understand visual inputs before reasoning can be effectively performed, and trains perception before reasoning for that reason (Chen et al., 16 Sep 2025). Percept-V supplies an especially clean motivation for such a separation: if controlled basic perception collapses under increasing compositional load, then reasoning-oriented evaluation alone is insufficient to characterize visual capability.

Percept-V therefore occupies a specific place in contemporary multimodal research. It is neither a general-purpose VLM benchmark nor a domain-specific visual QA suite. It is a controlled instrument for testing whether a system can actually perceive what is in the image as visual complexity increases. Its significance lies less in introducing exotic tasks than in demonstrating that elementary perception, when isolated and scaled systematically, remains a major bottleneck for current multimodal models (Ghosh et al., 28 Aug 2025).