AVI-Bench-PriSe: Primitive Sensation Benchmark

• AVI-Bench-PriSe is a benchmark that extends traditional audiovisual evaluations by using low-semantic, synthetic stimuli to probe primitive sensory competence.
• It employs controlled synthetic sounds and rendered visuals along with cross-modal multiple-choice and double-confirmation protocols to test model adaptation.
• Empirical results reveal substantial performance gaps between Omni-MLLMs and human baselines, emphasizing limitations in low-level sensory perception under unfamiliar domain shifts.

AVI-Bench-PriSe is the “Primitive Sensation” extension to AVI-Bench, introduced to evaluate whether Omni-Multimodal LLMs (Omni-MLLMs) can perform low-level audio-visual judgments on unfamiliar, low-semantic stimuli rather than relying on high-semantic, domain-familiar patterns. Within the AVI-Bench framework, PriSe operationalizes a fourth stage beyond Perception, Understanding, and Reasoning by probing primitive sensory competence with synthetic sounds and rendered visuals, using cross-modal multiple-choice questions and a double-confirmation protocol. Its stated purpose is to quantify domain-adaptation, defined as Level-4 in the AVI taxonomy, and to diagnose whether current models genuinely perceive basic sensory features under substantial domain shift (Wang et al., 1 Jun 2026).

1. Position within the AVI-Bench hierarchy

AVI-Bench organizes audio-visual intelligence as a four-stage, human-inspired hierarchy: Perception, Understanding, Reasoning, and Primitive Sensation. The first three stages evaluate detection, classification, localization, narrative integration, cross-modal retrieval, coarse-grained question answering, fine-grained language grounding, and hallucination detection on high-semantic, domain-familiar data. PriSe corresponds specifically to the fourth stage, Primitive Sensation, and is explicitly separated from the earlier task suites (Wang et al., 1 Jun 2026).

A central point in the design is that PriSe does not reuse the Perception, Understanding, or Reasoning tasks. Instead, it introduces a new suite of Sensation tasks intended to occur after training on high-semantic data, thereby stress-testing unfamiliar-domain adaptation. This makes PriSe conceptually distinct from standard multimodal benchmarks that evaluate semantic understanding in familiar real-world settings. A plausible implication is that the benchmark is intended less as a measure of world knowledge than as a probe of whether low-level sensory abstractions survive outside the training distribution.

2. Motivation and conceptual target

The benchmark is motivated by a specific gap in evaluation: although Omni-MLLMs have shown strong integration of vision, audio, and language, their audio-visual intelligence remains insufficiently characterized if evaluation is confined to familiar, semantically rich inputs. PriSe addresses the question of whether such models can exhibit human-like low-level sensation on inputs far removed from common training distributions (Wang et al., 1 Jun 2026).

The benchmark therefore uses controlled, unfamiliar, low-semantic stimuli and asks primitive sensory questions such as “Which sound is louder?” and “Which shape is rotating?”. The target of evaluation is not category recognition, event understanding, or narrative inference, but basic judgments over attributes such as volume, motion, and shape. The design differences from the main AVI-Bench tasks are explicit:

• Low semantic content: basic geometry and pure tones.
• Domain shift: no real-world context and minimal language cues.
• Primitive judgment: volume, motion, and shape rather than category or event understanding.

This framing also addresses a common misconception: PriSe is not a harder version of conventional multimodal question answering in the same domains. Its purpose is to measure unfamiliar-domain sensory competence under deliberately reduced semantics.

3. Stimulus construction and curation workflow

All PriSe stimuli were constructed offline, with no real photographs and no recorded sounds. The visual side consists of computer-rendered shapes—cubes, pyramids, cones, and spheres—undergoing geometric transformations including rotation, scaling, and translation, with either uniform textures or simple gradients. The audio side consists of synthetic tone sweeps, white or pink noise bursts, and simple percussion hits, each with controlled volume, duration, and timbre (Wang et al., 1 Jun 2026).

The curation pipeline is also specified. After a brief style guide, three annotator groups collaborated with differentiated responsibilities:

• Group B: generated the raw stimuli.
• Group C: provided example-based alignment and reviewed every 25% of the data.
• Group A: performed final verification and formatting.

This workflow indicates that PriSe was designed as a controlled benchmark rather than as a harvested dataset. A plausible implication is that the benchmark emphasizes internal experimental consistency and precise manipulation of sensory variables over ecological realism.

4. Task suite and dataset composition

Stage 4, Primitive Sensation, comprises four tasks and approximately 1,415 samples, reported as about 24% of the 5,864 total samples in AVI-Bench. The task inventory is as follows (Wang et al., 1 Jun 2026):

Task	Description	Sample count
ASQA	Audio Sensation QA	462
VSQA-image	Visual Sensation QA	306
VSQA-video	Visual Sensation QA	318
AVSQA	Audio–Visual Sensation QA	329

The task formulations are explicitly sensory. ASQA includes questions such as “Which of these two tones is louder?” with a confirmation prompt such as “Can you hear a sound at all?”. VSQA, in image and video forms, includes prompts such as “Which shape is rotating fastest?” with confirmation via “Is any object moving?”. AVSQA adds temporal and cross-modal alignment, for example: “Which object moves in sync with the longer sound?” plus “Do you see an object at that time?” (Wang et al., 1 Jun 2026).

The presence of both image- and video-based VSQA is notable because Primitive Sensation is not confined to static visual discrimination. The AVSQA task further requires joint reasoning over audio and visual timing, but at a primitive sensory level rather than at the level of semantic events. This suggests that PriSe is structured to isolate elementary cross-modal correspondences under reduced semantic priors.

5. Scoring protocol and formal metrics

All PriSe tasks are cast as multiple-choice questions with a double-confirmation mechanism. For each sample $i$, the benchmark defines an original question $q_i$, a confirmation question $c_i$, model answers $\hat y_i$ and $\hat z_i$, and ground-truth answers $y_i$ and $z_i$. A sample is counted as correct only if both answers are correct:

$$a_i = \begin{cases} 1 & \text{if } \hat y_i = y_i \;\wedge\; \hat z_i = z_i,\ 0 & \text{otherwise.} \end{cases}$$

The final Stage-4 score is the average over $N$ samples:

$S_{\mathrm{PriSe}} = \frac{1}{N}\sum_{i=1}^N a_i.$

The double-confirmation mechanism is structurally important because it penalizes answers that guess the primary choice correctly while failing the associated basic existence or consistency check. In the examples provided, a model may answer a comparative auditory question yet fail “Can you hear anything?”, which is treated as an error under the metric (Wang et al., 1 Jun 2026).

The evaluation setup is standardized across all 28 Omni-MLLMs, including both open-source and closed-source systems. All models are evaluated with the same prompts, using seed $q_i$0, and no manual prompt tuning. Closed-source models are accessed through publicly documented API versions, with “gemini-2.5-pro-preview-05-06” given as an example. This protocol reduces benchmark variance attributable to prompt engineering and makes the resulting differences more directly attributable to model capability rather than evaluator intervention.

6. Empirical performance and characteristic failure modes

The reported Stage-4 results indicate that Primitive Sensation is substantially challenging for current Omni-MLLMs. In the excerpted results, Gemini-2.5-pro attains ASQA $q_i$1, VSQA $q_i$2, AVSQA $q_i$3, and an average $q_i$4 of $q_i$5. Gemini-2.5-flash records $q_i$6, $q_i$7, $q_i$8, and $q_i$9, while Gemini-2.0-flash records $c_i$0, $c_i$1, $c_i$2, and $c_i$3. Averaged over all 28 models, the benchmark reports ASQA $c_i$4, VSQA $c_i$5, AVSQA $c_i$6, and overall $c_i$7 (Wang et al., 1 Jun 2026).

Several failure modes are identified directly. Audio sensation is described as the hardest task family, with the top model reaching only about $c_i$8 on ASQA. Cross-modal sensation is reported as worst overall, with the top model reaching $c_i$9 on AVSQA. The benchmark also notes that models often fail simple confirmation questions, including hallucinations on prompts such as “Can you hear anything?” (Wang et al., 1 Jun 2026).

The human baseline, measured with 6 subjects, is approximately $\hat y_i$0 on PriSe average, with a cited AVSQA contrast of $\hat y_i$1 versus $\hat y_i$2. This establishes a large sensory gap between current models and human performance. The contrast is particularly sharp because PriSe tasks are intentionally low-semantic; poor results cannot be attributed solely to failures of high-level narrative interpretation. This suggests that current Omni-MLLMs remain limited in unfamiliar-domain sensory grounding even when the queried attributes are elementary.

7. Role in the four-level AVI taxonomy

PriSe is not only a standalone benchmark component but also a constitutive element of AVI-Bench’s four-level taxonomy. In that taxonomy, PriSe provides the Unfamiliar-Domain score $\hat y_i$3, which is harmonically combined with the Familiar-Domain Stage-3 score $\hat y_i$4 to form the domain-adaptive score $\hat y_i$5:

$\hat y_i$6

This construction makes weak unfamiliar-domain performance strongly consequential for Level-4 ranking. In the excerpted table, Gemini-2.5-pro has $\hat y_i$7, $\hat y_i$8, $\hat y_i$9, and $\hat z_i$0; Gemini-2.5-flash has $\hat z_i$1, $\hat z_i$2, $\hat z_i$3, and $\hat z_i$4; Gemini-2.0-flash has $\hat z_i$5, $\hat z_i$6, $\hat z_i$7, and $\hat z_i$8; and Human-Omni-0.5B has $\hat z_i$9, $y_i$0, $y_i$1, and $y_i$2 (Wang et al., 1 Jun 2026).

The benchmark explicitly states that PriSe’s low $y_i$3 values drive down $y_i$4 for all models, making Domain-Adaptive intelligence a stringent and human-aligned benchmark. This clarifies another possible misunderstanding: strong performance in familiar-domain perception, understanding, or reasoning does not automatically imply high Level-4 capability. In the AVI-Bench formulation, domain-adaptive audio-visual intelligence is bottlenecked by primitive sensation under unfamiliar conditions.

Within the broader evaluation logic of AVI-Bench, PriSe therefore serves a diagnostic function. It tests whether multimodal competence extends beyond semantic familiarity into sensory generalization. The benchmark’s reported results indicate that, for current Omni-MLLMs, this remains an unresolved capability boundary.