Multimodal Visual Patterns (MMVP)

• MMVP is a framework that formalizes complex visual patterns through abstract rules governing objects, spatial layouts, and relationships.
• It benchmarks vision-language models by revealing their shortcomings in visual grounding, inductive pattern recognition, and structured reasoning.
• The framework employs diagnostic tasks such as CLIP-blind pairs and PuzzleVQA to quantify performance gaps and guide model improvements.

Multimodal Visual Patterns (MMVP) provide a rigorous lens on the visual understanding capabilities—and remaining deficiencies—of state-of-the-art vision–LLMs (VLMs) and multimodal LLMs (MLLMs). MMVP formalizes the space of visual patterns that require precise image perception and structured reasoning, advancing diagnostic granularity beyond traditional language–image alignment metrics. Contemporary studies leverage MMVP as both a benchmark (grounded in real-image “CLIP-blind” pairs (Tong et al., 11 Jan 2024)) and a formal framework (as in PuzzleVQA (Chia et al., 20 Mar 2024)) to evaluate failure modes in model visual grounding, inductive pattern recognition, and the intertwined language–vision interface.

1. Formal Definition and Taxonomy

In the MMVP paradigm, a multimodal visual pattern is an abstract relationship $P$ over conceptual visual objects $\mathcal{O}$, spatial layout $\mathcal{L}$, and a governing relation $R$. Each pattern instance $P \in \mathbb{P}$ is defined as:

• $\mathcal{O} = \{o_1, ..., o_n\}$: Set of conceptual objects (e.g., shapes, numbers, colors, sizes)
• $\mathcal{L}$: Layout mapping objects to image coordinates
• $R$: Abstract rule over $\mathcal{O}$ (e.g., symmetry, arithmetic progression)

Given raw pixel input $V \in \mathbb{R}^{H \times W \times C}$, models must implement a perception module $f_p$ that infers attributes $\hat{\mathcal{O}}$, and a reasoning pipeline that links these to infer or recognize $R$ and answer questions $x_q \in \mathcal{V}^*$ with discrete options $\mathcal{O}$ (Chia et al., 20 Mar 2024).

Patterns span a taxonomy of concepts:

Concept	Example Atoms	Role
Colors	Red, green, blue, etc.	Attribute detection
Numbers	1–9, sequences	Counting, ordering
Shapes	Triangle, pentagon, etc.	Form, structure
Size	Small, large	Comparison

MMVPs encompass both single-concept and dual-concept patterns, with duals (e.g., “Numbers ∧ Shapes”) requiring compositional relational reasoning.

2. Benchmark Construction and Dataset Design

The MMVP benchmark is constructed from real images that expose systematic deficits in vision encoders, especially the widely deployed CLIP framework. The core data-crafting procedure is:

• CLIP-blind pairs: Image pairs $(x_i, x_j)$ s.t. cosine similarity $s_\text{CLIP}(x_i, x_j) > 0.95$ but vision-only SSL similarity $s_\text{DINO}(x_i, x_j) < 0.60$, identifying image pairs nearly indistinguishable in CLIP space but separable by DINOv2 (Tong et al., 11 Jan 2024).
• Manual vetting: Minimal, unambiguous visual differences are confirmed by human annotators.
• Pattern taxonomy: Questions are clustered (using GPT-4) into nine patterns (see Section 3).
• Annotation: Each pair yields two simple VQA questions with a unique correct answer; evaluation score for a pair requires both to be answered correctly.
• PuzzleVQA synthesis: Automated pipeline (Python, PIL) generates 2,000 abstract pattern puzzles (single/dual concept) with known layouts, rule instantiation, and distractor generation (Chia et al., 20 Mar 2024).

3. Visual Pattern Categories and Difficulty Profiling

In the MMVP benchmark, nine visual pattern categories are defined:

1. Orientation & Direction
2. Presence of Specific Features
3. State & Condition
4. Quantity & Count
5. Positional & Relational Context
6. Color & Appearance
7. Structural & Physical Characteristics
8. Text
9. Viewpoint & Perspective (Tong et al., 11 Jan 2024)

Difficulty profiling quantifies zero-shot accuracy per pattern for VLMs:

Pattern	SOTA CLIP Accuracy	GPT-4V Pair Accuracy	Human Accuracy
Orientation & Direction	26.7%	56%	95.7%
Quantity & Count	~33–40%	(see above)	95.7%
Others (7/9)	<50%	(all <60%)	95.7%

Across nearly all patterns, both CLIP and MLLMs such as GPT-4V are substantially below human performance. Seven out of nine patterns remain under 50% accuracy even in the largest models (Tong et al., 11 Jan 2024).

PuzzleVQA provides further granularity. For 2,000 abstract puzzles, GPT-4V scores 46.4% (single concept), with breakdowns: Numbers (67.5%), Colors (42.0%), Size (35.0%), Shapes (41.0%). The random baseline is ∼27.1% (Chia et al., 20 Mar 2024).

4. Reasoning Pipeline and Model Evaluation

Solving MMVP tasks involves a chain-of-thought decomposition:

• Visual Perception ($f_p$): Extraction of symbolic attributes from image features (e.g., color sequences, shape labels)
• Inductive Reasoning ($f_i$): Identifying governing rule $R$ from demonstrations, mapping observed patterns to abstract relations (“opposites match color”)
• Deductive Reasoning ($f_d$): Applying inferred $R$ to masked/query regions to select answer (Chia et al., 20 Mar 2024)

Performance is measured via exact-match accuracy; for CLIP-blind pairs, both VQA sub-questions must be answered correctly (Tong et al., 11 Jan 2024). In PuzzleVQA, upper-bound performance is evaluated by sequentially providing ground-truth explanations for perception, induction, and deduction, revealing which stage is the principal bottleneck.

Stage Guidance	GPT-4V	Claude 3	Gemini Pro	LLaVA-13B
Q only	46%	39%	34%	27%
+Perception	70%	72%	43%	33%
+Perception & Induction	97%	98%	55%	41%
+All Three	99%	98%	80%	58%

Main errors are traced to perception failures (misclassified/missed attributes) and faulty induction (hypothesized patterns do not match ground-truth), rather than deduction slips.

5. Systematic Failures and Hallucinated Rationales

Empirical evaluation reveals that state-of-the-art MLLMs (InstructBLIP, MiniGPT-4, LLaVA, LLaVA-1.5, Bard, Gemini, GPT-4V) are challenged by otherwise trivial questions focused on basic visual patterns. GPT-4V, while outperforming open-source counterparts (at ~56% vs. 14–28% accuracy), still fails a substantial fraction of queries (Tong et al., 11 Jan 2024).

A pervasive failure mode is “hallucinated explanation,” wherein models generate plausible post hoc rationales for incorrect answers (e.g., inferring a closed umbrella due to “no ribs on the canopy” when the visual evidence is ambiguous). This behavior indicates the LLM is compensating for the deficiency of the vision module with its language prior, rather than actual visual grounding. The direct propagation of CLIP’s blind spots into VLMs is statistically demonstrated by the correlation between CLIP-blind pair difficulty and MLLM performance (Tong et al., 11 Jan 2024).

6. Mixture-of-Features (MoF): Diagnosing and Remediating Blind Spots

To probe and address MMVP deficits, the Mixture-of-Features (MoF) approach is introduced (Tong et al., 11 Jan 2024), instantiated in two primary configurations:

• Additive-MoF (A-MoF): Forms mixed embedding $z = \alpha e_\text{CLIP} + (1-\alpha) e_\text{SSL}$, sweeping $\alpha \in [0,1]$, allowing dynamic trade-off between CLIP and SSL (e.g., DINOv2) features.
• Interleaved-MoF (I-MoF): Passes CLIP and SSL visual tokens through separate adapters, then interleaves token streams before input to the LLM.

Key findings:

Method	MMVP Acc.	LLaVA Instr. Follow.	POPE Halluc.
LLaVA (base)	5.5%	81.8%	Baseline
Add. MoF	18.7%	~75%	–
Int. MoF	16.7%	81.8%	↓ by ~0.4%
LLaVA-1.5	24.7%	–	–
LLaVA-1.5+MoF	28.0%	–	↓ by ~0.4%

Interleaved-MoF largely restores instruction-following capacity while significantly improving MMVP accuracy. The data indicate that vision-only SSL features (e.g., DINOv2, MAE, MoCoV3) encode complementary visual patterns neglected by CLIP. However, naïve additive fusion can trade off language alignment, whereas interleaving better preserves linguistic fidelity.

7. Implications and Future Directions

The MMVP framework isolates fundamental open challenges in vision–language pre-training:

• Root cause: CLIP’s instance-level contrastive pre-training does not explicitly encode orientation, counting, small text, etc., producing systematic blind spots inherited by downstream VLMs (Tong et al., 11 Jan 2024).
• Scaling limitations: Increasing model size, data scale, or backbone strength confers minimal gains on most patterns; e.g., only two of nine MMVP categories substantially improve with scaling (EVA-CLIP, MetaCLIP, DFN).
• Benchmark and objective design: Systematic evaluations (e.g., MMVP, PuzzleVQA) are required that probe the spectrum of visual patterns necessary for downstream agents, surpassing legacy benchmarks such as ImageNet zero-shot.
• Model architecture: Modular reasoning pipelines based on “perceive–induce–deduce” principles offer a route to isolating and rectifying pattern-specific failures (Chia et al., 20 Mar 2024). Integration with symbolic pattern-mining (“program-aided solvers”) represents a promising research trajectory.
• Feature fusion: More sophisticated intra-token and inter-stream integration strategies beyond simple mixing may unlock further gains, especially for compositional abstraction.
• End-to-end optimization: Direct co-training of language and vision modules is posited as a means to close the CLIP-blind gap without relying on ensembling after the fact.

This suggests that accurate visual grounding remains a fundamental roadblock. MMVP-centric analysis and benchmarks expose critical deficiencies in current MLLM pipelines and provide actionable insights for next-generation model development (Tong et al., 11 Jan 2024, Chia et al., 20 Mar 2024).