Same or Not? Enhancing Visual Perception in Vision-Language Models

Abstract: Vision-LLMs (VLMs) excel at broad visual understanding but remain coarse-grained, exhibit visual biases, and miss subtle visual details. Existing training corpora reinforce this limitation by emphasizing general recognition ("Is it a cat or a dog?") over fine-grained perception. To address this, we introduce a new training corpus and task designed to enhance the perceptual abilities of VLMs. TWIN is a large-scale dataset of 561,000 image-pair queries that task models to determine whether two visually similar images depict the same object, encouraging attention to nuanced visual cues. The dataset spans a diverse range of everyday objects across contexts, viewpoints, and appearances. Fine-tuning VLMs on TWIN yields notable gains in fine-grained recognition, even on unseen domains such as art, animals, plants, and landmarks. To quantify these gains, we introduce FGVQA, a benchmark suite of 12,000 queries that repurposes fine-grained recognition and retrieval datasets from multiple domains. While existing VLMs struggle on FGVQA, when fine-tuned on TWIN they improve by up to 19.3%, without compromising performance on general VQA benchmarks. Finally, our TWIN dataset scales favorably with object annotations, and our analysis shows that scale is key to performance. We envision TWIN as a drop-in addition to open-source VLM training corpora, advancing perceptual precision of future models. Project webpage: https://glab-caltech.github.io/twin/

• The paper introduces a novel instance-level pairwise training scheme using the Twin dataset to improve fine-grained visual perception in VLMs.
• It employs reinforcement learning post-training with Group Relative Policy Optimization to enhance model stability and targeted perceptual precision.
• Extensive experiments on the Fgvqa benchmark demonstrate significant accuracy gains and robust cross-domain transfer in instance discrimination.

Enhancing Fine-Grained Visual Perception in Vision-LLMs via Instance-Level Pairwise Training

Motivation and Problem Formulation

Current VLMs exhibit strong capabilities in broad visual reasoning but are systematically limited in fine-grained perception, failing to reliably discern subtle differences between visually similar entities. The paper addresses the insufficiency of existing datasets, which predominantly promote category-level understanding and neglect instance-level visual discrimination. This bias results in models that conflate semantically similar but physically distinct instances, impeding downstream tasks requiring acute perceptual precision, such as embodied AI and detailed retrieval.

To rectify this, the authors propose a new training paradigm and evaluation suite centered on fine-grained instance-level comparisons, operationalized through a large-scale dataset—Twin—and a multi-domain benchmark—Fgvqa.

The Twin Dataset: Scalable Instance-Level Supervision

The Twin dataset comprises 561k image-pair VQA queries, each requiring a VLM to determine if two images represent the same real-world object instance. Unlike previous pairwise datasets (e.g., Birds-to-Words [birdstowords], SpotTheDiff [spotthediff]), Twin is both significantly larger and explicitly targets the discrimination of subtle cues in shape, texture, and geometry under significant intra-instance variation (background, lighting, viewpoint). Its construction incorporates:

• Systematic sourcing of diverse household object instances.
• Human-verified positive pairs and CLIP-facilitated hard negative mining.
• Synthetically generated negatives using DreamBooth, enabling challenging near-duplicate negatives at scale.

Figure 1: Illustration of the Twin dataset design and instance diversity.

This scalable approach leads to a quadratic increase in training queries with respect to instance count, providing a unique inductive bias for instance-level recognition.

Fgvqa: A Cross-Domain Fine-Grained VQA Benchmark

Existing VQA benchmarks insufficiently probe subtle perceptual skills. Fgvqa consists of 12,000 queries spanning six domains: retail (Twin, Ilias), wildlife (Inquire, CUB), landmarks, and artwork (MET). Query formats include (a) pairwise—does image A match B?—and (b) multi—how many candidates match a reference?

Figure 2: Fgvqa is a suite of fine-grained VQA benchmarks spanning retail, animals/plants, landmarks, and art.

This design rigorously evaluates models' ability to generalize instance discrimination beyond training distributions, measuring both in-domain and cross-domain transfer.

Reinforcement Learning Post-Training on Twin

The methodology for enhancing VLMs involves post-training via reinforcement learning (RL) using Group Relative Policy Optimization (GRPO). The reward is a binary indicator of answer correctness for the pairwise query, and the optimization is constrained to maintain stability with respect to the pretrained policy, preventing catastrophic forgetting.

Figure 3: Training VLMs on Twin with RL; the reward is based on predicted vs. ground truth instance assignment.

Significantly, this framework requires no descriptive textual labels—supervision is entirely via the instance identity binary.

Experimental Results: Strong Gains in Perceptual Precision

Quantitative Main Results

On Fgvqa, models post-trained on Twin achieve large relative gains in fine-grained accuracy, especially in out-of-distribution settings:

• Qwen2.5-VL (3B): Absolute accuracy increases of +18.3% on Ilias, +14.4% on CUB, and +19.3% on Inquire.
• Improvements generalize to domains unseen during training (art, landmarks, wildlife).

Qualitative Analysis

Post-trained models generate more reasoned, context-aware explanations, identifying subtle cues (e.g., part geometry, texture) previously overlooked by the base model, including in zero-shot cross-domain evaluations.

Figure 4: Outputs on Fgvqa for base VLMs and Twin-trained variants; Twin enables more precise visual grounding and error correction.

Data Efficiency and Scaling Laws

Performance gains demonstrably correlate with training set size. Increasing the image pair count from 5k to 561k yields monotonic improvement not only in in-domain, but also in out-of-domain transfer, underscoring the necessity of dataset scale for perceptual acuity.

Figure 5: Effect of training set size on Fgvqa accuracy; larger Twin subsets accelerate both in-domain and cross-domain gains.

Hard negatives—especially human-verified and DreamBooth-synthesized variants—are critical; their removal results in substantial accuracy degradation.

Retention of General Skill

Twin post-training does not degrade model performance on generic VQA or text-only benchmarks (RealworldQA, MMLU, Gsm8K). Minor improvements on spatial and logic-heavy benchmarks are occasionally observed.

Encoder Representation Enhancement

Linear and KNN probing show improved downstream accuracy on fine-grained classification datasets (e.g., Pets, Sun397, CIFAR100) from vision encoders of Twin-trained models, indicating that encoder representations themselves gain finer discriminatory power.

Comparative Analysis

Among open-source VLMs, Twin-trained Qwen2.5-VL surpasses unmodified baselines and closes a significant portion of the gap to proprietary models such as Gemini and GPT-4 on fine-grained instance discrimination. Notably, leading proprietary models presumably benefit from much larger and internally curated datasets, highlighting the potential accessibility and scalability impact of Twin.

Limitations and Future Work

Although Twin substantially elevates fine-grained perception, challenges persist, notably in cases with extreme viewpoint, lighting, or occlusion artifacts. The authors suggest advancing reward modeling (e.g., using multimodal verifiers), automating negative mining (leveraging model-in-the-loop active data engines), and incorporating 3D perception are promising avenues. Scaling Twin further in terms of both domain diversity and instance coverage would likely yield additional improvements.

Figure 6: Additional examples from Fgvqa illustrating the challenge and diversity in instance-level visual reasoning.

Conclusion

This work demonstrates that targeted, large-scale instance-level pairwise supervision via the Twin dataset, coupled with robust RL-based post-training, yields substantial improvements in fine-grained visual perception for VLMs. Gains are realized not only in in-domain tasks but—importantly—transfer to diverse domains, establishing a scalable paradigm for advancing cross-domain perceptual precision in open-source multimodal models. This approach is readily extensible, and its integration into future VLM training pipelines will likely accelerate progress toward robust embodied and task-specialized vision-language reasoning.

(2512.23592)