Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis

Abstract: Recent advancements in Large Vision-LLMs (LVLMs) have demonstrated remarkable multimodal perception and reasoning capabilities. While numerous benchmarks have evaluated LVLMs from holistic or task-specific perspectives, their capabilities on fine-grained image tasks-fundamental to computer vision-remain insufficiently understood. To address this gap, we introduce FG-BMK, a comprehensive fine-grained evaluation benchmark containing 1.01 million questions and 0.28 million images, covering diverse scenarios from common object-centric domains to specialized domains. FG-BMK jointly evaluates dialogue-level fine-grained semantic recognition and feature-level visual discriminability through human-oriented and machine-oriented paradigms, enabling diagnostic analysis of whether LVLM failures arise from insufficient visual representations, weak visual-to-semantic grounding, or limited fine-grained knowledge. Through extensive experiments on a diverse set of representative LVLMs/VLMs, we find that current LVLMs remain inadequate fine-grained recognizers, with failures arising from intertwined bottlenecks in visual representations, semantic grounding, modality alignment, and category-level knowledge. We further analyze training design factors for improving fine-grained capabilities and examine how visual and linguistic perturbations affect LVLM predictions. These findings provide diagnostic insights into the limitations of current LVLMs and offer guidance for future data construction and model design in developing more reliable LVLMs for fine-grained visual tasks. Our code is open-source and available at https://fg-bmk.github.io/.

• The paper proposes FG-BMK, a benchmark with over a million question-image pairs drawn from 13 fine-grained datasets to evaluate LVLMs.
• The paper reveals significant performance drops in LVLMs at finer granularities, highlighting issues of semantic misalignment and bias in attribute recognition.
• The study shows that contrastive training methods outperform generative approaches, stressing that data quality and tailored objectives are crucial for fine-grained accuracy.

Benchmarking Large Vision-LLMs on Fine-Grained Image Tasks: A Diagnostic Analysis

Introduction

This work proposes FG-BMK, a diagnostic benchmark that systematically evaluates large vision-LLMs (LVLMs) on fine-grained image recognition tasks. Contrary to holistic or task-specific multimodal benchmarks, FG-BMK fills a crucial gap by offering detailed analysis of LVLM behavior on fine-grained semantic and visual discriminability problems. The study probes models—ranging from instruction-tuned LVLMs to unified multimodal architectures—using a suite of machine- and human-oriented paradigms that encompass over a million question-image pairs spanning both common and specialized domains.

Benchmark Design and Evaluation Protocols

FG-BMK is constructed from 13 curated fine-grained datasets and comprises 1.01 million questions paired with 0.28 million images. The evaluation is organized along two orthogonal axes:

• Human-oriented paradigm: Assesses models via dialogue-centric question answering for semantic grounding. Tasks include attribute recognition (e.g., part-level color, shape, pattern), knowledge bias estimation (category-level accuracy variance), and hierarchical granularity recognition (taxonomy-level questions ranging from class to species).
• Machine-oriented paradigm: Probes models’ learned visual representations via image retrieval and classification tasks. The paradigm interrogates the separability and clustering properties of visual embeddings with both within- and across-meta-category settings.

Evaluation metrics include true/false, multiple-choice, and open-ended (short-answer) question styles for human-oriented tasks, and mean Average Precision (mAP) and Top-1 classification accuracy for machine-oriented tasks. The benchmark is designed to diagnose whether LVLM failures stem from visual representation inadequacy, weak visual-semantic alignment, or limited fine-grained knowledge.

Empirical Findings

Recognition Gaps of Current LVLMs

FG-BMK reveals several salient performance limitations:

• Accuracy Decreases at Finer Granularity: Across all evaluated LVLMs, recognition accuracy sharply declines as taxonomy fineness increases. For example, InternVL3’s accuracy falls from ~99% at the class level to ~61% at the species level on the CUB-200-2011 dataset, a pattern mirrored on iNat2021.
• Lagging Behind Fine-Grained Specialized Models: Even the best open-source and closed-source LVLMs remain inferior to state-of-the-art fine-grained tailored models, with absolute drops up to 29% in domain-specific benchmarks like FGVC Aircraft.
• Uneven Attribute Recognition: Attribute-level accuracy is highly variable; attributes such as color and pattern are recognized with moderate reliability, but shape-based features exhibit notably poor performance (often under 30% accuracy), directly hindering subordinate category reasoning.

Bottleneck Analysis

Through joint machine- and human-level evaluation, the study dissects bottlenecks:

• Semantic Bottleneck in Specialized Domains: In remote sensing and medical domains, linear probes on visual embeddings yield high accuracy (>90%), but dialogue-based recognition remains poor (<70%). This indicates semantic knowledge limitation rather than visual discrimination is dominant in specialized domains.
• Non-equivalence of Visual Discriminability and Semantic Grounding: Unified models (e.g., BLIP3-o, UniWorld-V1) display high discriminability on real images but fail to synthesize defining characteristics when generating category-conditioned images, indicating weak grounding of subordinate category semantics.
• Alignment-Induced Discriminability Loss: Vision-text alignment with mismatched granularity reduces feature separability for fine-grained categories. Fine-grained alignment data can restore discriminability, but the effect is task-dependent: content-level captions improve general reasoning, while category-level alignment specifically benefits fine-grained recognition.
• Training Data-Driven Knowledge Bias: LVLMs exhibit long-tail category bias, recognizing frequent or prominent categories substantially better than rare ones. Balanced fine-tuning mitigates this, implicating LLM pretraining data as the source of the bias rather than innate learning difficulty.

Training Strategy Impact

• Contrastive Paradigm Superiority: Visual encoders trained with contrastive objectives (EVA-CLIP, DINOv2) consistently outperform those trained with generative/reconstruction objectives (BEIT3, Qwen) on all fine-grained tasks, independent of vision encoder scale.
• Marginal Gains from Raw Scaling: Enlarging encoder size or scaling web data without high-quality curation yields only modest improvements in fine-grained discriminability. Data quality and objective design remain the principal drivers.
• Balanced Instruction Tuning: Including both fine-grained and general instruction data during SFT avoids catastrophic forgetting of general multimodal capabilities. Sequential SFT on only fine-grained data degrades generalization severely.

Robustness and Adversarial Susceptibility

• Susceptibility to Perturbations: Visual features are significantly more fragile on fine-grained tasks: adversarial perturbations reduce fine-grained classification accuracy by up to 60%, compared to ∼42% on generic datasets.
• Dominant Influence of Linguistic Priors: Misleading prompts in the language interface cause larger errors than visual perturbations, particularly for true/false queries, underscoring language-centric risks in multimodal models.

Theoretical and Practical Implications

FG-BMK conclusively demonstrates that high-level multimodal competence in LVLMs does not imply robust fine-grained visual-semantic understanding. The interplay between representation learning, semantic grounding, and dataset bias constitutes a multidimensional bottleneck that current LVLM architectures, training regimes, and evaluation strategies do not sufficiently address.

From a theoretical perspective, the decoupling of visual discriminability and semantic alignment signifies a need for granularity-aware alignment techniques and attribute/part-based reasoning modules. Practically, real-world applications such as biodiversity assessment, specialty industrial inspection, remote sensing, and medical diagnostics require not merely generic visual-linguistic understanding, but high-fidelity fine-grained category knowledge and robustness to both visual and linguistic input noise.

Future Directions

The findings suggest several promising research trajectories:

• Attribute- and part-aware LVLMs with explicit grounding in subordinate categories and local cues
• Granularity-matched visual-textual alignment protocols
• Evaluation and training with open-world, fine-grained dynamic scenarios
• Domain-adaptive and continual learning strategies to acquire and maintain specialized category semantics
• Robustness criteria incorporating adversarial and language-side perturbations as core benchmarks

Conclusion

FG-BMK provides an indispensable framework for diagnosing and improving LVLMs in fine-grained visual domains. The comprehensive evaluations and progressive diagnostic approach clarify that scaling model capacity and data is insufficient; advances will arise from redesigning alignment strategies, training objectives, and instruction data composition to explicitly target fine-grained recognition and robust semantic grounding. This benchmark sets a stringent standard and diagnostic pathway for next-generation multimodal AI systems.