ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs

Abstract: Recent advances in multimodal LLMs (MLLMs) have greatly improved image understanding and captioning capabilities. However, existing image captioning benchmarks typically suffer from limited diversity in caption length, the absence of recent advanced MLLMs, and insufficient human annotations, which potentially introduces bias and limits the ability to comprehensively assess the performance of modern MLLMs. To address these limitations, we present a new large-scale image captioning benchmark, termed, ICBench, which covers 12 content categories and consists of both short and long captions generated by 10 advanced MLLMs on 2K images, resulting in 40K captions in total. We conduct extensive human subjective studies to obtain mean opinion scores (MOSs) across fine-grained evaluation dimensions, where short captions are assessed in terms of fluency, relevance, and conciseness, while long captions are evaluated based on fluency, relevance, and completeness. Furthermore, we propose an automated evaluation metric, \textbf{ITIScore}, based on an image-to-text-to-image framework, which measures caption quality through reconstruction consistency. Experimental results demonstrate strong alignment between our automatic metric and human judgments, as well as robust zero-shot generalization ability on other public captioning datasets. Both the dataset and model will be released upon publication.

• The paper introduces ITIScore, a novel image-to-text-to-image metric that achieves the highest correlation with human judgment for captioning.
• It presents ICBench, a large-scale benchmark with extensive human annotations assessing fluency, relevance, conciseness, and completeness for both short and long captions.
• Experimental validation demonstrates ITIScore's superior performance across diverse MLLMs and benchmarks, highlighting its potential in real-world multimodal applications.

ITIScore: An Image-to-Text-to-Image Evaluation Metric for Image Captioning by MLLMs

Introduction and Motivation

The rapid progress in Multimodal LLMs (MLLMs) has brought significant advancements in image captioning. Despite these developments, evaluating generated captions—a critical component for both model development and real-world deployment—remains challenging. Main limitations of existing benchmarking efforts include restricted caption length diversity, lack of human annotations at scale and granularity, absence of comprehensive assessment for state-of-the-art MLLMs, and potential data leakage in open datasets. The paper introduces two key contributions to address these, (i) ICBench, a large-scale, high-diversity benchmark for captioning, and (ii) ITIScore, a novel automatic metric based on image-to-text-to-image reconstruction and multimodal reasoning.

ICBench: A Comprehensive Captioning Benchmark

ICBench is constructed to enable detailed and reliable evaluation of MLLM image captioning capabilities across both short and long captions.

ICBench is comprised of 2,040 images sampled from 12 diverse categories, with each image described by both a short and a long caption generated by each of 10 advanced MLLMs (including closed-source and open models). This results in 40,800 total captions. Captions are systematically annotated by 15 professional human judges, yielding 1.8 million Mean Opinion Scores (MOS) over multiple quality dimensions: fluency, relevance, and conciseness for short captions; fluency, relevance, and completeness for long captions.

Figure 1: MOS distribution of short caption and long caption across different evaluation dimensions.

Analysis of the MOS distributions demonstrates systematic weaknesses even among advanced MLLMs, particularly in generating succinct and comprehensive captions, with divergence between short- and long-caption abilities. Variance by category and caption type is nontrivial and provides a solid empirical foundation for rigorous benchmarking.

Figure 2: Performance comparison of different MLLMs on short captioning (fluency, relevance, conciseness) and on long captioning (fluency, relevance, completeness) across different image contents.

The comparative results reveal that while closed-source models (e.g., ChatGPT-5, Gemini-3.0-Flash) typically outperform open-source models across most dimensions, category-level and dimension-specific challenges remain, especially for images demanding high-level or domain-specific reasoning.

ITIScore: Image-to-Text-to-Image Quality Assessment Framework

The paper introduces ITIScore as an automatic metric for caption quality assessment, leveraging the image-to-text-to-image paradigm. Unlike conventional reference-based or reference-free textual metrics, ITIScore operationalizes caption evaluation as a reconstruction consistency problem: does the generated caption encode sufficient, correct, and visually faithful information such that a generative model can reconstruct the original image, and does a multimodal backbone extract agreement between original, reconstructed images, and text? This is further enhanced by a learned, uncertainty-aware scoring head.

Figure 3: Overview of our ITIScore. Given an image and its caption, a pretrained generative model reconstructs an image from the caption. The original image, reconstructed image, and caption are jointly fed into a multimodal LLM to obtain a unified representation. A lightweight MLP scoring head then predicts the mean score and uncertainty for each evaluation dimension.

ITIScore System Design

• Caption-Conditional Image Reconstruction: Given an image-caption pair $(I, C)$, a pretrained generative model $G$ reconstructs $\hat I = G(C)$.
• Multimodal Consistency Embedding: The tuple $(I, \hat I, C)$ is processed by a frozen MLLM backbone, which produces a unified embedding $H = f_{MLLM}(I, \hat I, C)$.
• Dimension-Aware Scoring: A lightweight MLP scoring head predicts the mean ($\mu$) and uncertainty ($\sigma$) for each evaluation dimension, modeled as a Gaussian. Instruction tuning enables specific dimension queries (e.g., "fluency" or "completeness").
• Uncertainty-Aware Multi-Dimensional Learning: Training objective is Gaussian negative log-likelihood, encouraging accurate mean prediction with uncertainty calibration. An auxiliary regression loss is used to stabilize mean aggregate quality predictions.

This procedure creates an automatic, reference-free metric that robustly approximates fine-grained human judgment.

Experimental Validation

The ITIScore framework is extensively benchmarked against strong reference-based (BLEU, METEOR, ROUGE, CIDEr, SPICE), reference-free (e.g., CLIP-S, PAC-S), and state-of-the-art MLLM-based metrics (e.g., FLEUR, ChatGPT-5, Gemini-3.0-Flash).

Key Results:

• ITIScore achieves the highest correlation with human judgment across all tested dimensions, for both short and long captions, and in overall aggregate rankings.
• ITIScore maintains superior zero-shot correlation performance on established external benchmarks (Composites, Flickr8k-CF, Flickr8k-Expert, MMHE, LongCap-Arena).
• Ablation studies confirm architecture choices: image reconstruction and multi-dimensional, uncertainty-aware regression substantially elevate human alignment over classical and MLLM-only approaches.

Implications and Future Directions

The ICBench resource and ITIScore metric set new standards for evaluating modern captioning models across typical and challenging settings. ITIScore’s architecture not only surpasses prior approaches in human-alignment but also enables nuanced evaluation of captioning quality along axes that matter in practical deployments, including in safety/robustness-sensitive and domain-centric applications.

The rigor of ICBench will facilitate fairer and more discerning comparison of future MLLMs, and ITIScore suggests that model-based evaluation using reconstruction and multimodal fusion is both highly effective and extensible. Further research can explore generalizing this methodology to other multimodal generation and evaluation tasks (e.g., video captioning, text-to-image synthesis) and the relationship between generative reconstruction quality and high-level semantic fidelity.

Conclusion

This work addresses persistent gaps in image captioning evaluation by delivering (i) ICBench, a large-scale fine-grained benchmark with extensive human annotation for short and long captions, and (ii) ITIScore, a robust, uncertainty-aware, image-to-text-to-image metric benchmarked to human preference. ITIScore consistently matches or exceeds human correlation metrics compared to alternative evaluation approaches, and generalizes across datasets and caption styles. This methodology and resource stand to inform the next stage of multimodal model development and evaluation.