T2I-Eval-R1: Reinforcement Learning-Driven Reasoning for Interpretable Text-to-Image Evaluation (2505.17897v1)

Abstract: The rapid progress in diffusion-based text-to-image (T2I) generation has created an urgent need for interpretable automatic evaluation methods that can assess the quality of generated images, therefore reducing the human annotation burden. To reduce the prohibitive cost of relying on commercial models for large-scale evaluation, and to improve the reasoning capabilities of open-source models, recent research has explored supervised fine-tuning (SFT) of multimodal LLMs (MLLMs) as dedicated T2I evaluators. However, SFT approaches typically rely on high-quality critique datasets, which are either generated by proprietary LLMs-with potential issues of bias and inconsistency-or annotated by humans at high cost, limiting their scalability and generalization. To address these limitations, we propose T2I-Eval-R1, a novel reinforcement learning framework that trains open-source MLLMs using only coarse-grained quality scores, thereby avoiding the need for annotating high-quality interpretable evaluation rationale. Our approach integrates Group Relative Policy Optimization (GRPO) into the instruction-tuning process, enabling models to generate both scalar scores and interpretable reasoning chains with only easy accessible annotated judgment scores or preferences. Furthermore, we introduce a continuous reward formulation that encourages score diversity and provides stable optimization signals, leading to more robust and discriminative evaluation behavior. Experimental results on three established T2I meta-evaluation benchmarks demonstrate that T2I-Eval-R1 achieves significantly higher alignment with human assessments and offers more accurate interpretable score rationales compared to strong baseline methods.

Reinforcement Learning-Driven Framework for Interpretable Text-to-Image Evaluation

This paper introduces T2I-Eval-R1, a novel framework utilizing reinforcement learning (RL) to evaluate text-to-image (T2I) generation models. The motivation stems from the growing need for effective and scalable evaluation methods that can automatically and accurately assess the quality of generated images. Leveraging reinforcement learning, the authors aim to circumvent the heavy reliance on exhaustive human annotations and proprietary LLMs, which often come with scalability issues and notable biases.

Technical Insights

T2I-Eval-R1 pivots away from supervised fine-tuning, traditionally used in multimodal LLMs (MLLMs) for T2I evaluations. Instead, it applies Group Relative Policy Optimization (GRPO), a reinforcement learning approach, to enable training of the evaluation models using only coarse-grained quality scores. This method essentially optimizes models to generate scalar scores and explanatory reasoning chains. The paper outlines several key improvements in this approach:

• Continuous Reward Formulation: To enhance the models' score diversity and optimization signals, continuous rewards are introduced, which provide a more nuanced differentiation between prediction outcomes compared to binary metrics.
• Group Relative Policy Optimization (GRPO): A distinctive feature integrated within the instruction-tuning process, GRPO fosters relative ranking behavior and structured rationales under weak supervision parameters.

These innovations culminate in a framework where the evaluator can produce both qualitative and quantitative output, making judgments about images generated by text-to-image models. The framework is designed to be modular, allowing easy adaptation to varying evaluation dimensions and criteria without requiring retraining, thus adding a layer of flexibility beyond existing methods.

Experimental Results

The experimental results highlight significant improvements over baseline methods, including proprietary models like GPT-4o and VIEScore. Specifically, T2I-Eval-R1 achieves higher alignment with human assessments across several established meta-evaluation benchmarks. For example, Spearman and Kendall correlation coefficients show marked improvements in domains such as appearance quality and intrinsic attribute consistency compared to competitive models.

Additionally, the paper provides evidence that T2I-Eval-R1 excels in areas related to interpretable score rationales. When assessed using human annotators and the GPT-4o model, T2I-Eval-R1's explanations outperformed previous methods in terms of clarity, completeness, and alignment with human judgments.

Implications and Future Work

The implications of T2I-Eval-R1 extend into both practical and theoretical realms. Practically, the deployment of such a framework may significantly reduce the cost and time associated with human annotation processes, ensuring scalable and efficient evaluations across large datasets. Theoretically, it invites further exploration into RL applications in AI, specifically the potential of GRPO and similar methods to replace or augment traditional supervised learning approaches across diverse multimodal tasks.

The robustness and interpretability of the evaluation rationales generated by T2I-Eval-R1 hint at promising future directions where reinforcement learning could yield evaluators capable of adapting and scaling seamlessly to evolving T2I generation technologies. Moreover, the introduction of continuous rewards in RL for semantic tasks opens the door to fine-grained optimization strategies that may enhance model training efficacy across other generative tasks.

Despite its demonstrable success, the paper acknowledges certain limitations, such as potential constraints in assessing evaluation dimensions unseen during training, thus orienting future research toward extending T2I-Eval-R1's generalizability across varied domains. Additionally, while the framework has been tested on MLLMs of specific scales, expanding to larger multimodal architectures might reveal further undiscovered potential in efficiency and evaluative power.

In conclusion, T2I-Eval-R1 presents a significant stride toward the scalable, interpretable evaluation of text-to-image models, leveraging reinforcement learning to offer nuanced insights into model behavior—a development that may well shape future trends in AI-assisted image evaluation.