Improved Image Captioning via Policy Gradient optimization of SPIDEr (1612.00370v4)

Abstract: Current image captioning methods are usually trained via (penalized) maximum likelihood estimation. However, the log-likelihood score of a caption does not correlate well with human assessments of quality. Standard syntactic evaluation metrics, such as BLEU, METEOR and ROUGE, are also not well correlated. The newer SPICE and CIDEr metrics are better correlated, but have traditionally been hard to optimize for. In this paper, we show how to use a policy gradient (PG) method to directly optimize a linear combination of SPICE and CIDEr (a combination we call SPIDEr): the SPICE score ensures our captions are semantically faithful to the image, while CIDEr score ensures our captions are syntactically fluent. The PG method we propose improves on the prior MIXER approach, by using Monte Carlo rollouts instead of mixing MLE training with PG. We show empirically that our algorithm leads to easier optimization and improved results compared to MIXER. Finally, we show that using our PG method we can optimize any of the metrics, including the proposed SPIDEr metric which results in image captions that are strongly preferred by human raters compared to captions generated by the same model but trained to optimize MLE or the COCO metrics.

• The paper introduces a policy gradient optimization method for directly maximizing the novel SPIDEr metric in image captioning.
• The method refines traditional MIXER techniques using Monte Carlo rollouts for improved convergence and higher human preference scores.
• Empirical evaluations show that SPIDEr-optimized models yield more semantically accurate and syntactically fluent captions compared to MLE-trained models.

Overview of Improved Image Captioning via Policy Gradient Optimization of SPIDEr

The paper addresses the significant challenge in image captioning of aligning model-generated captions with human understanding. Traditional methods rely heavily on maximum likelihood estimation (MLE) for training captioning models. However, it is recognized that MLE does not align well with human evaluations, nor do conventional metric evaluations like BLEU, METEOR, and ROUGE. Despite improvements, newer metrics such as SPICE and CIDEr, which better correlate with human judgment, have posed challenges in optimization. The paper introduces a policy gradient (PG) method to directly optimize a novel metric, SPIDEr, that combines SPICE and CIDEr to balance semantic faithfulness and syntactic fluency in captions.

Technical Contributions

The authors propose several notable innovations in this work:

1. Policy Gradient Approach: The paper refines the MIXER approach by incorporating Monte Carlo rollouts, offering a more stable optimization process than mixing MLE training with policy gradient techniques. This leads to improved convergence and performance, particularly for the SPIDEr metric.
2. SPIDEr Metric: The research introduces and validates the SPIDEr metric, a linear combination of SPICE and CIDEr, addressing the need for metrics that reflect both human-like caption quality (semantic fidelity) and syntactic fluency.
3. Empirical Evaluation: The authors demonstrate empirically that optimizing for SPIDEr via their proposed policy gradient method leads to captions that are preferred by humans over those optimized using MLE or traditional metrics, affirming the metric's relevance. Furthermore, they provide evidence of the SPIDEr-based model achieving higher human preference scores than models optimized with COCO metrics.

Implications and Future Directions

The findings have significant implications for both theoretical understanding and practical application in image captioning:

• Metric Design: The SPIDEr metric introduces a useful framework for evaluating and training image captioning models, presenting a balanced approach to metric formulation by integrating semantic and syntactic considerations.
• Optimization Strategies: The policy gradient optimization paradigm extends beyond image captioning, presenting broad applicability to other sequence generation tasks where differentiable metric optimization is desirable.
• Human Evaluation Alignment: By optimizing captions that align more closely with human judgment, this research underscores the importance of incorporating comprehensive metrics that reflect nuanced human preferences into the model training phase.

Looking forward, future research may explore extending this policy gradient-based optimization to more complex metrics and further refining metric synthesis to incorporate dynamic weights or even context-dependent adjustments. Moreover, additional work could investigate robustness across diverse datasets and evaluate how improvements in metric correlation with human judgments translate into progress in real-world applications, particularly those impacting accessibility and human-machine interaction.

This paper marks a step toward more human-aligned image captioning models, a pursuit vital for developing systems that understand and describe the world with human-like insight. The proposed methods and findings serve as a substantial foundation for subsequent exploration in semantic and syntactic optimization in natural language processing tasks.