Captioning Reinforcement Learning (CapRL)

• CapRL is a reinforcement learning framework for caption generation that optimizes sequence-level rewards like CIDEr and SPIDEr to reduce exposure bias.
• It employs methodologies such as self-critical training, actor-critic architectures, and GRPO variants to refine visual-to-text outputs.
• CapRL systems leverage diverse reward designs—from metric-based and semantic signals to human and verifiable feedback—to enhance discriminativeness, factual grounding, and completeness.

Captioning Reinforcement Learning (CapRL) denotes the application of reinforcement learning to caption generation so that a model is optimized against sequence-level objectives rather than only teacher-forced cross-entropy. In this literature, the reward can be a caption metric such as CIDEr or SPIDEr, a semantic or retrieval score, a human- or preference-derived signal, or a verifiable notion of utility such as question-answering accuracy from the caption alone. The common motivation is to reduce exposure bias, address the mismatch between token-level likelihood training and sequence-level evaluation, and push captioners toward properties that ordinary reference imitation does not guarantee, including discriminativeness, factual grounding, completeness, and downstream usefulness (Zhang et al., 2017, Phan et al., 2017, Seo et al., 2019, Xing et al., 26 Sep 2025).

1. Terminology, scope, and historical development

The term “CapRL” is used in two related senses. In the broad sense, it refers to reinforcement-learning-based captioning across image, video, and audio modalities. In the narrower sense, later work uses “CapRL” as the name of a specific Reinforcement Learning with Verifiable Rewards framework for dense image captioning, later generalized to image and video captioning in CapRL++ (Xing et al., 26 Sep 2025, Yang et al., 8 Jun 2026). Earlier foundational papers did not always use the exact term: the actor-critic image-captioning work states that it does not specifically coin “CapRL,” and the survey literature similarly discusses reinforcement learning for captioning without presenting “CapRL” as a standard name (Zhang et al., 2017, Hossain et al., 2018).

Historically, early captioning RL focused on image captioning with policy-gradient or actor-critic formulations that optimized non-differentiable caption metrics directly, replacing or refining maximum-likelihood training (Zhang et al., 2017, Shi et al., 2018). Closely related work then introduced self-critical and consensus-based baselines for variance reduction and faster optimization in video captioning, and multi-stage coarse-to-fine refiners used reinforcement learning to normalize each stage against earlier decoders and test-time inference outputs (Phan et al., 2017, Gu et al., 2017). Later developments broadened the field along three axes: richer architectures such as reconstruction networks and self-correction loops, broader modalities including audio and video, and more structured rewards that moved beyond metric overlap toward semantic alignment, human judgment, and verifiable utility (Zhang et al., 2019, Mei et al., 2021, Meng et al., 2 Jun 2025).

A concise way to organize the field is by reward source.

Regime	Representative papers	Reward basis
Metric optimization	(Phan et al., 2017, Zhang et al., 2019, Mei et al., 2021)	CIDEr, CIDEr_r, SPIDEr, BLEU, METEOR, ROUGE-L
Semantic or retrieval reward	(Shi et al., 2018, Chaffin et al., 2024, Dzabraev et al., 2024)	embedding similarity, CLIP, BLIP2-ITM
Human or preference reward	(Seo et al., 2019, Hegde et al., 18 Sep 2025)	offline human ratings, pairwise preferences
Verifiable or balanced reward	(Xing et al., 26 Sep 2025, Yang et al., 8 Jun 2026, Ye et al., 8 May 2026, Tang et al., 25 Feb 2026, Huang et al., 10 Mar 2026)	MCQ utility, balanced quality, completeness/correctness, rubric criteria

2. Formal problem and optimization paradigms

Most CapRL systems adopt the standard autoregressive factorization

$$p_\theta(y \mid x)=\prod_{t=1}^{T} p_\theta(y_t \mid y_{<t},x),$$

and replace pure cross-entropy minimization with maximization of an expected sequence reward

$$J(\theta)=\mathbb{E}_{y\sim p_\theta(\cdot\mid x)}[r(y)].$$

The classical REINFORCE form uses a baseline $b$ to reduce variance, so that updates depend on $(r(y)-b)\nabla_\theta \log p_\theta(y\mid x)$ rather than raw reward alone. This basic form appears in metric-optimized video captioning, in actor-critic image captioning, in audio captioning with SCST, and in later human-feedback formulations (Phan et al., 2017, Zhang et al., 2017, Mei et al., 2021, Seo et al., 2019).

A major branch of CapRL uses self-critical sequence training. In SCST, the baseline is the reward of the greedily decoded caption from the current policy. RecNet states the policy gradient in the form

$$\nabla_\theta L_{RL}(\theta)\approx -(r(S)-r(\hat S))\sum_t \nabla_\theta \log \pi_\theta(s_t),$$

while the audio-captioning system applies the same self-critical baseline to optimize CIDEr$_r$ after cross-entropy pretraining (Zhang et al., 2019, Mei et al., 2021). Consensus-based Sequence Training modifies this pattern by replacing the greedy baseline with a self-consensus baseline computed from the multiple human captions of the same video, and also shows that plugging a ground-truth caption into the REINFORCE derivative yields a weighted cross-entropy objective, which it uses for fast RL-based pretraining (Phan et al., 2017).

A second branch uses actor-critic structure. “Actor-Critic Sequence Training for Image Captioning” defines a per-token advantage

$A_t = Q^\pi(s_t,a_{t+1}) - V^\pi(s_t),$

and learns a critic $V(s_t)$ so that the actor receives token-dependent credit assignment rather than a single sentence-level advantage (Zhang et al., 2017). The earlier image-captioning actor/value-network formulation similarly couples a policy network that proposes next words with a value network that evaluates expected reward in a semantic embedding space (Shi et al., 2018).

More recent large-model captioning systems often move to GRPO-style optimization. VideoCap-R1 normalizes each rollout by the mean and standard deviation of rewards inside a sampled group,

$$\hat A_i=\frac{r_i-\mathrm{mean}(\{r_j\})}{\mathrm{std}(\{r_j\})},$$

and optimizes a clipped surrogate with KL regularization to a reference policy (Meng et al., 2 Jun 2025). BalCapRL argues that vanilla GRPO collapses distinct continuous rewards when they are summed before normalization, and therefore introduces c-GDPO, which normalizes precision, recall, and linguistic rewards separately before aggregation; it then applies a clipped token-level objective with the final batch-normalized advantage (Ye et al., 8 May 2026). This suggests a methodological shift from single-scalar reward optimization toward explicitly multi-objective policy optimization.

3. Architectural patterns in CapRL systems

Although the simplest CapRL systems place RL on top of a standard encoder-decoder, the literature repeatedly uses RL to support architectures whose internal structure is meant to improve semantic grounding. RecNet is exemplary: it augments an encoder-decoder with a reconstructor so that captioning uses a forward flow from video to sentence and a backward flow from sentence to video. A global reconstructor matches mean-pooled video representations, a local reconstructor aligns frame-wise features by attending over decoder states, and the total supervised or RL objective retains the reconstruction term during CIDEr optimization (Zhang et al., 2019).

Stack-Captioning uses a different structural idea: a coarse-to-fine stack of decoders, typically three stages, in which stage 1 produces a coarse caption and later stages refine it. Its reinforcement-learning formulation normalizes each stage not only against its own test-time inference output but also against the preceding stage, thereby training later decoders to outperform earlier ones and reducing the vanishing-gradient difficulties of multi-stage captioners (Gu et al., 2017). SC-Captioner extends the architectural scope further by introducing a two-turn self-correction process: the same VLM first emits an initial caption and then, conditioned on that caption plus a refinement instruction, generates a self-corrected caption. The RL loss is applied to the second pass and regularized against a frozen reference policy, while the reward depends only on semantic edits between the first and second outputs (Zhang et al., 8 Aug 2025).

CapRL also supports explicitly reason-then-describe pipelines. VideoCap-R1 prompts a video MLLM to generate a “Thinking” section that enumerates subjects, attributes, and actions before emitting the final caption, and uses specialized rewards for both stages. Its GRPO training is designed to connect the quality of structured reasoning to the factual accuracy of open-ended video descriptions (Meng et al., 2 Jun 2025). In a different direction, CLIP-guided dense captioning with ground-truth support uses OFA-tiny as the generator but supplements RL with three ground-truth roles: a discriminator over CLIP text embeddings for human-likeness regularization, reward-weighted teacher forcing on ground-truth captions, and strong textual baselines obtained by inserting ground-truth captions into the contrastive negative pool (Chaffin et al., 2024).

Not all architectural interventions occur inside the RL loop itself. The discriminativeness work on RL-trained captioners freezes the encoder and fine-tunes only the decoder classifier for one epoch, either with plain cross-entropy or with a bias-product weighting against a frozen RL-biased model. This is not a new RL algorithm, but it is an important post-RL architectural recalibration: the intervention targets the final classifier because the paper identifies RL’s main bottleneck as a high-frequency-word bias in the actual emitted vocabulary (Honda et al., 2022).

4. Reward design and the semantics of caption quality

Reward design is the central differentiator of CapRL systems. Metric-based rewards remain foundational. Consensus-based Sequence Training optimizes CIDEr and derives weighted cross-entropy from CIDEr-scored references; RecNet fine-tunes by CIDEr optimization via SCST; and the audio-captioning system uses CIDEr$_r$ during SCST fine-tuning after selecting the best cross-entropy checkpoint by SPIDEr on the development set (Phan et al., 2017, Zhang et al., 2019, Mei et al., 2021). These systems preserve the classical view that caption quality can be proxied by overlap-driven evaluation metrics, albeit optimized at sequence level rather than through token likelihood alone.

Other systems replace reference overlap with semantic or retrieval signals. The policy/value-network image-captioning method defines a reward from image-caption alignment in a learned semantic embedding space rather than directly from CIDEr or BLEU (Shi et al., 2018). CLIP-guided dense captioning defines a bidirectional decoupled contrastive reward over text and image pools, adds a discriminator score as a regularizer, and mixes both through

$$J(\theta)=\mathbb{E}_{y\sim p_\theta(\cdot\mid x)}[r(y)].$$0

with $$J(\theta)=\mathbb{E}_{y\sim p_\theta(\cdot\mid x)}[r(y)].$$1 (Chaffin et al., 2024). VLRM uses BLIP2-ITM logits, optional CLIP similarity, a reference-language-model naturalness term, and hand-engineered penalties for bad phrases, repetition, and missing EOS, then trains BLIP2-OPT-2.7B with an advantage-based policy gradient and a learned value head on unlabeled images (Dzabraev et al., 2024). These formulations treat captioning as a reward-modeling problem in which good captions are those that are semantically aligned, retrieval-useful, and linguistically acceptable even without paired references.

A third family uses human feedback directly. “Reinforcing an Image Caption Generator Using Off-Line Human Feedback” optimizes expected human ratings with off-policy importance weighting over a static ratings dataset and stabilizes the objective with an MLE regularizer on a larger caption corpus; it reports that only the off-policy formulation with true ratings yields statistically significant gains in human evaluation (Seo et al., 2019). In audio captioning, preference alignment is implemented through a CLAP-based reward model trained on pairwise preferences with a Bradley–Terry/logistic loss and then used inside SCST-style RLHF, together with a length penalty to mitigate reward hacking (Hegde et al., 18 Sep 2025).

The most recent shift is toward verifiable or structured rewards. The 2025 CapRL framework defines caption quality by the accuracy of a vision-free LLM answering MCQs about the image given only the caption, with options shuffled each time and group-normalized GRPO advantages during training (Xing et al., 26 Sep 2025). CapRL++ generalizes the same idea to image and video captioning, adds a temporal-format reward for timestamp validity and chronology, and uses a length-aware regularizer to suppress verbosity-based reward hacking (Yang et al., 8 Jun 2026). CCCaption splits the objective into completeness and correctness rewards, where completeness is measured by how many image-derived queries the caption answers and correctness is measured by validating sub-caption queries against the image; the combined reward is

$$J(\theta)=\mathbb{E}_{y\sim p_\theta(\cdot\mid x)}[r(y)].$$2

with $$J(\theta)=\mathbb{E}_{y\sim p_\theta(\cdot\mid x)}[r(y)].$$3 in experiments (Tang et al., 25 Feb 2026). RubiCap replaces scalar judge scores with sample-specific LLM-written rubrics whose binary criteria are weighted by severity and aggregated into a normalized reward (Huang et al., 10 Mar 2026). BalCapRL similarly decomposes caption quality into utility-aware correctness, reference coverage, and linguistic quality, then uses reward-decoupled normalization and length-conditional masking to preserve balance across these objectives (Ye et al., 8 May 2026). This suggests a broader redefinition of caption quality: from overlap with a reference sentence to a structured combination of factuality, coverage, usefulness, and human acceptability.

5. Empirical gains, trade-offs, and recurrent failure modes

CapRL repeatedly improves standard caption benchmarks when reward design and variance reduction are well matched to the task. On MSRVTT, Consensus-based Sequence Training raises CIDEr from 47.3 for the XE baseline to 49.7 with weighted cross-entropy pretraining, 53.6 with CST$$J(\theta)=\mathbb{E}_{y\sim p_\theta(\cdot\mid x)}[r(y)].$$4, and 54.2 for the variant whose baseline is computed from per-batch samples; it also reduces per-batch wall-clock from 10.377 s for the Greedy baseline to 5.256 s for the self-consensus baseline (Phan et al., 2017). On video captioning, RecNet with SCST and fused global-plus-local reconstruction reaches 49.5 CIDEr and 27.7 METEOR on MSR-VTT, 85.9 CIDEr on MSVD, and 38.88 on ActivityNet validation, consistently outperforming its SA-LSTM baseline (Zhang et al., 2019).

CapRL can also deliver strong gains outside reference-heavy evaluation. The audio-captioning system augmented with PANNs, AudioCaps pretraining, and RL reaches SPIDEr 0.305 on the Clotho v2 evaluation set, compared with 0.226 for the baseline trained from scratch, and it ranked 3rd in DCASE 2021 Task 6 while being the top single model without ensembling (Mei et al., 2021). The discriminativeness study shows that simple one-epoch post-RL classifier recalibration materially changes retrieval behavior: for a Transformer captioner on MS COCO, R@1/R@5/R@10 rises from 26.6/56.2/70.5 for the RL baseline to 33.5/65.9/78.2 with weighted fine-tuning plus bias-product decoding, while Unique-1 increases from 753 to 1,964 and Unique-S from 3,433 to 4,373 (Honda et al., 2022). Recent RLVR formulations report similarly large utility gains: CapRL-3B reaches an average Prism score of 48.3, matching Qwen2.5-VL-72B and exceeding its own 3B baseline by 8.4 points, while CapRL++ lifts a Qwen3-VL-4B captioner to a 47.5 video Prism average, above Qwen3-VL-32B at 46.9 (Xing et al., 26 Sep 2025, Yang et al., 8 Jun 2026).

At the same time, the field is marked by systematic reward pathologies. Audio captioning provides an explicit cautionary example: optimizing CIDEr$$J(\theta)=\mathbb{E}_{y\sim p_\theta(\cdot\mid x)}[r(y)].$$5 improves automatic metrics across BLEU, ROUGE-L, METEOR, SPICE, and SPIDEr, yet the authors show more repetitions, syntactic issues, and incomplete sentences after RL; the phrase “in the background” appears in 155 clips without RL but in 765 clips after RL, even though the ground-truth captions contain it for only 302 clips (Mei et al., 2021). In image captioning, the natural-language-prior paper observes that 46.44% of captions from a strong SCST baseline end with trivial phrases such as “with a,” “on a,” or “of a,” and reports that adding a hard n-gram prior reduces the bad-ending rate to 0.0 while also improving human preference, with 400 judgments favoring the 4-gram model against 251 favoring SCST and 349 ties (Guo et al., 2018).

A different pathology is vocabulary collapse toward frequent words. The discriminativeness study shows that RL-trained policies concentrate their sampled outputs in the highest-frequency word bins, generate fewer unique unigrams and sentences than their cross-entropy counterparts, and thereby lose discriminative detail; its proposed switch effectively treats this as a classifier-calibration problem rather than a new reward problem (Honda et al., 2022). Judge-based rewards introduce yet another failure mode. The off-line human-feedback paper reports that an on-policy method with a learned rating estimator can collapse when the estimator assigns high scores to ill-formed outputs, whereas the off-policy method with true ratings does not (Seo et al., 2019). RubiCap and the 2025 CapRL paper document related reward hacking in single-score judge systems: Reference-Likert can degenerate into self-praise, and LVLM-as-a-judge rewards can over-prefer very short or very verbose captions (Huang et al., 10 Mar 2026, Xing et al., 26 Sep 2025). These recurrent observations explain why many recent CapRL systems introduce explicit anti-hallucination criteria, length control, or decomposed rubric and utility rewards.

6. Dense captioning, pretraining, and current research directions

The current frontier of CapRL is no longer confined to benchmark optimization on a fixed caption dataset. Several recent systems use RL-trained captioners as scalable annotators for large multimodal corpora. CapRL-3B is used to annotate 5M images, creating CapRL-5M, and pretraining on those captions improves Qwen2.5-3B + Qwen2.5-ViT from a 55.5 average on 12 benchmarks in the vanilla setting to 62.0 with CapRL-5M; comparable gains also appear for Qwen2.5-7B and InternLM2.5-7B configurations (Xing et al., 26 Sep 2025). CapRL++ extends the same logic to both images and videos, generating CapRL-Image-5M and CapRL-Video-178K and showing strong continual-pretraining gains across more than 20 image and video benchmarks, with especially large improvements on temporal grounding benchmarks such as ActivityNet, Charades-STA, and QVHighlights (Yang et al., 8 Jun 2026). RubiCap likewise reports that replacing GPT-4V captions with RubiCap-annotated captions in a large pretraining recipe raises the downstream benchmark average from 41.75 to as high as 43.18 (Huang et al., 10 Mar 2026).

Research attention has therefore shifted from whether RL can improve caption metrics to how caption quality should be formalized. BalCapRL argues that utility-only objectives can yield noisy, hallucinated, or overlong captions, whereas arena-style objectives can produce fluent but generic descriptions, and it reports peak gains of +13.6 DCScore, +9.0 CaptionQA, and +29.0 CapArena across different base models when reward-decoupled normalization and length-conditional masking are used (Ye et al., 8 May 2026). CCCaption frames the same tension as completeness versus correctness and emphasizes correctness by setting $$J(\theta)=\mathbb{E}_{y\sim p_\theta(\cdot\mid x)}[r(y)].$$6 in its dual reward, obtaining a 52.80 Prism average and 46.39 CapArena with a 2B policy (Tang et al., 25 Feb 2026). VideoCap-R1 adds structured thinking and reports that a one-epoch GRPO post-training run on only 1.5k curated videos yields +4.4 event F1 on DREAM1K, +4.2 Acc on VDC, and +3.1 action F1 on CAREBENCH over Qwen2-VL-7B (Meng et al., 2 Jun 2025). SC-Captioner shows that even within image captioning, a second-pass self-correction reward on scene-graph edits can outperform direct preference optimization across object, attribute, and relation metrics on DOCCI500 and COCO-LN500 (Zhang et al., 8 Aug 2025).

Open problems recur across modalities. Several papers explicitly note computational overhead from reconstructors, reward models, or judge pipelines; sensitivity to reward design and judge bias; precision-recall or fluency-utility trade-offs; and limited temporal scale in video settings (Zhang et al., 2019, Hegde et al., 18 Sep 2025, Meng et al., 2 Jun 2025, Yang et al., 8 Jun 2026). Future directions named in the literature include multimodal feature expansion, actor-critic or entropy-regularized variants, KL-regularized RLHF, more robust reward modeling, better handling of long videos, and richer human- or preference-aligned objectives (Mei et al., 2021, Hegde et al., 18 Sep 2025, Yang et al., 8 Jun 2026). A plausible implication is that “CapRL” now names less a single algorithm than a design space: one in which caption generation is treated as policy learning, and the decisive research question becomes how to construct rewards that are informative, hard to exploit, and aligned with the desired semantics of a caption.