Caption-Regularized Policy Optimization (CapPO)

• The paper demonstrates that caption regularization improves multimodal reasoning accuracy by aligning image-conditioned and caption-conditioned responses via KL-weighted advantage estimation.
• It introduces a GRPO-based framework that uses captions as semantic anchors during training to reduce perceptual errors, showing significant gains on math benchmarks.
• The method applies adaptive rescaling of reinforcement signals with token-level KL divergence to balance correct and perceptually consistent outputs.

Caption-Regularized Policy Optimization (CapPO) is a reinforcement learning framework for multimodal LLMs (MLLMs) that improves reasoning reliability by explicitly enforcing perceptual consistency during policy optimization. It is introduced as a GRPO-based method in which image-conditioned responses are regularized toward caption-conditioned responses, while the reinforcement signal is adaptively rescaled by an image-versus-caption KL-derived consistency score. The method is motivated by the claim that many multimodal reasoning failures arise from incorrect visual grounding rather than from the reasoning algorithm itself; on Qwen2.5-VL-7B over MathVista, the reported error analysis attributes 51.3% of errors to perceptual failures. CapPO therefore uses captions as semantic anchors during training, but not at inference time, so the method modifies optimization rather than deployment behavior (Tu et al., 26 Sep 2025).

1. Problem formulation and motivation

CapPO is designed for multimodal reasoning over image-question pairs $(i,q)$, in which a model generates a response $o$ under an image-conditioned policy

$\pi_\theta(o \mid i, q).$

The framework also introduces a captioner $C(\cdot)$, which produces a caption

$c = C(i, q),$

thereby defining an alternative caption-conditioned policy

$\pi_\theta(o \mid c, q).$

The image-conditioned branch remains the task policy, while the caption-conditioned branch acts as a semantic reference during training (Tu et al., 26 Sep 2025).

The central motivation is that sparse final-answer rewards can reinforce trajectories that succeed despite flawed perception. The paper argues that standard RL fine-tuning methods such as GRPO can improve correctness and reasoning stability, but do not directly align visual grounding with the reasoning process. CapPO addresses this gap by requiring consistency between reasoning from raw images and reasoning from captions that summarize the same image. The supporting intuition is that high-quality captions provide textual semantic summaries that are often easier for the LLM to reason over than raw pixels. On ViRL39K, when captions generated by strong Qwen2.5-VL captioners are fed to the text-only reasoning model DeepSeek-R1-0528-Qwen3-8B, reasoning accuracy reportedly exceeds 75% with 32B/72B captioners. This suggests that caption-derived semantics can serve as a useful training-time anchor (Tu et al., 26 Sep 2025).

A common misconception is that CapPO is a caption-generation method. The paper does not present CapPO as an image captioning objective. Rather, captions are auxiliary supervision used to stabilize multimodal reasoning. Another common misunderstanding is to treat the method as generic visual consistency training; the formulation is specifically tied to RL fine-tuning over sampled reasoning trajectories and to token-level divergence between image-conditioned and caption-conditioned response distributions (Tu et al., 26 Sep 2025).

2. GRPO foundation and the CapPO objective

CapPO is built on Group Relative Policy Optimization (GRPO). For each prompt $q$, GRPO samples a group of $G$ outputs

$$\{o_1, o_2, \dots, o_G\} \sim \pi_{\theta_{\text{old}}}(\cdot \mid q),$$

computes rewards $r_i$, and forms the group-relative advantage

$o$0

with

$o$1

The GRPO objective used as CapPO’s base is

$o$2

where

$o$3

In CapPO’s final formulation, the authors state that they remove the KL divergence to the reference policy following prior studies (Tu et al., 26 Sep 2025).

The reward function is intentionally simple and rule-based. The model must output a reasoning trace inside `and place the final answer in\boxed{}`. The reward is

$o$4

Because this reward is sparse and final-answer-oriented, CapPO adds perceptual consistency signals rather than modifying the verifier itself (Tu et al., 26 Sep 2025).

The first additional term is caption-based consistency regularization. For sampled output $o$5, the loss is

$o$6

The discrepancy is computed at the token-distribution level. The paper states that in practice it uses an unbiased estimator of KL divergence that is always non-negative: $o$7

The full CapPO objective combines GRPO with KL-weighted advantages and caption consistency regularization: $o$8 Here $o$9 controls caption-regularization strength, and $\pi_\theta(o \mid i, q).$0 is shorthand for $\pi_\theta(o \mid i, q).$1 (Tu et al., 26 Sep 2025).

3. KL-weighted advantage estimation and training procedure

The second defining component of CapPO is KL-weighted advantage estimation. For a sampled response $\pi_\theta(o \mid i, q).$2, token-level KL terms are aggregated into a sequence-level perceptual inconsistency score

$\pi_\theta(o \mid i, q).$3

The paper then applies group-wise z-score normalization over $\pi_\theta(o \mid i, q).$4 within each GRPO group, but does not give an explicit formula for that normalization (Tu et al., 26 Sep 2025).

The weighting function is sign-aware: $\pi_\theta(o \mid i, q).$5 and the modified advantage is

$\pi_\theta(o \mid i, q).$6

If $\pi_\theta(o \mid i, q).$7 and KL is large, the positive reinforcement is reduced. If $\pi_\theta(o \mid i, q).$8 and KL is large, the penalty is amplified up to clipping. The paper explicitly notes that gradients are detached from $\pi_\theta(o \mid i, q).$9 while flowing through $C(\cdot)$0. This mechanism is intended to suppress spurious successful trajectories and strengthen pressure against trajectories that are both poor and perceptually inconsistent (Tu et al., 26 Sep 2025).

The training pipeline is a plug-in extension to GRPO. The RL dataset is first constructed from image-question pairs, then captions $C(\cdot)$1 are generated offline by an external large MLLM and stored with each sample. During each training iteration, the algorithm samples a triplet $C(\cdot)$2, generates a group of $C(\cdot)$3 candidate responses from the old image-conditioned policy, computes verifier rewards, forms standard group-relative advantages, evaluates the caption-conditioned distribution for each sampled response, computes token-level image-versus-caption KL, aggregates to sequence-level KL, normalizes the KL within the group, computes $C(\cdot)$4, forms $C(\cdot)$5, and optimizes $C(\cdot)$6. Captions are therefore used only as auxiliary supervision during training, not at inference time (Tu et al., 26 Sep 2025).

The captioning pipeline is itself part of the method definition. Captions are generated offline with structured prompts, using image and question as input and producing detailed textual descriptions intended to preserve problem-relevant semantics. In the main experiments, the captioner is Qwen2.5-VL-72B, and after filtering failures or formatting issues the final RL dataset contains 36,579 samples. This suggests that CapPO treats caption quality as an upstream determinant of RL signal quality rather than as a peripheral preprocessing choice (Tu et al., 26 Sep 2025).

4. Experimental setting and empirical results

CapPO is evaluated with Qwen2.5-VL-7B as the base policy model and also ablated on Qwen2.5-VL-3B. Training uses the EasyR1 framework and is described as RL only, with no extra SFT stage in the proposed method. The training data are drawn from ViRL39K, a multimodal reasoning dataset with about 38,870 questions, roughly 80% math-focused and roughly 20% general reasoning; after caption generation and filtering, the final cleaned training set contains about 36,579 valid samples. Evaluation covers five math-related benchmarks—MathVista, MathVerse, MathVision, WeMath, and DynaMath—and five general reasoning benchmarks—LogicVista, ScienceQA, MMStar, MMMU, and MMMU-Pro. The appendix states that evaluation uses VLMEvalKit with greedy decoding under bf16 precision (Tu et al., 26 Sep 2025).

The main reported result is that CapPO improves over the base Qwen2.5-VL-7B by +6.0 points average accuracy on math-related tasks and +2.4 points on general reasoning tasks. Benchmark-wise, the paper reports MathVista 69.2 $C(\cdot)$7 73.6, MathVerse 44.7 $C(\cdot)$8 49.4, MathVision 21.7 $C(\cdot)$9 30.3, WeMath 34.8 $c = C(i, q),$0 42.6, DynaMath 53.7 $c = C(i, q),$1 57.9, yielding a math average of 44.8 $c = C(i, q),$2 50.8. On general reasoning, it reports LogicVista 46.9 $c = C(i, q),$3 49.4, ScienceQA 88.8 $c = C(i, q),$4 91.0, MMStar 64.6 $c = C(i, q),$5 64.8, MMMU 56.3 $c = C(i, q),$6 60.0, and MMMU-Pro 40.8 $c = C(i, q),$7 44.1, yielding an average of 59.5 $c = C(i, q),$8 61.9. The method is said to achieve the best average across both benchmark categories, with particularly strong gains on perception-intensive math benchmarks (Tu et al., 26 Sep 2025).

The component ablations identify separate contributions from both CapPO modules. On Qwen2.5-VL-7B, the reported overall average is Base 52.2, +GRPO 54.7, +CapPO w/o KL-Reg 55.0, +CapPO w/o Adv-Weight 55.3, and +CapPO 56.4. On Qwen2.5-VL-3B, the corresponding sequence is 44.5, 47.0, 47.8, 47.3, and 48.7. The paper interprets these results as showing that RL alone helps, each CapPO module helps independently, and the full combination performs best (Tu et al., 26 Sep 2025).

Caption quality is also varied explicitly. Relative to the GRPO baseline average of 54.7, the paper reports 55.6 with a 3B captioner, 56.0 with a 7B self-captioner, and 56.4 with the 72B captioner. Even weaker or self-generated captions therefore improve over GRPO. Dataset scaling experiments further report that even with only 2% of the training data, corresponding to 732 samples, CapPO still outperforms the base model. These findings support the claim that the method is useful beyond the full-data, strongest-captioner regime, although the paper also notes dependence on caption quality as a limitation (Tu et al., 26 Sep 2025).

5. Error profile, implementation characteristics, and limitations

The paper’s error analysis directly targets the mechanism CapPO is intended to improve. On MathVista test_mini, the base model has 308 errors out of 1000, with 51.3% perception errors; GRPO has 299 total errors, but the perception-error share remains nearly unchanged; CapPO has 264 total errors, with the perception-error share reduced to 45.8%. The paper describes this as a 5.4 percentage point reduction in perception-error share compared to GRPO. Qualitative case studies further report that, relative to GRPO, CapPO better uses geometric constraints such as parallel lines, focuses on relevant symbols and numeric tokens, performs more complete object counting, and avoids heuristic guessing in open-domain visual tasks. The accompanying attention visualizations are presented as qualitative observations rather than formal proofs (Tu et al., 26 Sep 2025).

Training dynamics are also discussed. CapPO is reported to achieve the highest training reward and the most stable reward trajectory. Response length under the full method increases initially and then stabilizes; without KL regularization, response length grows monotonically; without advantage weighting, response length gradually decreases. The authors interpret this as evidence that both the caption-KL term and the advantage-weighting mechanism are needed for stable reasoning optimization. Hyperparameter sensitivity studies identify $c = C(i, q),$9 and $\pi_\theta(o \mid c, q).$0 as the best settings found, while excessively large values such as $\pi_\theta(o \mid c, q).$1 degrade performance because too much regularization over-constrains policy learning (Tu et al., 26 Sep 2025).

The implementation details are unusually explicit. The appendix reports a max prompt length of 4096, max response length of 8192, min image pixels of 200,704, max image pixels of 1,003,520, rollout batch size 512, training batch size 128, rollout sampling number or group size 8, rollout temperature 1.0, clip coefficient $\pi_\theta(o \mid c, q).$2, learning rate $\pi_\theta(o \mid c, q).$3, caption-KL coefficient $\pi_\theta(o \mid c, q).$4, advantage-KL coefficient $\pi_\theta(o \mid c, q).$5, weight bounds $\pi_\theta(o \mid c, q).$6 and $\pi_\theta(o \mid c, q).$7, number of epochs 2, and freeze vision encoder set to False. Evaluation uses greedy decoding and bf16 precision via VLMEvalKit (Tu et al., 26 Sep 2025).

The computational overhead is described as modest relative to GRPO. On 8 $\pi_\theta(o \mid c, q).$8 H100, GRPO requires about 11 hours and CapPO about 14 hours. Per-step, GRPO is reported as generation $\pi_\theta(o \mid c, q).$9s, update actor $q$0s, each step $q$1s, whereas CapPO uses generation $q$2s, update actor $q$3s, caption-logit $q$4s, and each step $q$5s. The method is therefore slightly slower because of extra caption-conditioned logit computation and somewhat longer responses, but remains within the same order of magnitude (Tu et al., 26 Sep 2025).

The limitations stated in the paper are specific. Manual hyperparameter tuning is still required for $q$6 and $q$7; the method depends on caption quality, though the ablations show it is not extremely fragile; it requires an additional captioning pipeline during training, typically an external MLLM; it is developed in a zero-shot RL setting, with extension to hybrid SFT+RL left as future work; it introduces some computational overhead from caption-based logits; and it is evaluated only within its reported benchmark scope, so generalization to other multimodal domains is not fully established (Tu et al., 26 Sep 2025).

6. Relation to adjacent policy-optimization methods

CapPO is explicitly positioned as a minimal modification of GRPO/PPO-style RL fine-tuning rather than as a new architecture. It retains group sampling, reward normalization, clipped surrogate optimization, and importance ratios, while adding a caption-conditioned KL regularizer and replacing the standard advantage $q$8 with the KL-weighted advantage $q$9. The paper also states that it does not formulate the method as a DPO-style preference optimization method and does not present direct DPO comparisons (Tu et al., 26 Sep 2025).

Within the broader literature on regularized policy optimization, CapPO is distinguished by what it regularizes. Ratio-Variance Regularized Policy Optimization regularizes the variance of the policy ratio as a local trust-region surrogate and is intended for on-policy RL, stale data reuse, LLM RL fine-tuning, and robotics; its stabilizer is purely distributional rather than caption-derived (Luo et al., 26 May 2026). Muesli combines a direct policy-gradient term with KL regularization to a clipped MPO-style target policy built from prior policy and model-based value estimates; it contains no caption or language-derived regularization (Hessel et al., 2021). Score Regularized Policy Optimization through Diffusion Behavior uses the score $G$0 of a pretrained diffusion behavior model as a gradient-level regularizer in offline RL; again, the regularizer is a behavior prior rather than a semantic caption anchor (Chen et al., 2023).

Among multimodal and caption-centered RL methods, BalCapRL is closer in topic but not in task definition. BalCapRL is an RL framework for MLLM image captioning that balances utility-aware correctness, reference coverage, linguistic quality, reward-decoupled normalization, and length-conditional reward masking. Its regularization is directed at caption quality itself, whereas CapPO uses captions to regularize multimodal reasoning trajectories (Ye et al., 8 May 2026). A separate source of confusion is acronymic: “CAPO” in “Consensus Aggregation for Policy Optimization” denotes a compute reallocation strategy for PPO based on aggregating multiple replicas on the same on-policy batch, not Caption-Regularized Policy Optimization (Su et al., 13 Mar 2026).

A plausible implication is that CapPO should be read as one instance of a broader design pattern: policy optimization can be stabilized not only by trust-region proxies, KL-to-target penalties, replay-aware regularizers, or behavior priors, but also by auxiliary semantic references that reshape the effective reinforcement signal. What is specific to CapPO is that the auxiliary reference is a caption-conditioned response distribution aligned to the same image-question pair, and that this reference influences both the policy objective and the per-trajectory advantage weighting (Tu et al., 26 Sep 2025).