CFPO: Counterfactual Policy Optimization for Multimodal Reasoning

Abstract: Large Vision-LLMs (LVLMs) have demonstrated remarkable capabilities in multimodal reasoning. However, prevailing reinforcement learning (RL) paradigms lack explicit counterfactual enhancement and causal learning mechanisms. This fundamental deficiency results in severe grounding failures, manifesting as a tendency to ignore visual evidence in favor of language priors or exhibiting hallucination drift during long chain-of-thought reasoning. To address this root cause, we propose CounterFactual Policy Optimization (CFPO), a novel framework that enforces causal consistency between visual perception and textual reasoning. CFPO introduces a cross-modal counterfactual enhancement mechanism, which regularizes the policy by maximizing the discrepancy between the model's predictions and those from a counterfactual state where critical visual cues are suppressed. This approach seamlessly integrates with standard algorithms like GRPO and DAPO without requiring external reward models or additional supervision. Extensive experiments demonstrate that CFPO significantly improves reasoning fidelity, achieving consistent gains of 3.17%-6.25% over standard RL baselines and 1.32%-2.13% over the state-of-the-art perception-aware method (PAPO). Code is available at https://github.com/Raven-July/CFPO.

• The paper introduces CFPO, a novel RL framework that enforces causal consistency in LVLMs through latent counterfactual interventions on high-saliency attention outputs.
• It integrates counterfactual regularization with GRPO and DAPO, demonstrating statistically significant gains (up to 6.25%) on multimodal and mathematical reasoning benchmarks.
• This approach mitigates issues like saliency deficiency and misalignment by forcing models to rely on causally relevant visual evidence, thereby enhancing robust and evidence-aligned reasoning.

Counterfactual Policy Optimization for Multimodal Reasoning

Introduction and Problem Formulation

Counterfactual Policy Optimization (CFPO) introduces a causally-grounded reinforcement learning (RL) framework for Large Vision-LLMs (LVLMs), focusing on cross-modal causal consistency during multimodal reasoning. Standard RL approaches such as GRPO and DAPO, which drive state-of-the-art LVLMs, optimize for correct output but lack explicit mechanisms to distinguish causally grounded reasoning from responses driven purely by statistical or linguistic priors. This results in persistent issues such as saliency deficiency (over-reliance on language priors and hallucination), saliency misalignment (attention to irrelevant but perceptually salient cues), and saliency inertia (failure to update inference according to hypothetical interventions). CFPO explicitly operationalizes cross-modal causal interventions at the policy level and rigorously tests the causal necessity of visual evidence during reasoning, circumventing both input-level randomization and manually curated counterfactual datasets.

Figure 1: Overview of CFPO, introducing a counterfactual path (red) produced by targeted masking of high-saliency attention outputs at the representation level, with divergence enforced between factual and counterfactual predictions.

Methodology

CFPO is instantiated via latent-space interventions, specifically within the attention output layer of the LVLM. The procedure leverages a causal abstraction of the attention mechanism, decomposing the attention matrix into high- and low-saliency regions. High-saliency blocks, identified via outlier detection on cross-modal attention maps, are replaced with mean feature priors while leaving low-saliency contributions unaltered. This generates a counterfactual representation $Z_{cf}$, contrasting with the unaltered, factual representation $Z$. The policy is then regularized by maximizing the divergence (KL) between response distributions derived from $Z$ and $Z_{cf}$, directly penalizing predictions insensitive to the intervention.

This intervention is integrated with GRPO and DAPO frameworks, augmenting their policy gradient objective with a counterfactual regularization term. The KL divergence between the factual and counterfactual policy distributions operationalizes the effective causal contribution of visual evidence. Stability is further ensured via entropy regularization in unconstrained setups (DAPO).

Empirical Results

Experiments demonstrate consistent and statistically significant improvements over both standard (GRPO, DAPO) and state-of-the-art perception-aware (PAPO) baselines across multiple multimodal and mathematical reasoning benchmarks. CFPO achieves relative gains of 3.17–6.25% over RL baselines and 1.32–2.13% over PAPO, particularly excelling on tasks with strong requirements for precise cross-modal grounding such as mathematical visual reasoning and complex chain-of-thought question answering.

Figure 2: Smoothed training reward curves showing that CFPO converges faster and achieves higher sample efficiency and reward than both GRPO and PAPO baselines.

Qualitative failure mode analysis reveals (Figure 3) that, compared to baselines which either hallucinate, fixate on distractors, or remain inert to causal interventions, CFPO systematically corrects such failure cases by forcing reasoning to attend to, and update with, causally relevant visual evidence. Saliency deficiency is circumvented (baselines hallucinate geometry, CFPO grounds on complementary angle relationships), inertia deficiency is corrected (CFPO factors out irrelevant visual context under hypothetical instructions), and misalignment as well as inertia collapse are mitigated through explicit causal regularization.

Figure 3: Comparison of reasoning failure modes between baselines and CFPO, with CFPO resolving saliency deficiency, inertia deficiency, misalignment, and inertia.

A detailed ablation (Tables 1, 2 in the appendix) demonstrates that the statistical outlier-based masking with $\lambda=2$ and mean visual prior imputation is critical, as noise or text-average value injection disrupts latent structure and lowers gains. The counterfactual coefficient $\gamma$ and the presence of entropy are essential for stability; over-regularization without entropy in DAPO triggers mode collapse, while a tuned balance enables optimal causal grounding and reward maximization.

Figure 4: Excessive counterfactual regularization ($\gamma=0.02$) in unconstrained DAPO settings impedes main task optimization, while milder constraint ($\gamma=0.01$) maintains stable convergence and optimal reward balance.

Theoretical and Practical Implications

Theoretically, CFPO advances causal RL for LVLMs by directly verifying and enforcing the necessity of attended multimodal information at the representation (not input or data) level, connecting deep causal abstraction in policy optimization to practical sample-efficient, robust, and interpretable reasoning. By aligning the latent intervention mechanism with structural causal modeling assumptions, CFPO formalizes a rigorous test and training protocol for causal faithfulness in high-capacity cross-modal networks—moving beyond probabilistic associative training towards necessity-aware control of cross-modal credit assignment.

Practically, the ability to reliably regularize against shortcut learning is essential for safety, hallucination mitigation, and robust deployment in settings demanding verifiable evidence alignment. The framework is generalizable to new architectures (e.g., Qwen3-VL series), and supports scale-up with manageable computational cost increases and no need for expensive data curation or external reward models.

Speculation and Future Directions

Future research should focus on (1) improving computational efficiency via custom kernelization for in-place interventions, (2) extending causal regularization to more granular cross-layer and multi-hop interventions, (3) formalizing the integration with universal SCM-driven policy objectives, and (4) establishing practical standards for verifiable visual grounding and reasoning sensitivity in benchmarking. The extension to multi-agent and temporally extended reasoning, as well as compositional cross-modal interventions, presents a promising direction for robust, trustworthy, and interpretable LVLMs.

Conclusion

CFPO establishes a new direction for multimodal RL: by intervening on high-saliency cross-modal evidence in the latent space, it empirically enforces causal consistency, elevates reasoning fidelity, and mitigates pathological overreliance on language priors in large-scale LVLMs. Its underlying paradigm paves the way for causally sensitive AI systems that move beyond outcome optimization to principled enforcement of evidence-dependent reasoning.