VIMPO: Value-Implicit Policy Optimization for LLMs

Abstract: Reinforcement learning with verifiable rewards has become a central tool for improving the reasoning ability of LLMs, but current methods face a trade-off between simplicity and credit assignment. Group-relative methods such as GRPO avoid training a critic, but typically assign a trajectory-level advantage to every token. Actor-critic methods provide denser learning signals, but require a learned value function with its own training instability. We introduce VIMPO, a critic-free policy optimization method that derives a policy-implied value function from the optimality conditions of KL-regularized reinforcement learning. For autoregressive generation, the resulting value recurrence can be written in terms of policy-reference log-ratios and anchored by the terminal condition that no future reward remains at the end of a trajectory. This gives a simple value loss that incorporates outcome-level verifiable rewards without training a critic. The same derivation also yields a critic-free actor advantage, allowing VIMPO to separate reward incorporation through the value loss from policy improvement through a PPO-style actor update. On mathematical RLVR benchmarks, VIMPO improves over GRPO across MATH-500, AIME 2024, AIME 2025, and OlympiadBench, with especially larger gains on competition-style evaluations. Under noisy rewards, VIMPO retains a consistent advantage over GRPO, suggesting that policy-implied value optimization can provide finer credit assignment while preserving the practical simplicity of critic-free training.

• The paper presents VIMPO, a critic-free RL method that achieves token-level, Bellman-consistent credit assignment for LLMs.
• It reformulates KL-regularized policy optimization as a deterministic MDP, eliminating the need for a trainable value network.
• Empirical evaluations show VIMPO outperforms GRPO by 2–3% in accuracy and demonstrates robust performance under noisy rewards.

VIMPO: Value-Implicit Policy Optimization for LLMs

Motivation and Context

LLMs benefit significantly from reinforcement learning post-training, particularly in domains that require complex multi-step reasoning, such as mathematical problem solving. The dominant RL approaches can be dichotomized into actor-critic methods, which enable dense token-level credit assignment using a learned value function, and critic-free group-relative methods, which broadcast a trajectory-level advantage across tokens while avoiding the instabilities associated with learning an additional critic network. This dichotomy sets up a trade-off between learning signal granularity and training simplicity.

VIMPO (Value-IMplicit Policy Optimization) is introduced to reconcile this trade-off. It enables token-level, Bellman-consistent credit assignment in a critic-free setting, eliminating the need for a learned value network while maintaining dense supervision characteristics traditionally unique to actor-critic methods. The approach is theoretically grounded in the optimality conditions of KL-regularized RL for deterministic-transition MDPs, as instantiated by autoregressive language generation.

Figure 1: Overview of VIMPO. Given a prompt $q$, the policy generates a completion $o$, scored to obtain an outcome reward $r$. VIMPO uses this reward to train the policy-implied value loss, while the policy and frozen reference model define a token-level TD signal used to form the actor advantage. This separates reward incorporation from policy improvement without training an explicit critic.

Methodological Advances

VIMPO reformulates KL-regularized policy optimization for sequence generation as a deterministic MDP problem and derives a closed-form recurrence for the value function based on the log-likelihood ratio between the current policy and a frozen reference policy. This eliminates the need for an additional trainable value network. The terminal optimality condition, $V^*({s}_T) = 0$, is leveraged as an anchor, allowing a straightforward squared-error loss on the value function at the end of each trajectory. The cumulative sum of per-token policy-reference log-ratios is trained to match the centered final reward—a scalar outcome based on verifiable evaluation, as is common in math benchmarks.

The same recurrence yields a policy-implied token-level advantage signal—specifically, a Bellman residual formulated via log-ratios only. This advantage is directly usable in standard PPO-style clipped actor updates, mirroring the dense, step-wise policy improvement of actor-critic algorithms without the need for value model co-adaptation.

This dual construction allows VIMPO to maintain the practical simplicity of group-based rollout strategies while achieving higher granularity in credit assignment. The methodology ensures that external rewards influence the value loss, while advantage estimation—and thus policy improvement—relies solely on internal quantities computable from the policy and reference.

Empirical Evaluation

VIMPO is evaluated against GRPO, a widely adopted group-relative critic-free baseline, across several arithmetic RLVR benchmarks: MATH-500, AIME 2024, AIME 2025, and OlympiadBench. All experiments use Qwen3-4B-Base as the model initializer, maintaining consistent hyperparameters across methods to enable isolated comparison of optimization objectives.

Figure 2: Main comparison among naive GRPO, GRPO, and VIMPO under clean verifier rewards, reporting metrics across training.

VIMPO outperforms both naive and token-level GRPO variants, reaching higher validation accuracies and improving training dynamics, especially on competition-style reasoning benchmarks (e.g., AIME 2025). Notably, VIMPO achieves an average accuracy gain of approximately 2–3 percentage points compared to GRPO across tasks. These improvements are not attributed to longer completions or increased entropy alone—VIMPO explores longer outputs early but converges to shorter, more accurate responses as training progresses.

Robustness to imperfect reward signals is a prominent outcome. Under a synthetic scenario where rewards are randomly corrupted at a 25% flip rate, VIMPO's final accuracy consistently exceeds GRPO, with particularly large gains on the more difficult AIME datasets.

Figure 3: Noisy-reward stress test: with 25% label flipping, VIMPO maintains higher clean performance than GRPO, reflecting improved robustness to reward corruption.

Ablations and Training Dynamics

Critical ablation studies examine the roles of the value loss coefficient ($\beta$) and the actor update coefficient ($c_A$). The value-only variant ($c_A=0$) demonstrates steady, albeit slower, improvement, while adding the PPO-style update accelerates learning and increases the policy's deviation from the reference distribution. Excessive regularization via high $\beta$ or high $c_A$ constrains policy movement and hinders further improvement.

Figure 4: Ablation on VIMPO coefficients. Full objective achieves fastest accuracy gain but with increased KL to the reference; high-regularization slows both learning and policy updates.

Response-length dynamics and policy entropy curves further differentiate VIMPO from GRPO. While GRPO variants exhibit a monotonic entropy decrease consistent with known "entropy-collapse," VIMPO's entropy fluctuates in later training stages, indicating a more flexible adaptation in the policy distribution.

Token-Level Credit Assignment

A defining aspect of VIMPO is its capacity to provide token-aligned, policy-implied advantage signals. Qualitative analysis on mathematical reasoning traces shows that VIMPO can localize high advantage to tokens critical to correct reasoning paths and assign negative advantage near erroneous or overcounting steps.

Figure 5: Token-aligned comparison between the VIMPO GAE actor signal and a Monte Carlo rollout-based temporal-difference estimate. Both capture advantage structure correlating with reasoning correctness.

Figure 6: Token-level view of critical regions in a combinatorics solution. Colors highlight directional advantage, distinguishing valid from erroneous reasoning steps.

Implications and Future Directions

VIMPO demonstrates that critic-free policy optimization need not sacrifice token-level credit assignment granularity. Its decoupling of external rewards (value loss) and internal log-ratio advantages (actor update) leads to better policy stability and robustness, especially under imperfect reward signals—a critical asset in RLVR setups where verifier noise is nontrivial.

The approach naturally interfaces with established PPO and RLHF pipelines but can be extended further. Adaptive schedules of the reference constraint parameter ($\beta$) and actor coefficient could balance early stabilization against the need for exploration/later improvement. Approximate KL computation methods could reduce computational burdens for large-scale or long-sequence models. Future research should explore scaling up VIMPO to larger models, applying it to code synthesis, open-ended instructions, or robotics, and benchmarking against tuned actor-critic methods where dense state rewards are more prevalent.

Conclusion

VIMPO introduces a theoretically principled, practically effective critic-free optimization method that restores dense token-level credit assignment for long-horizon, outcome-supervised LLM RL. Empirical results emphasize not only stronger learning dynamics and improved competitive performance—especially on challenging reasoning tasks—but also heightened robustness to reward noise. VIMPO thereby bridges the methodological gap between critic-free and actor-critic paradigms in RL for LLMs, advancing the stability, efficiency, and interpretability of RLVR pipelines (2606.20008).