Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards (2509.24981v1)

Abstract: RL with Verifiable Rewards (RLVR) has emerged as a promising paradigm for improving the reasoning abilities of LLMs. Current methods rely primarily on policy optimization frameworks like PPO and GRPO, which follow generalized policy iteration that alternates between evaluating the current policy's value and improving the policy based on evaluation. While effective, they often suffer from training instability and diversity collapse, requiring complex heuristic tricks and careful tuning. We observe that standard RLVR in math reasoning can be formalized as a specialized finite-horizon Markov Decision Process with deterministic state transitions, tree-structured dynamics, and binary terminal rewards. Though large in scale, the underlying structure is simpler than general-purpose control settings for which popular RL algorithms (e.g., PPO) were developed, suggesting that several sophisticated techniques in existing methods may be reduced or even omitted. Based on this insight, we prove a surprising result: the optimal action can be recovered from the Q-function of a fixed uniformly random policy, thereby bypassing the generalized policy iteration loop and its associated heuristics. We introduce Random Policy Valuation for Diverse Reasoning (ROVER) to translate this principle into a practical and scalable algorithm for LLM math reasoning, a minimalist yet highly effective RL method that samples actions from a softmax over these uniform-policy Q-values. ROVER preserves diversity throughout training, allowing sustained exploration of multiple valid pathways. Across multiple base models and standard math reasoning benchmarks, ROVER demonstrates superior performance in both \textbf{quality} (\textbf{+8.2} on pass@1, \textbf{+16.8} on pass@256) and \textbf{diversity} (\textbf{+17.6\%}), despite its radical simplification compared to strong, complicated existing methods.

• The paper proves that in deterministic, tree-structured MDPs, evaluating Q-values of a uniform random policy and acting greedily is sufficient for optimal math reasoning.
• It introduces softmax sampling over Q-values with a temperature parameter to balance exploration and exploitation, thereby preserving diversity.
• Empirical results on benchmarks like AIME and OlympiadBench show that ROVER outperforms traditional RL methods in both pass@1 and pass@k metrics while reducing computational overhead.

Random Policy Valuation for LLM Reasoning with Verifiable Rewards

Overview and Motivation

The paper "Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards" (2509.24981) introduces ROVER, a minimalist RL algorithm for post-training LLMs on math reasoning tasks with verifiable rewards. The central insight is that, for deterministic, tree-structured MDPs with binary terminal rewards—characteristic of math reasoning with LLMs—the optimal policy can be derived by evaluating the Q-function of a fixed uniformly random policy and acting greedily with respect to these Q-values. This approach bypasses the generalized policy iteration (GPI) loop and its associated instability and diversity collapse, which are prevalent in standard RLVR methods such as PPO and GRPO.

Theoretical Foundations

The paper formalizes LLM math reasoning as a finite-horizon MDP with deterministic transitions and binary terminal rewards. Unlike general RL environments, the state space is tree-structured, and each action deterministically expands the tree. The authors prove that, in this setting, the Q-values under a uniform random policy encode the probability of reaching a correct solution from a given state-action pair. Greedy selection over these Q-values yields the optimal policy (Theorem 1), a result that does not hold in general MDPs with cycles or stochastic transitions.

Figure 1: Comparison of learned Q-value maps in a tabular MDP. ROVER covers all optimal modes, while Q-learning and ROVER (greedy) collapse to a single mode.

However, greedy selection alone leads to mode collapse and poor diversity. To address this, the authors propose sampling actions via a softmax over the uniform-policy Q-values, introducing a temperature parameter $\rho$ to balance exploration and exploitation. Theoretical analysis (Theorem 2) provides a lower bound on the value of the induced policy, showing that as $\rho \to 0$, the policy approaches optimality, while higher $\rho$ increases diversity.

Algorithmic Implementation

ROVER is implemented by parameterizing the Q-function directly with the LLM's parameters, avoiding the need for a separate value network. The Q-value for a state-action pair is computed as a relative log-probability between the current and previous policy, scaled by $\rho$:

$$Q(s_t, a_t) = \rho \left( \log \pi_\theta(a_t|s_t) - \log \pi_{\theta_{\text{old}}}(a_t|s_t) \right)$$

To stabilize training, rewards are mean-centered within groups of sampled responses, and the centered reward is broadcast to all tokens in the response. The BeLLMan target for Q-value updates combines the centered reward and the expected Q-value of successor states under the uniform policy.

Empirical Results

ROVER is evaluated on both synthetic (Countdown) and competition-level math reasoning benchmarks (AIME24, AIME25, HMMT25, OlympiadBench, AMC23, MATH500, GPQA-diamond) using Qwen3-8B-Base, Qwen3-4B-Base, and DeepSeek-R1-Distill-Qwen-1.5B models. ROVER consistently outperforms PPO, GRPO, DAPO, and REINFORCE++ in terms of pass@1 and pass@$k$ metrics, with especially strong gains on challenging tasks and large $k$ values.

Figure 2: Pass@1 and Pass@256 results on Qwen3-8B-Base, averaged over AIME24, AIME25, and HMMT25. ROVER achieves superior quality and diversity.

Figure 3: pass@k of ROVER and baselines on Qwen3-8B-Base. ROVER maintains performance as k increases, unlike baselines which saturate or decline.

ROVER maintains higher policy entropy throughout training, preventing diversity collapse and enabling robust test-time scaling (majority voting). It discovers a greater number of distinct reasoning strategies per prompt, as measured by semantic clustering with an LLM judge.

Figure 4: Strategies discovered by Qwen3-8B-Base, GRPO, and ROVER. ROVER uncovers additional novel strategies absent from baselines.

Figure 5: Quality-diversity trade-off with different decoding temperatures (AIME24). ROVER consistently improves the Pareto front.

Ablation studies confirm the necessity of the BeLLMan target's Q-term for maintaining entropy and response length, and show that ROVER is robust to the scaling of this term. The temperature parameter $\rho$ is critical for balancing diversity and quality; $\rho=1$ is effective across tasks.

Figure 6: Impact of coefficient $\beta$ in ROVER on pass@1 and pass@64, entropy, and response length. Performance is stable for $\beta \in [0.2, 1.0]$.

Figure 7: Training curves of entropy for ROVER and baselines. ROVER maintains higher entropy, indicating expanded exploration space.

Practical Implications and Limitations

ROVER's radical simplification of RLVR for LLM reasoning tasks offers several practical advantages:

• Implementation Simplicity: No iterative policy improvement, no KL regularization, no value network, and no complex heuristics.
• Scalability: Efficiently parameterizes Q-values with the LLM, enabling training on large models and long horizons.
• Diversity Preservation: Maintains exploration and prevents mode collapse, improving pass@$k$ and generalization to OOD tasks.
• Resource Efficiency: Achieves SOTA results with reduced computational and data requirements compared to baselines.

However, the approach relies on the deterministic, tree-structured nature of the MDP induced by autoregressive LLM generation. Extensions to settings with tool calls, intermediate feedback, or non-tree dynamics may require further theoretical and algorithmic development. Approximation errors may arise when scaling to extremely large action spaces or horizons, though empirical results suggest robustness.

Future Directions

ROVER establishes a foundation for rethinking RLVR in domains where the MDP structure is amenable to random policy valuation. Future work may explore:

• Generalization to non-deterministic or graph-structured MDPs.
• Integration with tool-augmented LLMs or agentic workflows.
• Automated selection or adaptation of the temperature parameter for optimal diversity-quality trade-off.
• Application to other verifiable reasoning domains beyond mathematics.

Conclusion

ROVER demonstrates that, for LLM reasoning tasks with verifiable rewards, random policy valuation suffices to achieve both high-quality and diverse solutions. This challenges the necessity of complex RL algorithms in structurally simple MDPs and provides a scalable, theoretically grounded alternative for RLVR. The approach is empirically validated across multiple benchmarks and model scales, with strong improvements in both pass@1 and pass@$k$ metrics, diversity, and generalization. ROVER's simplicity and effectiveness suggest new directions for RLVR research and practical LLM post-training.