Vector Policy Optimization: Training for Diversity Improves Test-Time Search

Abstract: LLMs must now generalize out of the box to novel environments and work inside inference-scaling search procedures, such as AlphaEvolve, that select rollouts with a variety of task-specific reward functions. Unfortunately, the standard paradigm of LLM post-training optimizes a pre-specified scalar reward, often leading current LLMs to produce low-entropy response distributions and thus to struggle at displaying the diversity that inference-time search will require. We propose Vector Policy Optimization (VPO), an RL algorithm that explicitly trains policies to anticipate diverse downstream reward functions and to produce diverse solutions. VPO exploits that rewards are often vector-valued in practice, like per-test-case correctness in code generation or, say, multiple different user personas or reward models. VPO is essentially a drop-in replacement for the GRPO advantage estimator, but it trains the LLM to output a set of solutions where individual solutions specialize to different trade-offs in the vector reward space. Across four tasks, VPO matches or beats the strongest scalar RL baselines on test-time search (e.g. pass@k and best@k), with the gap widening as the search budget grows. For evolutionary search, VPO models unlock problems that GRPO models cannot solve at all. As test-time search becomes more standardized, optimizing for diversity may need to become the default post-training objective.

• The paper introduces Vector Policy Optimization (VPO), a novel RL algorithm that preserves reward-space diversity by generating multiple candidate responses in LLM post-training.
• It employs multi-answer autoregressive generation and stochastic scalarization via Dirichlet sampling to effectively cover the Pareto frontier across competing reward objectives.
• Empirical results across various domains show that VPO outperforms traditional scalar RL methods, ensuring higher diversity and improved test-time search performance.

Vector Policy Optimization: Post-Training for Reward-Space Diversity in LLMs

Motivation and Theoretical Framing

The paper "Vector Policy Optimization: Training for Diversity Improves Test-Time Search" (2605.22817) addresses a core limitation in RL post-training of LLMs: the collapse of output diversity when optimizing over scalar rewards. In practice, LLMs are often deployed within a search-augmented pipeline leveraging techniques such as evolutionary search (e.g., AlphaEvolve, OpenEvolve), best-of-$k$ sampling, or self-consistency, where downstream components select among a pool of candidate responses based on diverse reward criteria. Classical policy-gradient algorithms, particularly scalar GRPO, consistently compress the candidate pool onto a redundant set of near-duplicate responses, diminishing the ability of test-time search to exploit output diversity.

The authors argue that RL post-training for LLMs should decouple exploration and exploitation: training should center on producing a diverse set of competent solutions spanning multiple reward trade-offs, leaving exploitation to test-time search. This perspective leverages naturally vector-valued reward spaces in practical tasks (e.g., per-test-case correctness in code generation, multi-hop QA, tool use), which enable Pareto-optimal specialization across distinct reward axes.

Vector Policy Optimization: Algorithmic Constructs

Vector Policy Optimization (VPO) is a novel RL algorithm promoting reward-space diversity. It comprises two main components:

• Multi-answer autoregressive generation: The LLM is trained to emit $m$ candidate completions within a single rollout, with explicit delimiters and sequential context, empowering in-context exploration strategies to diversify solutions.
• Stochastic scalarization via Dirichlet sampling: For each rollout, scalarization weight vectors $w$ are sampled from $\mathrm{Dir}(\alpha)$, uniformly spanning the simplex. The model's output set is rewarded based on the expected maximum scalarized reward across candidates and weightings:

$$R(S) = \mathbb{E}_{w\sim\mathrm{Dir}(\alpha)}\left[\max_{y\in S}w^\top r(x, y)\right]$$

This transforms the RL objective to explicitly cover the Pareto front in reward space, incentivizing specialization among candidates under orthogonal reward trade-offs.

VPO can be instantianted as a drop-in replacement for scalar GRPO in standard PPO-based pipelines. The combination of in-context multi-answer generation and stochastic reward scalarization forms a set-level reward mechanism fostering reward-space diversity rather than token-space variation alone.

Figure 1: VPO trains the LLM to optimize candidate sets across the reward simplex, preventing collapse onto a single scalar optimum and improving test-time search performance, as shown on LiveCodeBench.

Figure 2: VPO outputs multiple candidates per prompt, scored via sampled weight vectors, incentivizing specialization across objectives and promoting coverage of the reward Pareto frontier.

Empirical Evaluation Across Domains

Experiments span four domains with distinct reward decompositions:

• Maze navigation (engineered trade-offs among gold, diamond, lava-avoidance, and completion)
• MuSiQue multi-hop QA (binary citation per hop + continuous answer-F1)
• EUREQA chain reasoning (binary entity identification)
• ToolRL function-calling (binary format + continuous F1s for tool selection, argument keys/values)

VPO is benchmarked against scalar GRPO, Multi-RLVR (multi-answer scalar training), random-weighting GRPO (single-answer stochastic scalarization), Max-at-$k$ (direct best@$k$ optimization), MaxRL, and goal-conditioned GRPO.

Strong numerical results:

• Across all domains, VPO matches or exceeds the best scalar RL baselines on test-time best@$k$, with the gap widening as candidate pool size increases.
• On Maze and ToolRL, VPO consistently achieves highest diversity in reward space, outperforming Max-at-$k$ and scalar baselines in realized best@$k$.
• On LiveCodeBench, VPO enables the search pipeline (OpenEvolve loop) to solve problems unreachable by GRPO, particularly in evolutionary inference regimes.

  Figure 3: On MuSiQue and EUREQA, scalar GRPO plateaus quickly as $m$0 increases, signifying diversity collapse; VPO continues to unlock new high-reward candidates.

  

  Figure 4: On LiveCodeBench, VPO overtakes scalar GRPO in best@$m$1 and pass@$m$2 as sample budget grows, and maintains progress under evolutionary search.

  

  Figure 5: On Maze and ToolRL, VPO matches or exceeds scalar baselines across candidate pool sizes, with ToolRL exhibiting near-reward-ceiling convergence but highest reward-space diversity under VPO.

  

  Figure 6: VPO sustains reward-space diversity throughout training, measured by average $m$3 distance between reward vectors, while Multi-RLVR collapses except in the rare domain (EUREQA) where reward components correlate.

Analysis: Ablations and Regime Predictions

The gains from VPO are not due merely to multi-answer prompting or stochastic scalarization, but to the synergy and set-level optimization. Extensive ablations show:

• Multi-RLVR (multi-answer scalar training) supplies only token-space diversity, but reward-space diversity collapses early.
• Direct goal-conditioning fails to produce robustly diverse candidates across reward axes, even with explicit reward-weight encoding.
• Theoretical and empirical analysis predicts VPO's benefits vanish when reward components are collinear or redundant (as in ArmoRM-5 on UltraFeedback), validated by negligible improvement relative to scalar baselines.

Algorithmic normalization (e.g., GDPO) and increased resource allocation (3$m$4 evaluator calls in GRPO) improve baseline metrics only marginally; scalar RL still inherently suppresses alternative strategies.

Practical and Theoretical Implications

VPO offers critical practical advances for LLM systems integrating inference-time search:

• Test-time search efficacy depends on candidate pool diversity in reward space, not mere token variation.
• Scaling compute at inference (sampling/exploration) is optimal only if candidate diversity persists; VPO enables exploitation of larger candidate budgets.
• Evolutionary search and sophisticated pipelines (AlphaEvolve, OpenEvolve, ThetaEvolve) benefit from VPO's explicit preservation of sub-strategies and alternate reasoning chains.
• The regime where VPO excels is characterized by reward functions with orthogonal or competing objectives; when reward is scalar or near-collinear, standard scalar RL suffices.

Theoretically, VPO reframes RL post-training in LLMs as coverage over the reward Pareto frontier, aligning with multi-objective RL literature, lexicase selection in evolutionary computation, and Pareto-efficient policy optimization.

Speculation: Future Directions

Future developments may include:

• Integration of VPO-style objectives in RLHF pipelines with complex reward models and explicit user preference alignment (Hamid et al., 29 Sep 2025, Verine et al., 13 Aug 2025).
• Extension to larger-scale models and settings, including code synthesis on unseen competitive benchmarks and agentic tool use.
• Automated characterization of reward-space geometry (collinearity, dimensionality) to select appropriate RL objectives.
• Synthesis with outcome-based exploration and uncertainty quantification (Orney et al., 19 Apr 2026, Cui et al., 28 May 2025).
• Dynamically adaptive stochastic scalarization and set-level optimization during RLHF, potentially leveraging policy soups or group normalization [(Liu et al., 8 Jan 2026), 71095-71134].

Conclusion

VPO transforms RL post-training for LLMs into a set-level diversity optimization problem, ensuring candidate pools retain alternatives across reward axes and enabling test-time search procedures to exploit model variety. The empirical and theoretical results demonstrate clear superiority over scalar baselines in domains with meaningful reward decompositions. This work establishes reward-space diversity as a central post-training objective in search-augmented LLM pipelines, providing a foundation for further algorithmic advances in multi-objective RL, evolutionary selection, and search-integrated alignment.