VGPO: Value-Anchored Group Policy Optimization

• VGPO is a framework that aligns flow matching-based image generators with complex objectives by redefining value estimation over temporal and group dimensions.
• It leverages a Temporal Cumulative Reward Mechanism and Adaptive Dual Advantage Estimation to assign precise per-step credit and stabilize policy gradients.
• Empirical benchmarks demonstrate that VGPO improves image quality and task-specific accuracy while mitigating issues like reward hacking.

Value-Anchored Group Policy Optimization (VGPO) is a framework for aligning flow matching-based image generators with complex objectives by redefining value estimation in both temporal and group dimensions. VGPO targets the limitations of Group Relative Policy Optimization (GRPO) when adapted to generative modeling, addressing imprecise temporal credit assignment and unstable optimization signals resulting from reduced reward diversity. The method incorporates dense process-aware value estimation and a dual anchoring mechanism to enable precise per-step updates and stable policy optimization, yielding state-of-the-art image quality and improved task-specific accuracy while mitigating reward hacking (Shao et al., 13 Dec 2025).

1. Formulation of Flow Matching as a Markov Decision Process

Flow matching-based image generation frameworks treat the denoising trajectory as a Markov Decision Process (MDP) parameterized over continuous time $t \in [0, 1]$, with discrete steps $T$. Let $x_0$ denote a clean image and $x_1$ denote pure noise; the forward process follows the linear path $x_t = (1-t)x_0 + t x_1$, while the reverse synthesis is driven by a learned velocity field. At each time step $t$, the model executes an action $a_t$—typically a stochastic sample update (SDE step)—transitioning from $s_t$ to $s_{t-1}$. A reward function $R(x_0, y)$, representing human preference or a task-specific metric, is available only at the end of the rollout, yielding a single sparse terminal reward $R_T$ per image.

2. Group Relative Policy Optimization: Limitations

GRPO, effective for LLM alignment, applies intra-group normalization for policy updates:

$$\hat{A}_t^i = \frac{R_T^i - \text{mean}_i(R_T^i)}{\text{std}_i(R_T^i)}$$

where samples $i = 1, \ldots, G$ are drawn per prompt and $G$ is the group size. This approach is limited in flow matching image generation due to:

• Uniform reward assignment across time: GRPO applies the same advantage $A_T$ to all temporal steps, disregarding the differential impact of early structure formation versus late-stage refinement on final image quality.
• Dependence on reward diversity: As the policy converges and $\text{std}_i(R_T^i) \rightarrow 0$, advantages explode or vanish, causing optimization stagnation or reward-hacking behaviors.

3. Temporal Cumulative Reward Mechanism (TCRM)

VGPO introduces process-aware value estimation to resolve the temporal misallocation of reward signals. For each time step $t$, VGPO computes instant and cumulative action values as follows:

• Instant Reward $R_t(s_t, a_t)$: After each action $a_t$, perform a one-step ODE from $s_{t-1}$ to a projected terminal state $$\hat{x}_0 = s_{t-1} - \tau_{t-1} v_\theta(s_{t-1}, \tau_{t-1})$$, and evaluate $R_t = RM(\hat{x}_0, y)$ using a pretrained reward model $RM$.
• Long-term Value $Q^{\pi}(s_t, a_t)$:

$$Q_t^i = \mathbb{E}_\pi \left[ \sum_{k=0}^{t-1} \gamma^k R_{t-k} \mid s_t, a_t \right]$$

with discount factor $\gamma \in [0, 1)$, estimated for each trajectory by accumulating discounted instant rewards.

• Bellman Recursion:

$$Q^{\pi}(s_t, a_t) = R_t + \gamma \mathbb{E}_{a_{t-1} \sim \pi}[Q^{\pi}(s_{t-1}, a_{t-1})]$$

This enables temporally precise credit assignment by propagating feedback to critical timesteps.

• Per-step Weighting:

$\omega_t^i = \frac{Q_t^i}{\text{mean}_t(Q_t^i)}$

Actions at timesteps contributing greater cumulative value are assigned proportionally larger weight in policy gradients.

4. Adaptive Dual Advantage Estimation (ADAE)

ADAE modifies GRPO’s normalization to maintain stable optimization signals as reward diversity changes:

• Relative Component: $$\hat{A}_t^i(\text{rel}) = \frac{Q_t^i - \text{mean}_i(Q_t^i)}{\text{std}_i(Q_t^i)}$$
• Absolute Component and Adaptive Anchoring:

$\alpha = k \cdot \text{std}_i(Q_t^i)$

for constant $k$.

$$\hat{A}_t^i = \omega_t^i \frac{ (1+\alpha) Q_t^i - \text{mean}_i Q_t^i }{ \text{std}_i Q_t^i }$$

As $\text{std}_i(Q_t^i) \rightarrow 0$, $(1+\alpha) \rightarrow 1$, shifting the numerator towards $Q_t^i$ and converting the advantage to an absolute signal, which persists even when reward diversity collapses, thereby stabilizing overall optimization.

5. Algorithmic Workflow

A high-level outline of VGPO training integrates TCRM and ADAE mechanisms throughout policy optimization:

for each training iteration: sample prompts {y} from C θ_old ← θ for each prompt y: for i in 1...G: s_T ∼ N(0, I) for t = T...1: s_{t-1} = SDE_step(s_t) \hat{x}_0 = ODE_project(s_{t-1}, τ_{t-1}, v_θ) R_t^i = RM(\hat{x}_0, y) Q_t^i = sum_{k=0}^{t-1} γ^k R_{t-k} ω_t^i = Q_t^i / mean_t Q_t^i α = k * std_i(Q_t^i) for each t, i: Â_t^i = ω_t^i * ((1+α)Q_t^i - mean_i Q_t^i) / std_i Q_t^i update θ maximizing policy objective and KL penalty

The update step maximizes:

$$\mathbb{E}_{\text{rollouts}} \left[ \min(r_t(\theta)\hat{A}_t^i, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon)\hat{A}_t^i) \right] - \beta \text{KL}(\pi_\theta\|\pi_{\theta_{\text{old}}})$$

6. Empirical Benchmarks and Outcomes

VGPO was empirically validated on three standard benchmarks:

Benchmark	Metric	Flow-GRPO	VGPO (w/o KL)
GenEval	Accuracy	0.95	0.97
GenEval	Quality	baseline	+9%
OCR	Accuracy	0.93	0.95
OCR	Aesthetic	improved	improved
PickScore	Task Score	modest	modest
PickScore	Pref. Metrics	improved	improved

VGPO consistently elevated both alignment (GenEval, OCR, PickScore) and image preference metrics (Aesthetic, DeQA, ImageReward, UnifiedReward). Ablation studies attribute accelerated convergence and improved task accuracy to TCRM, and late-stage stability with enhanced quality to ADAE; their combined effect is essential for full VGPO performance.

7. Implications and Extensions

VGPO resolves temporal credit misallocation by translating terminal rewards into dense, stepwise cumulative values, and secures persistent group-level signals via adaptive dual advantage. This dual anchoring approach prevents misleading updates and mitigates reward hacking, as the absolute advantage discourages optimization of negligible reward differences and the relative advantage fosters ongoing sample discrimination. VGPO accelerates model convergence and stabilizes late-phase training. Potential future directions include more efficient instant reward inference (e.g., streamlined ODE approximation), dynamic $\alpha$ scheduling, and extension to broader generative RL settings such as diffusion or video flow matching. A plausible implication is that VGPO’s framework may generalize to alignment scenarios exhibiting similar sparse reward and collapsed diversity characteristics (Shao et al., 13 Dec 2025).