On-policy Generator Optimization

• On-policy generator optimization is a training paradigm where models update their parameters using samples generated from their current behavior.
• It encompasses diverse architectures such as sequence generators, latent policies, and diffusion models, employing techniques like REINFORCE and PPO.
• This approach improves task-specific metrics and alignment by tightly coupling sample generation with immediate parameter updates, despite challenges in sample efficiency.

Searching arXiv for recent and foundational papers on on-policy generator optimization across RL, generative policies, and sequence generation. On-policy generator optimization denotes a class of training procedures in which a generator or policy is updated using samples produced by its current parameters rather than by a stale behavior distribution. In the cited literature, the optimized object may be a token generator, a conditional assignment generator, a latent-action policy paired with a decoder, a diffusion or flow policy for continuous control, a trajectory planner, a HyperNetwork that emits policy weights, or a deterministic one-step image generator. The common operational feature is that rollout data, candidate samples, or preference pairs are generated by the current model snapshot and then used immediately or within a strictly bounded update regime, often to preserve alignment between optimization signals and realized behavior (Hossam et al., 2020, Schulman et al., 2017, Zhang et al., 2 Dec 2025, Gao et al., 16 Apr 2026).

1. Conceptual scope and historical framing

In policy-gradient reinforcement learning, the generator is the policy itself. Proximal Policy Optimization formulates the policy as an action generator $\pi_\theta(a \mid s)$ and treats on-policy optimization as alternating between sampling data through interaction with the environment and optimizing a surrogate objective under the behavior policy $\pi_{\theta_{\mathrm{old}}}$, without mixing in older replay data (Schulman et al., 2017). In this sense, on-policy generator optimization is not a separate field from on-policy RL; rather, it is a broader viewpoint in which a policy is understood as a generative model over actions, tokens, trajectories, or other structured outputs.

That viewpoint appears explicitly outside classical control. OptiGAN treats a sequence generator as a stochastic policy over tokens and updates it with REINFORCE using rollouts sampled from the current generator, with no replay buffer or off-policy correction (Hossam et al., 2020). A conditional policy generator for dynamic constraint satisfaction frames a stateless contextual-bandit-like problem in which complete assignments are sampled from the current policy and optimized by entropy-regularized REINFORCE plus a class-conditional likelihood term (Lee et al., 21 Sep 2025). SGPO applies the same on-policy principle to LLM alignment: both winner and loser responses are generated from the same policy snapshot, after which a DPO-style loss is optimized against that snapshot as reference (Lee et al., 27 Jul 2025).

Recent work extends the concept to expressive generative policies whose action distributions are not conveniently Gaussian. GoRL performs on-policy optimization only in a tractable latent space while delegating action generation to a conditional decoder trained separately (Zhang et al., 2 Dec 2025). GenPO and GenPO++ instead recover exact or Jacobian-free likelihood ratios for diffusion- or flow-based policies so that PPO-style updates can operate directly on generative policies in online RL (Ding et al., 24 May 2025, Hu et al., 5 Jun 2026). GSB-PPO lifts proximal control from terminal actions to full generation trajectories, explicitly formulating PPO in path space (Gong et al., 23 Mar 2026). RAD-2 introduces On-policy Generator Optimization as a closed-loop planning mechanism that turns outcome feedback into structured longitudinal supervision for a diffusion trajectory generator (Gao et al., 16 Apr 2026). DrPO and PODPO show that some on-policy generator optimizers can be likelihood-free: the former uses reward-ranked candidates to synthesize a detached feature-space target, while the latter uses positive-only advantage-weighted local contrastive drifting (Jiang et al., 1 Jun 2026, Zhang, 15 Apr 2026).

This diversity suggests that “on-policy” is best understood as a property of data provenance and update coupling, not as a commitment to a single loss family, likelihood model, or architectural form.

2. Objectives and update mechanisms

The most direct formulation appears in sequence generation. OptiGAN defines the sequence-level objective

$J(\theta) = E_{x \sim \pi_\theta}[R(x)],$

with REINFORCE gradient

$$\nabla_\theta J(\theta) = E_{x \sim \pi_\theta}\left[\sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\,(R(x)-b)\right].$$

Its implementation uses generator hidden states $h_t$ as states, a time-dependent return $U_t$ from on-policy Monte Carlo rollouts, and a running-average baseline:

$$\nabla_\theta J(\theta)=E_{X\sim P_G}\left[\sum_t (U_t-b(h_t)) \nabla_\theta \log p_G(X \mid h_t,\theta)\right].$$

The full generator objective then combines maximum likelihood, a non-saturating GAN term, and the on-policy RL term (Hossam et al., 2020).

PPO replaces unclipped REINFORCE with a proximal likelihood-ratio surrogate. With

$$r_t(\theta)=\frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)},$$

the clipped objective is

$$L^{\mathrm{CLIP}}(\theta)=E_t\left[\min\left(r_t(\theta)A_t,\;\mathrm{clip}(r_t(\theta),1-\epsilon,1+\epsilon)A_t\right)\right].$$

The update remains on-policy because the batch is collected under $\pi_{\theta_{\mathrm{old}}}$ and discarded after a small number of epochs; importance ratios and clipping keep the optimized policy close to the behavior generator (Schulman et al., 2017).

Several later systems preserve the on-policy regime while relocating optimization. GoRL factorizes the policy into a tractable latent policy $\pi_{\theta_{\mathrm{old}}}$0 and a conditional generative decoder $\pi_{\theta_{\mathrm{old}}}$1, then applies PPO in latent space:

$\pi_{\theta_{\mathrm{old}}}$2

augmented with $\pi_{\theta_{\mathrm{old}}}$3 (Zhang et al., 2 Dec 2025). GSB-PPO generalizes the proximal ratio from terminal actions to generator trajectories,

$\pi_{\theta_{\mathrm{old}}}$4

and defines both clipped and penalty-based path-space surrogates (Gong et al., 23 Mar 2026).

A separate line of work makes on-policy optimization feasible for expressive generative policies by restoring exact density information. GenPO constructs an invertible mapping with doubled dummy actions so that PPO can use exact log-likelihoods, unbiased entropy, and unbiased KL estimates (Ding et al., 24 May 2025). GenPO++ uses a reversible high-order solver with solver-history memory; its Jacobian determinant depends only on fixed solver coefficients, so exact likelihood ratios are Jacobian-free and remain compatible with PPO-style trust-region control (Hu et al., 5 Jun 2026).

Other papers abandon explicit likelihood ratios while remaining on-policy. DrPO samples candidates from the current one-step generator, ranks them with a target reward, synthesizes a dipole preference field plus a reference drift, and optimizes a detached feature-space regression target (Jiang et al., 1 Jun 2026). PODPO likewise uses a likelihood-free drifting loss on positive-advantage on-policy samples only, with no log-likelihoods, KL terms, or gradient clipping (Zhang, 15 Apr 2026). These formulations indicate that on-policy generator optimization can be realized either through exact ratios, latent-space surrogates, or non-parametric drift fields.

3. Representative domains and architectures

The literature spans heterogeneous generators and optimization targets.

Paper	Generator being optimized	On-policy mechanism
"OptiGAN" (Hossam et al., 2020)	LSTM sequence generator	REINFORCE with current-policy rollouts
"Conditional Policy Generator for Dynamic Constraint Satisfaction and Optimization" (Lee et al., 21 Sep 2025)	Factorized assignment generator	Entropy-regularized REINFORCE on current samples
"Goal-Conditioned Generators of Deep Policies" (Faccio et al., 2022)	HyperNetwork generating policy weights	Current-generator rollouts populate supervised evaluator targets
"Upside Down Reinforcement Learning with Policy Generators" (Ventura et al., 27 Jan 2025)	Command-conditioned HyperNetwork	Fresh generated policies can be used in a strict on-policy variant
"GoRL" (Zhang et al., 2 Dec 2025)	Latent policy plus generative decoder	On-policy PPO in latent space
"RAD-2" (Gao et al., 16 Apr 2026)	Diffusion trajectory generator	Closed-loop on-policy longitudinal target construction

In sequence generation, OptiGAN uses an autoregressive LSTM generator, Gumbel-Softmax for discrete text, and Gaussian outputs for real-valued sequences. Its task rewards are BLEU-N for text and McGrew score for air-combat trajectories, with credit assignment performed by on-policy rollouts from partially generated prefixes (Hossam et al., 2020). In conditional CSP optimization, the policy factorizes across statistically independent variables and uses independent feedforward softmax heads; dynamic constraints are encoded as class labels concatenated with Gaussian noise (Lee et al., 21 Sep 2025).

Hypernetwork-based approaches move the generator into parameter space. GoGePo treats a return-conditioned HyperNetwork as the generator of policy weights, then collects on-policy rollouts from the generated policy and trains a differentiable evaluator $\pi_{\theta_{\mathrm{old}}}$5 alongside the generator (Faccio et al., 2022). UDRLPG removes the evaluator and instead regresses directly from return commands to stored policy weights. Its canonical algorithm is buffered and therefore off-policy, but the paper explicitly describes a strict on-policy variant in which updates use only fresh policies sampled from the current generator (Ventura et al., 27 Jan 2025).

In continuous-control RL with expressive action distributions, several architectural solutions appear. GoRL decouples optimization from generation by freezing a decoder during PPO updates and freezing the latent policy during decoder refinement (Zhang et al., 2 Dec 2025). GenPO uses exact diffusion inversion with doubled dummy actions; the environment executes the averaged action while PPO is computed on the invertible dummy-action representation (Ding et al., 24 May 2025). GenPO++ preserves the original action dimension by making the solver reversible with history states (Hu et al., 5 Jun 2026). GSB-PPO models the policy itself as a reverse-time stochastic generation chain and optimizes the full path measure (Gong et al., 23 Mar 2026). DBPO equips a pretrained one-step Drift-Based Policy backbone with an observation-conditioned diagonal Gaussian exploration head, stores latent noise samples, and computes exact executed-prefix likelihoods without latent marginalization (Gao et al., 4 Apr 2026).

Closed-loop planning and alignment introduce still other generator objects. RAD-2 optimizes a diffusion planner in closed-loop driving, but only through structured longitudinal targets derived from safety and efficiency feedback, leaving candidate reranking to an RL-trained discriminator (Gao et al., 16 Apr 2026). SGPO unifies improver and policy within one LLM and performs on-policy preference optimization by refining current responses into slightly better ones (Lee et al., 27 Jul 2025). DrPO applies the same logic to deterministic one-step text-to-image models by synthesizing feature-space targets from current samples and reward rankings (Jiang et al., 1 Jun 2026).

4. Credit assignment, stability, and regularization

A central technical problem is that on-policy updates are usually better aligned with the current generator but often noisier and less sample-efficient. Different papers stabilize this regime in markedly different ways.

OptiGAN addresses sequence-level credit assignment with on-policy Monte Carlo rollouts at each time step, using $\pi_{\theta_{\mathrm{old}}}$6 rollouts on MS-COCO and $\pi_{\theta_{\mathrm{old}}}$7 on EMNLP News, plus a fixed running-average baseline and generator gradient clipping at $\pi_{\theta_{\mathrm{old}}}$8 (Hossam et al., 2020). It does not report entropy regularization, gradient penalties, spectral normalization, or KL constraints against a prior policy. The paper argues instead that a hybrid ML+GAN objective minimizes

$\pi_{\theta_{\mathrm{old}}}$9

at equilibrium, thereby counteracting the diversity collapse observed in RL-only training.

PPO stabilizes on-policy learning by clipping or KL penalties, learned value baselines, advantage normalization, entropy bonuses, and optional KL early stopping (Schulman et al., 2017). GePPO is explicitly not on-policy, but it is instructive as a contrast because it generalizes PPO’s clipping window to reused recent samples and frames sample reuse as a controlled deviation from the on-policy trust-region spirit (Queeney et al., 2021). This contrast clarifies that “on-policy” and “stable” are separable properties: stability may come from proximity control, while strict on-policyness concerns data collection.

GoRL introduces a different stabilization principle: decoupling optimization from generation. Policy gradients touch only the tractable latent encoder, while the expressive decoder is trained with Flow Matching or diffusion losses on recent on-policy buffers, always pairing $J(\theta) = E_{x \sim \pi_\theta}[R(x)],$0 with fresh $J(\theta) = E_{x \sim \pi_\theta}[R(x)],$1 rather than $J(\theta) = E_{x \sim \pi_\theta}[R(x)],$2 (Zhang et al., 2 Dec 2025). The method further uses a latent KL regularizer, an identity-like decoder warm-up, staged refinement, advantage normalization, and standard PPO entropy regularization.

Systems with exact generative likelihoods use proximal machinery but must solve an additional numerical problem: how to evaluate policy ratios faithfully. GenPO restores unbiased entropy and KL estimates by exact diffusion inversion, enabling KL-adaptive learning rates and entropy regularization in PPO (Ding et al., 24 May 2025). GenPO++ removes dummy-action overhead by making the solver exactly reversible and obtaining a log-determinant that is constant per step, so the augmented KL becomes a conservative trust-region monitor without neural Jacobian evaluation (Hu et al., 5 Jun 2026). GSB-PPO argues that clipping becomes brittle when ratios compound across many generator steps, and therefore recommends a path-space penalty

$J(\theta) = E_{x \sim \pi_\theta}[R(x)],$3

that approximates path KL through drift mismatch (Gong et al., 23 Mar 2026).

Likelihood-free methods replace trust regions with drift construction and detached targets. DrPO uses a non-parametric dipole preference field from reward-ranked on-policy samples plus a reference drift from a frozen base generator, then optimizes only a regression loss in feature space (Jiang et al., 1 Jun 2026). PODPO relies on positive-only advantage weighting, local contrastive drifting, and multi-temperature variance compression, and explicitly avoids post-hoc penalization of negative-advantage samples (Zhang, 15 Apr 2026). RAD-2 introduces Temporally Consistent Group Relative Policy Optimization for its discriminator and couples it with generator-side On-policy Generator Optimization; the generator update is deliberately restricted to longitudinal components so that sparse closed-loop rewards do not destabilize high-dimensional trajectory denoising (Gao et al., 16 Apr 2026).

A recurring misconception is that on-policy generator optimization must either be plain REINFORCE or PPO over tractable action densities. The cited literature contradicts that view. On-policy updates may be exact-likelihood, latent-space, surrogate-path, detached-regression, or likelihood-free, provided the optimization signal is derived from samples generated by the current model.

5. Empirical behavior across tasks

The empirical record is mixed in one respect and consistent in another. The mixed part is that raw on-policy optimization often improves task scores but can degrade diversity or stability if left unregularized. The consistent part is that coupling on-policy updates with an appropriate backbone or trust mechanism usually improves the intended objective without catastrophic collapse.

OptiGAN illustrates both sides. On MS-COCO, SeqGAN reports BLEU-4 $J(\theta) = E_{x \sim \pi_\theta}[R(x)],$4 and NLL $J(\theta) = E_{x \sim \pi_\theta}[R(x)],$5; OptiGAN-OnlyRL reaches BLEU-4 $J(\theta) = E_{x \sim \pi_\theta}[R(x)],$6 and NLL $J(\theta) = E_{x \sim \pi_\theta}[R(x)],$7; OptiGAN-OnlyGAN reaches BLEU-4 $J(\theta) = E_{x \sim \pi_\theta}[R(x)],$8 and NLL $J(\theta) = E_{x \sim \pi_\theta}[R(x)],$9; and full OptiGAN reaches BLEU-4 $$\nabla_\theta J(\theta) = E_{x \sim \pi_\theta}\left[\sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\,(R(x)-b)\right].$$0 and NLL $$\nabla_\theta J(\theta) = E_{x \sim \pi_\theta}\left[\sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\,(R(x)-b)\right].$$1 (Hossam et al., 2020). On EMNLP 2017 News, full OptiGAN obtains BLEU-4 $$\nabla_\theta J(\theta) = E_{x \sim \pi_\theta}\left[\sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\,(R(x)-b)\right].$$2 and NLL $$\nabla_\theta J(\theta) = E_{x \sim \pi_\theta}\left[\sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\,(R(x)-b)\right].$$3, which is the best NLL among the reported variants. On real-valued Stern Conversion trajectories, the McGrew score rises from $$\nabla_\theta J(\theta) = E_{x \sim \pi_\theta}\left[\sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\,(R(x)-b)\right].$$4 for an LSTM baseline to $$\nabla_\theta J(\theta) = E_{x \sim \pi_\theta}\left[\sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\,(R(x)-b)\right].$$5 for OptiGAN, close to the ACE-Zero simulator average of $$\nabla_\theta J(\theta) = E_{x \sim \pi_\theta}\left[\sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\,(R(x)-b)\right].$$6. The paper explicitly notes that RL-only “cheats” BLEU by repeating frequent $$\nabla_\theta J(\theta) = E_{x \sim \pi_\theta}\left[\sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\,(R(x)-b)\right].$$7-grams or producing short, low-diversity outputs.

GoRL reports strong online RL gains from latent-space on-policy optimization with expressive decoders. On HopperStand, GoRL(Diff) reaches a normalized return above $$\nabla_\theta J(\theta) = E_{x \sim \pi_\theta}\left[\sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\,(R(x)-b)\right].$$8, while Gaussian PPO is reported at $$\nabla_\theta J(\theta) = E_{x \sim \pi_\theta}\left[\sum_{t=1}^T \nabla_\theta \log \pi_\theta(a_t \mid s_t)\,(R(x)-b)\right].$$9, and FPO and DPPO collapse to near-zero values on the same task (Zhang et al., 2 Dec 2025). More broadly, GoRL reports final normalized returns of $h_t$0 on CheetahRun for GoRL(Diff) and $h_t$1 on WalkerWalk for GoRL(FM). The paper also reports that GoRL’s HopperStand action distribution evolves from unimodal at $h_t$2M steps to clearly bimodal at $h_t$3M steps.

RAD-2 reports that its unified generator-discriminator framework reduces the collision rate by $h_t$4 compared with strong diffusion-based planners, and its ablation isolates a sizable contribution from generator-side OGO alone: collision rate falls from $h_t$5 to $h_t$6, Safety@1 rises from $h_t$7 to $h_t$8, and Safety@2 rises from $h_t$9 to $U_t$0 (Gao et al., 16 Apr 2026). The best reported closed-loop BEV-Warp results arise when OGO is combined with discriminator-side TC-GRPO, yielding CR $U_t$1, Safety@1 $U_t$2, and EP-Mean $U_t$3.

On-policy self-generation also shows strong alignment gains in LLMs. For Qwen2.5-Base (7B), SGPO reports AlpacaEval 2.0 LC $U_t$4 and WR $U_t$5 versus DPO LC $U_t$6 and WR $U_t$7, and Arena-Hard WR $U_t$8 versus $U_t$9 for DPO (Lee et al., 27 Jul 2025). For Llama3-Base (8B), SGPO reports AlpacaEval 2.0 LC $$\nabla_\theta J(\theta)=E_{X\sim P_G}\left[\sum_t (U_t-b(h_t)) \nabla_\theta \log p_G(X \mid h_t,\theta)\right].$$0 and WR $$\nabla_\theta J(\theta)=E_{X\sim P_G}\left[\sum_t (U_t-b(h_t)) \nabla_\theta \log p_G(X \mid h_t,\theta)\right].$$1 versus DPO LC $$\nabla_\theta J(\theta)=E_{X\sim P_G}\left[\sum_t (U_t-b(h_t)) \nabla_\theta \log p_G(X \mid h_t,\theta)\right].$$2 and WR $$\nabla_\theta J(\theta)=E_{X\sim P_G}\left[\sum_t (U_t-b(h_t)) \nabla_\theta \log p_G(X \mid h_t,\theta)\right].$$3. An ablation on Llama3-8B shows SPIN at LC $$\nabla_\theta J(\theta)=E_{X\sim P_G}\left[\sum_t (U_t-b(h_t)) \nabla_\theta \log p_G(X \mid h_t,\theta)\right].$$4, WR $$\nabla_\theta J(\theta)=E_{X\sim P_G}\left[\sum_t (U_t-b(h_t)) \nabla_\theta \log p_G(X \mid h_t,\theta)\right].$$5; an intermediate SGPO variant at LC $$\nabla_\theta J(\theta)=E_{X\sim P_G}\left[\sum_t (U_t-b(h_t)) \nabla_\theta \log p_G(X \mid h_t,\theta)\right].$$6, WR $$\nabla_\theta J(\theta)=E_{X\sim P_G}\left[\sum_t (U_t-b(h_t)) \nabla_\theta \log p_G(X \mid h_t,\theta)\right].$$7; and full on-policy SGPO at LC $$\nabla_\theta J(\theta)=E_{X\sim P_G}\left[\sum_t (U_t-b(h_t)) \nabla_\theta \log p_G(X \mid h_t,\theta)\right].$$8, WR $$\nabla_\theta J(\theta)=E_{X\sim P_G}\left[\sum_t (U_t-b(h_t)) \nabla_\theta \log p_G(X \mid h_t,\theta)\right].$$9.

For deterministic one-step image generators, DrPO improves SD-Turbo and SDXL-Turbo on multiple preference metrics while removing reward-model backpropagation. On HPSv3, under matched effective-batch, update time drops from $$r_t(\theta)=\frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)},$$0 s to $$r_t(\theta)=\frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)},$$1 s and backward time from $$r_t(\theta)=\frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)},$$2 s to $$r_t(\theta)=\frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)},$$3 s, corresponding to a $$r_t(\theta)=\frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)},$$4 reduction in training computation (Jiang et al., 1 Jun 2026). For online locomotion with implicit generative policies, PODPO reports approximately $$r_t(\theta)=\frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)},$$5 higher converged return than PPO on Unitree GO2 gait locomotion under identical network and training configurations (Zhang, 15 Apr 2026).

These results do not imply that on-policy optimization is universally superior. They do, however, support a narrower conclusion present across the literature: when current-policy sampling is paired with an update rule that respects generator structure, on-policy optimization can improve goal metrics, preference alignment, or closed-loop robustness without forcing a return to simple unimodal policies.

6. Limitations, misconceptions, and open directions

The most persistent limitation is sample efficiency. OptiGAN states directly that on-policy REINFORCE requires many rollouts and has high variance, and does not use actor-critic or learned baselines beyond a running average (Hossam et al., 2020). PPO, while more stable, still discards batches after a few epochs and is therefore less sample-efficient than replay-based methods (Schulman et al., 2017). GoRL improves stability by optimizing only a latent encoder, but it still incurs the cost of maintaining two models and two training phases, and its performance depends on decoder quality and update scheduling (Zhang et al., 2 Dec 2025).

Another limitation is structural assumption. The conditional policy generator for dynamic CSPs assumes statistical independence of variables and disjoint solution regions for dynamic classes; the paper explicitly notes that extending to dependent variables may require more expressive architectures such as attention mechanisms (Lee et al., 21 Sep 2025). GoGePo’s evaluator-based generator optimization depends on the accuracy of a parameter-conditioned surrogate $$r_t(\theta)=\frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)},$$6, and UDRLPG notes increased variance when the evaluator is removed (Faccio et al., 2022, Ventura et al., 27 Jan 2025). RAD-2’s OGO relies on BEV feature warping and longitudinal-only adjustments; the paper presents extension to broader latent-world-model settings as future work (Gao et al., 16 Apr 2026).

A separate misconception is that exact likelihoods are always necessary. They are necessary for PPO-style ratio control in GenPO, GenPO++, and GSB-PPO, and these papers spend most of their technical effort on recovering or reformulating those ratios (Ding et al., 24 May 2025, Hu et al., 5 Jun 2026, Gong et al., 23 Mar 2026). But DrPO and PODPO show that on-policy generator optimization can also be likelihood-free when the optimization target is a detached drift field synthesized from current samples (Jiang et al., 1 Jun 2026, Zhang, 15 Apr 2026). The more precise statement is therefore conditional: exact likelihoods are necessary for likelihood-ratio objectives, not for on-policy optimization as such.

There is also a recurring trade-off between expressiveness and control. GenPO doubles the action space with dummy actions to obtain exact inversion, which adds computation and requires a compression loss (Ding et al., 24 May 2025). GenPO++ removes the action-doubling but introduces sensitivity to the solver coefficient $$r_t(\theta)=\frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)},$$7 and to the number of reversible solver steps (Hu et al., 5 Jun 2026). GSB-PPO finds that path-space clipping is compatible with on-policy training but underperforms a penalty formulation in both stability and performance (Gong et al., 23 Mar 2026). DBPO preserves strict one-step inference but depends on anchor regularization to remain near a pretrained one-step backbone, and removing the anchor reduces online gains (Gao et al., 4 Apr 2026).

Open directions named within the papers are notably concrete. OptiGAN proposes actor-critic or stronger stabilizers as obvious next steps, as well as latent-space control for discrete sequences (Hossam et al., 2020). Conditional CSP generation points toward dependent-variable architectures and auxiliary classifiers for high-dimensional class likelihoods (Lee et al., 21 Sep 2025). GoRL emphasizes better scheduling and decoder training, while RAD-2 highlights extension beyond BEV-specific simulation (Zhang et al., 2 Dec 2025, Gao et al., 16 Apr 2026). DrPO explicitly suggests that sample-based gradient synthesis may be useful beyond online reward ranking, including offline settings (Jiang et al., 1 Jun 2026).

Taken together, these works define on-policy generator optimization not as a single algorithm but as an organizing principle: the generator is updated from its own current behavior, and the main design question becomes how to transform those current samples into stable, information-rich optimization signals.