GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratios

Abstract: Generative policies provide expressive and multimodal action distributions, making them attractive for reinforcement learning (RL) in complex continuous-control tasks. Among them, flow-based policies are especially appealing because they generate actions through deterministic transport maps. However, applying such generative policies to likelihood-based on-policy learning remains limited by the difficulty of evaluating the probability of executed actions. Existing flow RL methods either replace the true action-density ratio with approximate surrogates, which can introduce biased updates, or recover exact likelihoods through dummy-action augmentation, which enlarges the policy space and increases computation. In this work, we propose GenPO++, a reversible generative policy optimization framework that uses history states as auxiliary memory in a high-order reversible ODE solver, yielding exact inversion without changing the original action dimension. The resulting generative policy map has a log-determinant determined only by fixed solver coefficients, enabling exact and Jacobian-free likelihood-ratio computation. This design preserves the expressiveness of generative flow policies while avoiding both action ratio bias and dummy-action overhead. We evaluate GenPO++ on large-scale simulated control, fine-tuning, and real-world robotic manipulation tasks, where it achieves competitive or superior performance over state-of-the-art on-policy RL methods, while improving training stability and computational efficiency.

• The paper introduces an exact, Jacobian-free method for computing likelihood ratios in on-policy RL via reversible high-order solvers.
• It eliminates the need for dummy-action augmentation, ensuring stable policy updates and efficient computation in continuous control.
• Experimental results demonstrate superior sample efficiency and sim-to-real transfer in both simulated benchmarks and real robotic tasks.

GenPO++: Jacobian-Free Reversible Generative Policy Optimization

Introduction

GenPO++ introduces a new framework for on-policy reinforcement learning (RL) with expressive generative policies. Building upon advancements in flow-based and diffusion models, the paper specifically addresses the bottleneck of exact action-likelihood evaluation in continuous control: a critical aspect for robust on-policy algorithms such as PPO, KL control, and entropy regularization. Most prior works either use approximate likelihood ratios—thus biasing policy updates—or exactly compute action likelihoods at significant computational cost by augmenting the action space with dummy variables. GenPO++ achieves exact, Jacobian-free likelihood-ratio computation while keeping the native action dimensionality, leveraging solver-history states from high-order reversible ODE solvers. This enables stable, efficient RL with expressive generative policies in both simulated and real-world robotic platforms (2606.06967).

Figure 1: Schematic comparison among FPO, GenPO, and GenPO++. GenPO++ achieves exact inversion and Jacobian-free likelihood-ratio computation using solver-history states, in contrast to prior methods that rely on biased surrogates or action augmentation.

Technical Approach

Reversible High-Order Flow-Policy Solvers

The fundamental innovation of GenPO++ is to implement an invertible solver by recording history states (i.e., previous ODE steps) as auxiliary memory. Unlike GenPO, which requires dummy-action augmentation and thus increases the policy's action space, GenPO++ operates in the original action space. The update uses a high-order Adams–Bashforth-like transition of the form:

$$x_{i+1} = (1-\sigma) x_i + \sigma x_{i-1} + (1+\sigma)\Delta_t v_\theta(x_i, t_i | s)$$

where $\sigma$ is a scalar controlling history correction. This design is reversible due to the explicit presence of $x_{i-1}$ in the state, allowing closed-form inversion during likelihood evaluation. Importantly, the Jacobian determinant of each step is a constant (a function of $\sigma$ only) and fully independent of the neural velocity field, yielding computationally trivial likelihood-ratio estimates.

Figure 2: Pipeline of GenPO++. The method achieves exact invertibility and efficient likelihood-ratio computation via reversible high-order solvers and solver-history states, contrasting the action-augmentation requirement of GenPO.

Exact Jacobian-Free Likelihood Ratios

The likelihood of taking an action under the flow policy is computed via the change-of-variable formula, but with GenPO++’s structure, the only determinant involved is constant, decoupled from $v_\theta$. Thus, likelihood ratios between policies (for PPO-style clipped objectives or KL control) reduce to ratios between simple base distributions (standard Gaussians), and gradients do not incur Jacobian-trace or double backward overhead. The augmented likelihood ratio aligns with the true action likelihood, as formalized by the consistency results in the paper.

Optimization and Theoretical Properties

This solver-history augmentation matches the high-order accuracy of the Adams–Bashforth (AB2) method for $\sigma=1$ and flexibly interpolates between Euler and AB2 for general $\sigma$. The authors rigorously analyze local truncation error, demonstrating empirical robustness to moderate $\sigma$ choices and stable reversibility essential for RL credit assignment.

Experimental Results

Large-Scale Simulated Control (IsaacLab)

GenPO++ is evaluated on eight diverse IsaacLab continuous-control benchmarks spanning classical locomotion, articulated manipulation, and whole-body control. It consistently outperforms or matches baseline methods, including Gaussian PPO, Flow Policy Optimization (FPO), Diffusion Policy PPO (DPPO), and classic GenPO, both in terms of sample efficiency and final return.

Figure 3: Learning curves across 8 IsaacLab benchmarks. GenPO++ shows stable, superior performance compared to PPO, FPO, and GenPO, confirming the practical impact of exact, efficient likelihood ratios.

Wall-clock time comparisons also confirm that GenPO++ achieves significant reductions in training overhead compared to GenPO and FPO due to its Jacobian-free updates, while maintaining the expressiveness of reversible flow models.

Manipulation Policy Fine-Tuning

GenPO++ demonstrates strong fine-tuning performance on three Robomimic manipulation tasks (Can, Box, Threading) using pretrained flow-matching policies. Across zero- and random-noise evaluation protocols, GenPO++ rapidly and stably improves task success, outperforming competing generative on-policy baselines.

Figure 4: Online fine-tuning results on Robomimic benchmarks show GenPO++ offers fast, stable, and superior policy improvement with both deterministic and stochastic rollouts.

Real-World Robotic Manipulation

GenPO++ is deployed on a 12-DoF dexterous hand (RobotEra Xhand) for in-hand nut-bolt manipulation, emphasizing sim-to-real transfer capability. The method achieves higher reward acquisition and greater final performance than PPO, successfully controlling the hand to loosen nuts across several geometries without simulation privileged information.

Figure 5: Video frames of GenPO++ real-world deployment: robust nut loosening on multiple bolt geometries. Policies were trained in simulation but generalize to physical hardware due to improved policy expressiveness and robust on-policy optimization.

Empirical Analysis

Ablation experiments show that GenPO++ is robust to moderate values of the solver-history coefficient $\sigma$. Memory and compute analysis reveals a favorable scaling profile compared to classical invertible flow policy methods, especially in high-dimensional action settings. Additional experiments also confirm sample-efficiency gains across various solver step numbers.

Implications and Future Directions

The main practical implication is that GenPO++ enables scalable, stable, and efficient use of flow-based generative models in on-policy RL, eliminating the core bottleneck of exact action-density computation. Notably, it avoids dummy-action overheads and works natively with any expressive pretrained generative policy—thereby facilitating robust fine-tuning and sim-to-real transfer.

Theoretically, this framework opens research directions toward adaptive solver-history management (e.g., dynamic $\sigma$ scheduling), application to more complex stochastic process models, and scaling to hierarchical or long-horizon autoregressive policies. The approach—embedding solver states instead of action augmentation—could inform invertibility designs for broader probabilistic inference tasks in RL and decision making.

Conclusion

GenPO++ advances generative policy optimization by providing a reversibility mechanism that is both exact and computationally efficient. Through solver-history state augmentation, GenPO++ achieves exact, Jacobian-free likelihood-ratio computation without inflating the action space or incurring costly neural Jacobian-trace operations. Empirical results—across simulated physics, policy fine-tuning, and real robotic tasks—consistently support the claim that exact reversible likelihood-ratio optimization improves both stability and final performance over prior art. Careful tuning of the solver coefficient $\sigma$0 remains an open question, but this work provides a robust and scalable template for deploying expressive generative policies in high-dimensional, real-world RL environments.