Expressive Policy Optimization (EXPO)

• Expressive Policy Optimization (EXPO) is a framework that decouples expressive policy models from value-based optimization to resolve gradient instability in reinforcement learning.
• It leverages techniques like two-policy decomposition and reversible flow models to maintain stability while capturing complex, multimodal behaviors.
• EXPO enables robust training in continuous control and language preference tasks, demonstrating improved sample efficiency and performance over traditional methods.

Expressive Policy Optimization (EXPO) refers to a family of optimization frameworks and algorithms for reinforcement learning (RL) and related settings that enable the full utilization of highly expressive, non-Gaussian policy models—such as diffusion, flow-matching, or other generative models—while maintaining stable, sample-efficient, and theoretically principled training dynamics. EXPO was developed to resolve the inherent instability and gradient propagation challenges that arise when deploying complex policy parameterizations in value-based or policy-gradient RL, particularly in online or offline-to-online learning regimes.

1. Motivation and Problem Setting

Classical RL policy optimization relies on tractable parametric families (e.g., unimodal Gaussians) for both analytic log-probabilities and efficient gradient propagation. However, real-world control, robotic manipulation, and language preference modeling increasingly demand policies with multi-modal, non-linear, or highly non-Gaussian action distributions. Generative models—such as continuous normalizing flows, diffusion processes, and score-based samplers—provide this expressivity, but at the cost of introducing deep, numerically stiff architectures where naive end-to-end value maximization leads to severe instability due to vanishing or exploding gradients, ill-conditioned objectives, or out-of-distribution policy updates (Dong et al., 10 Jul 2025, Doo et al., 13 May 2026, Hu et al., 5 Jun 2026).

EXPO addresses these issues by decoupling the representational flexibility of the expressive policy model from the value optimization process, thereby enabling stable and reliable training of arbitrarily rich policies even in deep denoising or ODE-based settings (Dong et al., 10 Jul 2025, Doo et al., 13 May 2026).

2. Core Methodologies and Algorithmic Structures

Two primary EXPO variants have emerged in the literature, reflecting the key settings in which expressive policies find use: RL in continuous control domains, and preference optimization for LLMs.

2.1 Two-Policy Decomposition for RL

The canonical RL EXPO framework, as instantiated in (Dong et al., 10 Jul 2025), partitions the policy architecture into two components:

• Expressive base policy ($\pi_\theta$): A high-dimensional, generative policy (e.g., diffusion or flow model), trained exclusively via imitation learning (e.g., score-matching or denoising loss). Gradients from the value function or Q-critic do not propagate through $\pi_\theta$.
• Lightweight edit policy ($\pi_\phi$): A low-dimensional, Gaussian (or similar) "edit" model that perturbs base actions locally to ascend the Q-value. This is trained via standard off-policy or entropy-regularized RL objectives.

Actions for both agent sampling and TD backup are selected on-the-fly by drawing base policy samples, editing them with $\pi_\phi$, then choosing the maximally-valued candidate under the current critic (“on-the-fly RL policy”):

$$\tilde{a}^* = \arg\max_{a \in \{a_i, a_i + \hat{a}_i\}_{i=1}^N} Q(s,a)$$

This decomposition isolates the large policy from brittle Q-gradients and localizes value maximization to a robust, well-conditioned problem in the edit space (Dong et al., 10 Jul 2025).

2.2 Flow-Based and Reversible Policy Optimization

In continuous flows, EXPO frameworks such as Q-Flow (Doo et al., 13 May 2026) and GenPO++ (Hu et al., 5 Jun 2026) address the Jacobian and value propagation problem by:

• Parameterizing $\pi_\theta(a|s)$ as the pushforward of a base noise $z \sim \mathcal{N}(0,I)$ through an ODE or sequence of invertible transformations.
• Formulating policy learning objectives that leverage deterministic flow dynamics without ever backpropagating through the numerical solver; instead, value gradients are propagated via intermediate value functions or specialized analytic likelihood-ratio constructions.

In Q-Flow (Doo et al., 13 May 2026), a “flow-consistent” intermediate value function $V^\pi(s,x_\tau,\tau)$ is solved for all latent flow states, and velocity field updates are carried out by local gradient matching:

$$\mathcal{L}_\pi(\theta) = \mathbb{E}\left[ \| v_\theta(x_\tau, \tau; s) - \mathrm{stopgrad}[v_{\mathrm{tar}}(s,x_\tau,\tau)] \|^2 \right]$$

This updates the expressive policy in a stable, non-truncated, and analytically justified manner.

GenPO++ (Hu et al., 5 Jun 2026) leverages reversible ODE solver structures to obtain exact, Jacobian-free likelihood ratios for on-policy updates, thus preserving both expressiveness and algorithmic tractability.

2.3 Explicit Preference Optimization in LLMs

In LLM preference modeling, EXPO is formulated as “Explicit Preference Optimization” to address RLHF without implicit reparameterized reward models (Hu et al., 9 Jun 2025). EXPO replaces DPO-style objectives with losses of the form:

$$\ell_{\mathrm{EXPO}}(\pi_\theta ; \pi_{\mathrm{ref}}, \lambda) = \underbrace{\ell_{\mathrm{sup}}(\pi_\theta)}_{\text{preference fit}} + \lambda \underbrace{\ell_{\mathrm{unsup}}(\pi_\theta, \pi_{\mathrm{ref}})}_{\text{KL regularizer}},$$

where the supervised term fits model preferences to the empirical preference data and the unsupervised term regularizes toward $\pi_\theta$0, yielding provable preservation and strong interpolation between $\pi_\theta$1 and $\pi_\theta$2.

3. Theoretical Properties and Guarantees

Across settings, EXPO-type algorithms deliver critical theoretical benefits:

• Stability: Decoupling expressive policy optimization from direct Q-backpropagation prevents destabilizing gradient flow through deep generative chains (Dong et al., 10 Jul 2025, Doo et al., 13 May 2026).
• Expressivity: No architectural simplifications or flow truncations are needed; full expressivity of the underlying generative model is retained, supporting complex multimodal behaviors (Doo et al., 13 May 2026, Hu et al., 5 Jun 2026, Huang et al., 13 Nov 2025).
• Sample Efficiency: The on-the-fly policy maximizes value efficiently, leveraging offline datasets and immediate improvement mirroring Q-value advancements (Dong et al., 10 Jul 2025).
• Exactness: With appropriate reversible structures, exact likelihood ratios can be computed efficiently for on-policy algorithms (e.g., PPO), without recourse to ELBO surrogates or dummy-variable augmentation (Hu et al., 5 Jun 2026).
• Regularization: In preference optimization, EXPO satisfies “preservation” and “strong interpolation,” ensuring that improvements on poor regions do not degrade performance where the reference policy is already optimal (Hu et al., 9 Jun 2025).

4. Empirical Evaluations and Benchmarks

EXPO instantiations have been evaluated extensively on both RL control and preference modeling tasks.

Method	Domain	Key Benchmarks	Notable Results
EXPO (Dong et al., 10 Jul 2025)	RL, continuous	AntMaze, Adroit, Robomimic, MimicGen	2–3× sample efficiency improvement; robust offline-to-online transfer
Q-Flow (Doo et al., 13 May 2026)	RL, flow policy	OGBench (50 tasks)	+10.6 points over prior SOTA (FQL); +23–31 pts on long-horizon tasks; seamless offline–online adaptation
GenPO++ (Hu et al., 5 Jun 2026)	RL, on-policy	IsaacLab locomotion/manip.; Robomimic; real-world robotics	Matches/outperforms FPO, GenPO, Gaussian PPO; major training efficiency gain (20 min vs. 132 min); robust sim-to-real transfer
UEPO (Huang et al., 13 Nov 2025)	RL, diffusion policy	D4RL (HalfCheetah, Hopper, Walker2d, Adroit)	+5.9% (locomotion), +12.4% (manipulation) over unified baselines
EXPO (Hu et al., 9 Jun 2025)	LLM preference	Anthropic HH, IMDb, bandit tests	State-of-the-art win rates (e.g., 61% vs. 58% for DPO on HH); provable interpolation

These results demonstrate that EXPO algorithms support stronger, more diverse behavior and robust adaptation without compromising learning stability, even as policy models become increasingly complex.

5. Extensions and Unified Generative Frameworks

EXPO's principles have been unified and extended in recent frameworks such as UEPO (Huang et al., 13 Nov 2025), which combines multi-seed diffusion policy ensembles, dynamics-aware divergence regularization, and diffusion-based data augmentation within a single offline-to-online RL pipeline. UEPO demonstrates efficient coverage of multimodal behavior without the need for multiple policy deployments per mode and provides robust generalization under distributional shift. This suggests the central applicability of EXPO-based approaches in unified, scalable robotic and control learning.

6. Comparative Analysis

A central distinction between EXPO and classical approaches lies in the treatment of “expressiveness vs. stability.” Previous attempts to optimize highly expressive policies either restricted policy complexity or resorted to unstable, high-variance value propagation through deep generative chains (Dong et al., 10 Jul 2025, Doo et al., 13 May 2026). EXPO instead leverages architectural or algorithmic decompositions, reversible solvers, or explicit regularization criteria to guarantee both stability and expressiveness.

Other EXPO strengths, such as immediate on-the-fly policy improvement with every critic update (in contrast to SARSA-like lag in traditional actor-critic RL), and provable preservation in preference optimization, further set it apart from earlier Q-learning or RLHF-alternatives (Dong et al., 10 Jul 2025, Hu et al., 9 Jun 2025).

7. Practical Implications and Open Directions

EXPO has demonstrated broad impact across decision-making domains:

• Facilitating the reliable deployment of pre-trained, expressive control policies in robotic systems as both offline initialization and online fine-tuning.
• Enabling preference learning in LLMs with direct control over regularization, interpretability, and preservation of desired behaviors.
• Reducing computational cost and achieving scalable ensemble behaviors without multiplying model parameters (Huang et al., 13 Nov 2025, Hu et al., 5 Jun 2026).

Future work may further explore advanced forms of expressivity (e.g., transformer-based policies), finer-grained dynamics-aware regularization, and integrations with large-scale model pretraining and transfer.

---

EXPO and its variants realize a principled approach to reinforcement learning and preference optimization with expressive policies. Through architectural decoupling, algorithmic innovation, and explicit regularization, EXPO achieves the dual aim of full representational capacity and practical, stable optimization in domains where prior methods suffered either from expressivity collapse or catastrophic instability (Dong et al., 10 Jul 2025, Doo et al., 13 May 2026, Hu et al., 5 Jun 2026, Huang et al., 13 Nov 2025, Hu et al., 9 Jun 2025).