---
title: Metric-Oriented Policy Optimization
url: https://www.emergentmind.com/topics/metric-oriented-policy-optimization
type: topic
---

# Metric-Oriented Policy Optimization

Searching arXiv for recent and foundational papers on metric-oriented policy optimization, including Wasserstein, trust-region, metric-learning, and Riemannian formulations.
Metric-oriented policy optimization denotes a family of reinforcement-learning formulations in which the choice of metric, divergence, or geometric structure is treated as a primary design element of the optimization problem rather than as a secondary implementation detail. Across recent work, this idea appears in several distinct but related forms: policy optimization as a Wasserstein gradient flow on probability measures over actions or parameters [1808.03030]; trust-region methods that replace Kullback–Leibler divergence with Wasserstein or Sinkhorn distances [2306.14133]; policy optimization on a Riemannian manifold induced by the \(2\)-Wasserstein geometry of action distributions and stationary state distributions [2604.14765]; parameter-space methods that learn a metric tensor and regularize the divergence of the policy-gradient vector field [2305.11017]; and offline goal-conditioned RL methods that learn a state metric whose distance ordering recovers optimal behavior [2402.10820]. A broader implication is that policy optimization can be organized around an explicit notion of distance—between policies, action distributions, occupancies, parameters, or latent states—and that this choice determines gradients, geodesics, Hessians, trust regions, and update stability.

## 1. Conceptual scope and historical formulations

A central conceptual shift in this literature is to stop treating a policy merely as a parameter vector \(\theta\) and instead treat it as a probability measure or a measurable map into a space of probability measures [1808.03030]. In the indirect formulation of Wasserstein gradient flows, one optimizes a distribution \(\mu(\theta)\) over parameters rather than a single parameter vector; in the direct formulation, one optimizes \(\pi(\cdot\mid s)\) as a probability distribution over actions for each state [1808.03030]. In both cases, the search space is a Wasserstein space of probability measures equipped with the \(2\)-Wasserstein distance \(W_2\), and optimization is defined as gradient flow in that metric space [1808.03030].

A second line of work formulates metric-oriented policy optimization through trust-region design. Trust-region methods based on Kullback–Leibler divergence are described as pervasive in policy optimization, but the Wasserstein and Sinkhorn extensions replace the usual KL ball with a Wasserstein or entropically regularized optimal-transport ball over action distributions [2306.14133]. In this view, the trust region is defined not just in probability space but in a metric space of actions, using a ground cost \(d(a,a')\) that measures action similarity [2306.14133].

A third line of work pushes the metric orientation further by endowing the policy space itself with a Riemannian structure. Policies are modeled as measurable maps
\[
\pi: S \to P_2(A),
\]
where \(P_2(A)\) is the space of probability measures on the action space with finite second moment, equipped with the \(2\)-Wasserstein distance \(W_2\) [2604.14765]. The policy manifold is then given a Riemannian metric obtained by averaging Wasserstein tangent-space inner products across states, weighted by the stationary state distribution \(\mu_\pi\) induced by the policy [2604.14765].

Other papers instantiate the same theme in different spaces. OMPO uses a divergence between transition occupancies as the object that regularizes policy learning under policy or dynamics shifts [2405.19080]. MetricRL uses metric learning over state embeddings so that the learned distance is monotonic with shortest-path distance to goals in deterministic, sparse-reward offline RL [2402.10820]. Deep Metric Tensor Regularized Policy Gradient learns a metric tensor \(g_{ab}\) over policy parameters and uses it to smooth the policy-gradient vector field by minimizing its absolute divergence in the induced Riemannian manifold [2305.11017]. Orthogonalized Policy Optimization, in RLHF rather than standard control, explicitly separates sampling geometry from optimization geometry, using an \(\alpha\)-divergence-inspired sampling weight together with a Bregman-divergence-based value metric [2601.12415]. This suggests that “metric-oriented” refers not to a single algorithmic family but to a design principle: choose the geometry first, then define optimization relative to that geometry.

## 2. Wasserstein geometry on policies and policy distributions

The Wasserstein formulation is the most explicit geometric realization of metric-oriented policy optimization. In “Policy Optimization as Wasserstein Gradient Flows,” the policy optimization problem is placed into the space of probability measures and interpreted as Wasserstein gradient flow [1808.03030]. For indirect policy learning, the energy functional is
\[
F(\mu) = KL(\mu\|p_\theta),
\]
where \(p_\theta(\theta)\propto \exp(J(\pi_\theta)/\alpha)\); for direct policy learning, at each state \(s\),
\[
F_s(\pi) = -\int Q(a,s)\,\pi(a\mid s)\,da + \int \pi(a\mid s)\log\pi(a\mid s)\,da
= KL\big(\pi(\cdot\mid s)\,\|\,p_{s,\pi}\big),
\]
with \(p_{s,\pi}(a\mid s)\propto e^{Q(a,s)}\) [1808.03030]. The associated Wasserstein gradient flow is expressed by the continuity equation
\[
\partial_\tau \mu_\tau
=
\nabla\cdot\Big(\mu_\tau \nabla \frac{\delta F}{\delta \mu_\tau}(\mu_\tau)\Big),
\]
and the JKO scheme
\[
\mu_{k+1}^{(h)} \in \arg\min_{\mu} \Big\{F(\mu) + \frac{1}{2h}W_2^2(\mu,\mu_k^{(h)})\Big\}
\]
provides the discrete-time optimization principle [1808.03030].

The 2026 Wasserstein formulation generalizes this to a fully Riemannian policy manifold. The action space \(A\) is a Polish metric space, and
\[
W_2^2(\mu,\nu) := \inf_{\gamma\in\Gamma(\mu,\nu)} \int_{A\times A} d_A(x,y)^2\,d\gamma(x,y)
\]
defines the \(2\)-Wasserstein metric on \(P_2(A)\) [2604.14765]. Policies are measurable fields of action distributions,
\[
\Pi := L^0(S;P(A)),
\]
and the environment induces a stationary measure \(\mu_\pi\) through the Markov kernel \(P^\pi\) [2604.14765]. The policy-space Riemannian metric is then
\[
\langle \xi, \eta \rangle_\pi
:=
\int_S \int_A
\langle v^\xi(s,a), v^\eta(s,a)\rangle\,
d\pi(s)(a)\,d\mu_\pi(s),
\]
where \(v^\xi,v^\eta\) are tangent velocity fields in action space [2604.14765].

This metric is “local” in actions and “global” in states: locally one uses the Wasserstein metric on action distributions; globally one averages over states according to the invariant visitation frequencies \(\mu_\pi\) [2604.14765]. The framework then defines geodesics, tangent spaces, gradients, and Hessians intrinsically on policy space. In particular, for long-run average cost
\[
J(\pi)
:=
\int_S \int_A c(s,a)\,d\pi(s)(a)\,d\mu_\pi(s),
\]
the Riemannian gradient is
\[
\mathrm{grad}_\pi J(s,a) = \nabla_a Q^\pi(s,a),
\]
where \(Q^\pi\) is defined through a Poisson/Bellman equation [2604.14765]. The corresponding Wasserstein gradient flow is
\[
\partial_t \pi_t(s) + \mathrm{div}_a\bigl(\pi_t(s)(-\nabla_a Q^{\pi_t}(s,\cdot))\bigr)=0,
\]
so policy improvement becomes transport of probability mass in action space along \(-\nabla_a Q^\pi\) [2604.14765].

This directly contrasts with KL/Fisher approaches. The Wasserstein metric is derived from the metric structure of the action space and remains finite and meaningful when supports are disjoint, whereas KL can be infinite or undefined under support mismatch [2604.14765, 1808.03030]. A plausible implication is that Wasserstein geometry is especially aligned with continuous control tasks in which Euclidean proximity of actions carries physical meaning.

## 3. Trust regions defined by Wasserstein, Sinkhorn, and related metrics

Metric-oriented policy optimization also appears as a redesign of trust-region constraints. In Wasserstein Policy Optimization and Sinkhorn Policy Optimization, the local surrogate improvement objective is constrained by a Wasserstein or Sinkhorn distance between successive action distributions rather than by KL divergence [2306.14133]. For a discrete action set with ground cost \(D_{ij}=d(a_i,a_j)\), the per-state WPO subproblem is
\[
\max_{\pi(\cdot\mid s)} \sum_a \pi(a\mid s)A^{\pi_k}(s,a)
\quad
\text{s.t.}\quad
W_1\big(\pi(\cdot\mid s),\pi_k(\cdot\mid s)\big)\le \epsilon_s.
\]
The corresponding Lagrangian is
\[
\mathcal{L}_s(\pi,\beta_k)
=
\sum_a \pi(a\mid s)A^{\pi_k}(s,a)
-
\beta_k\,W_1\big(\pi(\cdot\mid s),\pi_k(\cdot\mid s)\big),
\]
and the closed-form update transports the probability mass of each old action \(a_j\) to the action \(a_i\) maximizing
\[
A^{\pi_k}(s,a_i)-\beta_k D_{ij}
\]
[2306.14133]. Thus each piece of mass is redistributed according to an advantage–distance tradeoff rather than reweighted independently by an exponential function of advantage.

The Sinkhorn version adds entropic regularization to the transport problem. The update becomes soft rather than hard, with transport weights of the form
\[
f_s^k(i,j)\propto \exp\Big(\frac{1}{\varepsilon}(A^{\pi_k}(s,a_i)-\beta_k D_{ij})\Big),
\]
again followed by mass aggregation
\[
\pi_{k+1}(a_i\mid s)=\sum_j \pi_k(a_j\mid s)f_s^k(i,j)
\]
[2306.14133]. The paper proves that WPO guarantees monotonic performance improvement, that SPO converges to WPO as the entropic regularizer vanishes, and that with a decaying Lagrangian multiplier both converge to a globally optimal policy in finite MDPs under exact expectations [2306.14133].

The continuous-time continuous-space RL paper gives an analogous metric-oriented reading of TRPO/PPO in a diffusion setting. It derives performance-difference and local-approximation formulas from a discounted occupation-time measure and extends them to continuous-time PG and TRPO/PPO methods [2305.18901]. Although the detailed trust-region machinery differs from the discrete setting, the role of the metric is similar: a local approximation of \(\eta(\pi_{\text{new}})-\eta(\pi)\) is combined with a policy-distance control term.

CIM-PPO modifies PPO at the metric level by replacing KL divergence with the Liu-Correntropy Induced Metric, a symmetric, bounded kernel-induced metric in an RKHS [2110.10522]. The PPO-CIM surrogate is
\[
\mathcal{L}_\pi^{\text{CIM}}
=
\mathbb{E}\left[
\frac{\pi_{\text{new}}(a\mid s)}{\pi_{\text{old}}(a\mid s)}\hat A_\pi(s,a)
-
\alpha\,\text{CIM}_\sigma\big(\pi_{\text{old}}(\cdot\mid s),\pi_{\text{new}}(\cdot\mid s)\big)
\right],
\]
and the motivation is explicit: asymmetry in KL divergence can misguide the PPO-KL surrogate, especially for high-dimensional Gaussian policies with differing variances [2110.10522]. The paper reports better empirical performance than both PPO-KL and PPO-Clip on most of its MuJoCo-style tests [2110.10522].

A related but distinct unification appears in “A Parametric Class of Approximate Gradient Updates for Policy Optimization,” which organizes many updates around two scalar signals,
\[
\Delta_R = \hat T(s,a)-q_\theta(s,a),\qquad
\Delta_O=\log\frac{\pi_\theta(a\mid s)}{\pi_b(a\mid s)},
\]
and a scaling function \(f(\Delta_O,\Delta_R)\) that modulates the update magnitude [2206.08499]. PPO corresponds to a specific clipped scale function,
\[
f(x,y)=e^x y\,\tau_\epsilon(x,y),
\]
while a broader family \(f_{MLA(\alpha_o,\alpha_r)}\) interpolates between value-based, policy-gradient, self-imitation, and PPO-like behaviors [2206.08499]. This suggests a metric-oriented reading in which the scaling function defines a sample-wise optimization geometry.

## 4. Metrics on occupancies, latent states, and parameter space

Not all metric-oriented formulations operate directly on policy distributions. OMPO identifies transition occupancy matching as a unified principle for RL under policy shifts, nonstationary dynamics, and domain adaptation [2405.19080]. The key object is the transition occupancy
\[
p^{\pi,T}(s,a,s')
=
(1-\gamma)\sum_{t=0}^{\infty}\gamma^t
\Pr(s_t=s,a_t=a,s_{t+1}=s'\mid \pi,T),
\]
and the surrogate learning objective explicitly penalizes divergence between current and replay occupancies [2405.19080]. After dual reformulation, the policy is trained with a critic and a discriminator estimating a transition-occupancy log-ratio
\[
R(s,a,s')=\log\frac{p^{\pi,T}(s,a,s')}{p^{\tilde\pi,\tilde T}(s,a,s')},
\]
which functions as a learned metric on transition distributions [2405.19080]. This is metric-oriented because the policy is not optimized only for reward: it is optimized for reward minus occupancy discrepancy.

MetricRL does something analogous in state space for goal-conditioned offline RL. Under deterministic transitions, invertible actions, sparse rewards, and connectivity assumptions, the optimal value is tied to graph shortest-path distance to the goal [2402.10820]. The method learns an embedding
\[
\phi_\theta:S\to Z\subseteq \mathbb{R}^n
\]
with Euclidean metric \(d_Z\), and introduces the property of distance monotonicity:
\[
d_S(s_1,s_3)\le d_S(s_2,s_3)
\implies
d_Z(\phi(s_1),\phi(s_3))\le d_Z(\phi(s_2),\phi(s_3)).
\]
If this property holds, greedy control based on latent distance to the goal recovers the optimal policy [2402.10820]. The approximate value function is
\[
\tilde V(s)=\gamma^{d_Z(\phi(s),\phi(s_g))}r_g,
\]
and the actor is trained by weighting behavior-cloning updates by \(\tilde V(s')-\tilde V(s)\), so actions that move closer to the goal in latent metric space are reinforced [2402.10820].

Deep Metric Tensor Regularized Policy Gradient moves the metric orientation to policy parameter space. Standard policy gradient treats \(\theta\in\mathbb{R}^n\) as Euclidean; the paper instead learns a metric tensor \(g_{ab}(\theta)\) over parameter space and defines the policy-gradient vector field
\[
J^a(\theta)=g^{ab}(\theta)\nabla_b J(\theta),
\]
then minimizes the squared divergence
\[
\min_\phi [\operatorname{Div}(J^a)|_\theta]^2
\]
with respect to the metric-network parameters \(\phi\) [2305.11017]. The chosen metric has the structured form
\[
G_\theta = I_n + \vec u(\theta)\vec u(\theta)^\top,
\]
so the inverse metric yields a preconditioned gradient
\[
\vec J|_\theta
=
G_\theta^{-1}\nabla_\theta J(\theta)
=
\nabla_\theta J(\theta)
-
\frac{\vec u(\theta)^\top \nabla_\theta J(\theta)}{1+\vec u(\theta)^\top \vec u(\theta)}\,\vec u(\theta)
\]
[2305.11017]. The paper then derives a geodesic-regularized gradient and reports that the metric-tensor regularization reduces the absolute divergence of the policy-gradient vector field and improves performance on several benchmarks [2305.11017].

Orthogonalized Policy Optimization introduces another decomposition: sampling geometry determines which samples dominate the gradient signal, while optimization geometry determines how deviations are penalized [2601.12415]. The general objective is expressed as
\[
\mathcal{L}(\theta)
=
\mathbb{E}_{y\sim\pi_{\text{sample}}}\left[
w_\alpha(y)\cdot \mathcal{D}_\phi(\text{deviation}_\theta(y),\text{target}(y))
\right],
\]
and OPO instantiates this with \(\alpha\)-weighted importance sampling and a Pearson \(\chi^2\)-induced quadratic regularization in ratio coordinates [2601.12415]. This makes the metric choice explicit and separates it from the sample-weighting mechanism.

## 5. Theoretical properties: convexity, Hessians, monotonicity, and well-posedness

One recurring claim in this literature is that choosing the right metric can reveal hidden convexity or better conditioning. In policy optimization as Wasserstein gradient flow, the JKO step
\[
\mu_{k+1}^{(h)}
=
\arg\min_{\mu}
\Big\{F(\mu)+\frac{1}{2h}W_2^2(\mu,\mu_k^{(h)})\Big\}
\]
is convex over measures for the Itô-type energies under the stated assumptions, even though the corresponding parameter optimization problem is highly nonconvex [1808.03030]. The paper formalizes convergence of the flow to a target distribution induced by return or soft \(Q\)-values [1808.03030].

The 2026 Wasserstein manifold paper develops a second-order theory. Along a constant-speed geodesic \(\pi_t\) with initial velocity \(\xi\), the Hessian is
\[
\mathrm{Hess}_\pi J(\xi,\xi)
=
\mathbb{E}_{s\sim\mu_\pi,a\sim\pi}
\big[\mathrm{Hess}_a Q^\pi(s,a)(v_0,v_0)\big]
+
2\,\mathbb{E}_{s,a}
\big[\langle \nabla_a Q^{\Psi^\pi[v_0]}(s,a), v_0(s,a)\rangle\big],
\]
with the second term capturing Poisson-feedback curvature induced by dynamics [2604.14765]. In the special case where transitions do not depend on actions, the nonlocal term vanishes and geodesic convexity of \(J\) is equivalent to displacement convexity of the action-cost \(c(s,\cdot)\) for almost every state [2604.14765].

Trust-region papers supply monotonic-improvement guarantees. WPO derives a TRPO-style lower bound and shows that in the exact-advantage setting WPO produces a non-decreasing sequence of returns [2306.14133]. SPO converges to WPO as entropic regularization vanishes [2306.14133]. CIM-PPO proves that under suitable kernel conditions the new policy remains within a trust region defined by the kernel-induced metric, while also arguing that the metric is computationally cheaper than KL-based policy gradients [2110.10522].

The metric-tensor paper focuses on divergence and Hessian trace. In Euclidean coordinates, the divergence of the policy-gradient vector field is the trace of the Hessian; by learning a Riemannian metric \(g_{ab}\) and reducing \(\operatorname{Div}(J^a)\), the algorithm attempts to smooth the gradient field in the learned manifold [2305.11017]. This suggests that metric design can be used as a second-order regularizer even when the update remains first order in implementation.

A different notion of well-posedness appears in “Compactly Restrictable Metric Policy Optimization Problems,” which studies deterministic MDPs with metric state and action spaces [2207.05850]. The paper defines a metric policy optimization problem as an MDP on separable metric state and action spaces together with a policy class admitting a partial order, and shows that a compact forward-invariant restriction yields a well-posed Bellman operator, existence of optimal policies, and value-iteration convergence [2207.05850]. In this usage, “metric” refers primarily to the topology and compactness structure required for dynamic programming. This suggests a broader interpretation of metric-oriented policy optimization that includes well-posedness results for continuous-control problems on metric spaces.

## 6. Practical algorithms, implementations, and broader implications

Several papers translate the geometric ideas into practical algorithms. The Wasserstein-gradient-flow paper uses particle approximations and JKO discretization, giving IP-WGF for parameter distributions and DP-WGF / DP-WGF-V for direct action-distribution updates [1808.03030]. The Riemannian Wasserstein paper gives both low-dimensional particle methods and high-dimensional deterministic-policy implementations, where the pullback metric on parameters becomes
\[
M(\theta)
=
\int_S [\nabla_\theta \mu_\theta(s)]^\top \nabla_\theta \mu_\theta(s)\,d\mu^\pi(s),
\]
and the parameter update solves
\[
M(\theta)v=\nabla_\theta J
\]
via conjugate gradient [2604.14765].

MetricRL is practical in offline settings because it never estimates a \(Q\)-function for out-of-distribution actions; the critic is induced by metric distance, and the actor only reweights observed actions [2402.10820]. OMPO remains actor–critic in appearance but uses a distribution discriminator and a small local buffer to estimate transition-occupancy discrepancy under changing policies and dynamics [2405.19080]. The metric-tensor method augments SAC and TD3 with an inner loop that trains the metric tensor network before each policy update [2305.11017]. CIM-PPO keeps the overall PPO structure and simply replaces the KL penalty by a kernel metric, using triangular or biweight kernels in the reported experiments [2110.10522]. The parametric family of approximate gradient updates can be dropped into PPO-style code by replacing the standard clipped scale with \(f_{MLA(\alpha_o,\alpha_r)}\tau_\epsilon\) [2206.08499].

The literature also contains metric-oriented formulations outside standard control RL. MePoly is framed around maximum-entropy stochastic control and explicit information-theoretic metrics such as Shannon entropy and KL divergence, using a polynomial energy-based model to keep \(\log\pi\), entropy, and KL tractable for highly multimodal policies [2602.17832]. Orthogonalized Policy Optimization treats the divergence or Bregman geometry itself as a first-class design variable in RLHF [2601.12415]. A plausible implication is that metric-oriented policy optimization is becoming a general design language for both reinforcement learning and alignment.

There are also objective cautions. The paper on Metric Aggregation Divergence argues that in agent-based model plus multi-objective evolutionary pipelines, different stages can silently use inconsistent implementations of the same metric, leading to different “champion” policies and different statistical conclusions [2606.29038]. Its proposed remedy is a metric contract: a single shared callable enforced across optimization, post-processing, and inference [2606.29038]. Although this work is outside standard policy-gradient RL, it highlights that metric-oriented optimization is only as reliable as the consistency of the metric across the pipeline.

Overall, the research record supports a common conclusion. Metric-oriented policy optimization is not simply “policy optimization with a different regularizer.” It is a perspective in which the geometry of policy space, action space, occupancy space, latent state space, or parameter space is made explicit and then used to define the optimization problem itself. Depending on the chosen metric, this yields Wasserstein gradient flows [1808.03030, 2604.14765], Wasserstein/Sinkhorn trust regions [2306.14133], kernel-metric PPO variants [2110.10522], occupancy-matching objectives [2405.19080], latent-metric offline RL [2402.10820], learned Riemannian parameter manifolds [2305.11017], or explicit decompositions of sampling and optimization geometry [2601.12415]. This suggests that future work will continue to treat metric choice not as a technical detail, but as one of the central modeling decisions in policy optimization.

Source: https://www.emergentmind.com/topics/metric-oriented-policy-optimization