Relative Policy-Transition Optimization (RPTO)

• RPTO is a unified framework that decomposes transfer loss into dynamics-induced and policy-induced gaps via the value relativity lemma.
• Relative Policy Optimization (RPO) improves target returns by adjusting the policy using sampled trajectories and entropy regularization, ensuring stability even with significant dynamics differences.
• Relative Transition Optimization (RTO) refines the source transition model to closely match target dynamics, enabling efficient and reliable policy transfer in continuous-control tasks.

Relative Policy-Transition Optimization (RPTO) is a principled framework for fast policy transfer between Markov Decision Processes (MDPs) with shared state and action spaces but differing transition dynamics. It operates by minimizing the "relativity gap"—the difference in expected cumulative returns when policies and environment models are swapped between source and target domains. RPTO unifies relative policy and transition optimization in a closed-loop process, enabling a policy to adapt efficiently in a new environment by simultaneously updating both the policy and an intermediate model of the environment, grounded in explicit theoretical guarantees derived from value-function decompositions (Xu et al., 2022).

1. Value Relativity Lemma and the Relativity Gap

The central concept underlying RPTO is the value relativity lemma, which formalizes the decomposition of the difference—i.e., the relativity gap—between two cumulative expected returns, each computed under possibly different dynamics and policies. Let $\mathcal{E}$ and $\mathcal{E}'$ be two MDPs sharing state space $\mathcal{S}$, action space $\mathcal{A}$, and reward function $r(s,a,s')$, but with transition dynamics $P(\cdot|s,a)$ for the source, and $P'(\cdot|s,a)$ for the target. If $J(P,\pi)$ denotes the expected discounted return under policy $\pi$ and dynamics $P$, the relativity gap is

$\mathcal{E}'$0

The lemma yields an exact decomposition:

$\mathcal{E}'$1

Here, $\mathcal{E}'$2 is the dynamics-induced gap,

$\mathcal{E}'$3

and $\mathcal{E}'$4 is the policy-induced gap:

$\mathcal{E}'$5

This result allows for targeted minimization of transfer loss attributable to dynamics and policy discrepancies.

2. Relative Policy Optimization (RPO)

Relative Policy Optimization addresses the dynamics-induced gap by optimizing a policy evaluated in the source environment for maximal return in the target environment. Given a pre-trained policy $\mathcal{E}'$6 in the source dynamics $\mathcal{E}'$7 and a target dynamics $\mathcal{E}'$8, RPO aims to maximize the return $\mathcal{E}'$9.

A key result is a lower bound for the dynamics gap:

$\mathcal{S}$0

With

$\mathcal{S}$1

and $\mathcal{S}$2 scaling with the maximum total-variation distance between $\mathcal{S}$3 and $\mathcal{S}$4 and between $\mathcal{S}$5 and $\mathcal{S}$6.

In practice, RPO replaces the expectation with sampled trajectories and incorporates entropy regularization as in Soft Actor-Critic (SAC):

$\mathcal{S}$7

where $\mathcal{S}$8 is trained on source data.

3. Relative Transition Optimization (RTO)

Relative Transition Optimization focuses on learning a parameterized source transition model $\mathcal{S}$9 that approximates the target dynamics $\mathcal{A}$0 for a fixed policy $\mathcal{A}$1. The transfer objective is to minimize the absolute dynamics gap $\mathcal{A}$2. The key supervised learning surrogate becomes:

$\mathcal{A}$3

with weighting function

$\mathcal{A}$4

Unweighted loss ($\mathcal{A}$5) recovers classical model-fitting. The objective induces the source model to better align with target transitions, conditioned on their importance for expected long-term value under $\mathcal{A}$6.

4. Algorithmic Structure and Closed-Loop RPTO

RPTO integrates RPO and RTO into a unified algorithmic loop, enabling simultaneous adaptation of both policy and source-model transitions. The key steps are as follows:

Step	Description (Algorithmic Stage)	Buffer Use
a	Run $\mathcal{A}$7 in $\mathcal{A}$8	Store data in $\mathcal{A}$9
b	Run $r(s,a,s')$0 in $r(s,a,s')$1	Store data in $r(s,a,s')$2
c	Update $r(s,a,s')$3 on $r(s,a,s')$4	Model evaluation
d	Update policy $r(s,a,s')$5 via RPO using $r(s,a,s')$6	Policy adaptation
e	Update dynamics $r(s,a,s')$7 via RTO using $r(s,a,s')$8	Model adaptation

The closed loop continues until convergence, with each component minimizing its explicit contribution to the relativity gap.

5. Theoretical Guarantees

RPTO's theoretical foundations comprise:

• Value Relativity Lemma: Exact decomposition of transfer loss into dynamics- and policy-induced components.
• RPO Lower Bound: Guarantees on the approximation tightness for the improvement in return on the target environment as a function of policy and dynamics discrepancies.
• RTO Bound: Provable controls of model mismatch, showing that minimizing the model's weighted supervised loss suffices to proxy the impact on expected return.

The error terms in all bounds scale with the total-variation distances between dynamics and policies, the discount factor, and reward maxima. Explicit sample complexity guarantees are not provided, but all bounds reveal how gaps propagate to transfer performance.

6. Empirical Study in MuJoCo Environments

RPTO was evaluated on MuJoCo continuous-control tasks—Ant, Hopper, HalfCheetah, Walker2d, and Swimmer—with source environments given by standard Gym settings and target environments perturbed by $r(s,a,s')$9 in joint length and friction. For each task, four target variants were constructed and run over 16 random seeds.

Model architectures used included:

• Dynamics: seven-member ensembles of probabilistic neural networks (four layers of 400 units, Gaussian output for $P(\cdot|s,a)$0, model horizon = 1, replay ratio = 1).
• Policy/Q-value: two-layer networks with 256 units (SAC-like architectures).

Baselines included SAC-warm and TRPO-warm (source-policy warm starts), MBPO, SLBO, PTP, and PDML (also warm-started).

RPTO consistently achieved faster ascent in target return and reached higher final performance than all baselines. RPO in isolation failed when the dynamics gap $P(\cdot|s,a)$1 was large, highlighting the necessity of the transition optimization component. Conversely, RTO alone improved the model but did not yield performant policies without RPO. The combined approach leverages a smooth curriculum of intermediate models, enabling the policy to adapt through a continuum bridging the source and target MDPs.

A didactic CartPole experiment further illustrated that RPO alone transferred successfully only for small pole length changes, RTO allowed recovery of the correct length via model minimization, and RPTO enabled the fastest transfer even for large environment discrepancies.

7. Significance and Implications

RPTO establishes a unified framework for policy transfer that explicitly accounts for and minimizes each component of return discrepancy between environments. Its algorithmic structure and theoretical results clarify the interplay between policy and model adaptation and indicate the limitations of single-component adaptation in large dynamics gaps. The empirical results on diverse continuous-control benchmarks validate the framework’s efficacy for rapidly transferring policies from inexpensive simulators to costly or perturbed real-world settings. A plausible implication is that such closed-loop approaches could generalize to broader domains of reinforcement learning requiring robust, reliable transfer across significant domain shifts (Xu et al., 2022).