Trust-PCL Off-Policy RL Algorithm

• Trust-PCL is an off-policy reinforcement learning algorithm that uses KL-regularization and pathwise consistency to balance reward optimization and policy similarity.
• It minimizes a squared-error loss over full trajectories, enabling off-policy updates that achieve trust-region-like stability and high sample efficiency.
• Its equivalence to Relative Trajectory Balance (RTB) connects RL fine-tuning with generative modeling, enhancing energy-based marginal matching.

Trust-PCL is an off-policy reinforcement learning (RL) algorithm that optimizes a KL-regularized objective via squared-error minimization of a pathwise consistency loss, unifying trust-region policy optimization (TRPO-style stability) with high sample efficiency. Trust-PCL emerged as an extension of maximum-entropy RL frameworks, introducing a relative-entropy (KL) penalty to enforce closeness to a reference policy and enable off-policy learning, with guarantees rooted in soft Bellman fixed-point equations (Nachum et al., 2017). Recent research has established an exact equivalence between Trust-PCL and the Relative Trajectory Balance (RTB) objective proposed for Generative Flow Networks (GFlowNets), further situating Trust-PCL at the intersection of generative modeling, RL fine-tuning, and KL-regularized objectives (Deleu et al., 1 Sep 2025).

1. KL-Regularized RL Objective and Pathwise Consistency

Trust-PCL arises from a KL-regularized RL objective designed to balance reward maximization against policy proximity to a reference ("prior") policy. In a finite-horizon Markov Decision Process (MDP) $M=(S\cup\{\bot\},A,P,r)$, the goal is to solve

$$\pi^* = \arg\max_\pi\;\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^T r(s_t,s_{t+1}) - \alpha\,\mathrm{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{prior}}(\cdot|s_t))\right],$$

where $\alpha>0$ controls the reward/KL tradeoff. The soft-optimal policy is given by

$$\pi^*(s'|s)\;\propto\;\pi_{\mathrm{prior}}(s'|s)\; \exp\left\{\frac{Q^*_{\mathrm{soft}}(s,s')-V^*_{\mathrm{soft}}(s)}{\alpha}\right\},$$

with associated soft Bellman equations

$$Q^*_{\mathrm{soft}}(s,s') = r(s,s') + V^*_{\mathrm{soft}}(s'),\qquad V^*_{\mathrm{soft}}(s) = \alpha \log \sum_{s'} \pi_{\mathrm{prior}}(s'|s) \exp\left(\frac{Q^*_{\mathrm{soft}}(s,s')}{\alpha}\right).$$

A crucial insight underlying Trust-PCL is that $(\pi^*, V^*)$ satisfy a multi-step pathwise consistency: for any sampled trajectory, a Bellman-like consistency relation holds at all sub-intervals, providing the foundation for off-policy updates (Nachum et al., 2017, Deleu et al., 1 Sep 2025).

2. Trust-PCL Squared-Error Loss and Algorithm

Trust-PCL operationalizes the KL-regularized objective by defining a per-trajectory residual and minimizing its squared expectation over off-policy data. Given policy parameters $\phi$ and value parameters $\psi$, for a trajectory $\tau = (s_0, ..., s_T, \bot)$,

$$\Delta_T(\tau; \phi, \psi) = -V^\psi(s_0) + \sum_{t=0}^T r(s_t, s_{t+1}) + \alpha \sum_{t=0}^T \log \frac{\pi_{\mathrm{prior}}(s_{t+1}|s_t)}{\pi_\phi(s_{t+1}|s_t)}$$

The learning objective is

$$\pi^* = \arg\max_\pi\;\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^T r(s_t,s_{t+1}) - \alpha\,\mathrm{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{prior}}(\cdot|s_t))\right],$$0

where $$\pi^* = \arg\max_\pi\;\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^T r(s_t,s_{t+1}) - \alpha\,\mathrm{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{prior}}(\cdot|s_t))\right],$$1 denotes the behavior policy (typically using a replay buffer) (Deleu et al., 1 Sep 2025). In every training iteration, a minibatch of trajectories is sampled, $$\pi^* = \arg\max_\pi\;\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^T r(s_t,s_{t+1}) - \alpha\,\mathrm{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{prior}}(\cdot|s_t))\right],$$2 computed for each, and stochastic gradients taken with respect to both policy and value parameters. The method does not require explicit discounting for finite-horizon tasks ($$\pi^* = \arg\max_\pi\;\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^T r(s_t,s_{t+1}) - \alpha\,\mathrm{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{prior}}(\cdot|s_t))\right],$$3).

3. Connection to Relative Trajectory Balance (RTB)

RTB, introduced for GFlowNets but equivalent to Trust-PCL under reparameterization, defines its residual for a trajectory as

$$\pi^* = \arg\max_\pi\;\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^T r(s_t,s_{t+1}) - \alpha\,\mathrm{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{prior}}(\cdot|s_t))\right],$$4

with loss

$$\pi^* = \arg\max_\pi\;\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^T r(s_t,s_{t+1}) - \alpha\,\mathrm{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{prior}}(\cdot|s_t))\right],$$5

By identifying the per-step policy $$\pi^* = \arg\max_\pi\;\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^T r(s_t,s_{t+1}) - \alpha\,\mathrm{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{prior}}(\cdot|s_t))\right],$$6 with $$\pi^* = \arg\max_\pi\;\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^T r(s_t,s_{t+1}) - \alpha\,\mathrm{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{prior}}(\cdot|s_t))\right],$$7, value $$\pi^* = \arg\max_\pi\;\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^T r(s_t,s_{t+1}) - \alpha\,\mathrm{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{prior}}(\cdot|s_t))\right],$$8, and outcome reward $$\pi^* = \arg\max_\pi\;\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^T r(s_t,s_{t+1}) - \alpha\,\mathrm{KL}(\pi(\cdot|s_t)\;\|\;\pi_{\mathrm{prior}}(\cdot|s_t))\right],$$9 at the terminal step, one obtains $\alpha>0$0 and $\alpha>0$1 (Proposition 1, (Deleu et al., 1 Sep 2025)). Thus, Trust-PCL and RTB are the same algorithm up to a scale factor and parameterization of the value function.

4. Practical Implementation and Hyperparameters

Key hyperparameters and implementation protocols include:

• $\alpha>0$2 (KL-regularization weight/temperature, equivalent to $\alpha>0$3 in RTB);
• $\alpha>0$4 (learning rates for policy and value networks);
• Minibatch size $\alpha>0$5, replay buffer size, trajectory rollout length $\alpha>0$6;
• Behavior policy $\alpha>0$7 (commonly an $\alpha>0$8-greedy or delayed copy of $\alpha>0$9);
• Optional annealing schedule for $$\pi^*(s'|s)\;\propto\;\pi_{\mathrm{prior}}(s'|s)\; \exp\left\{\frac{Q^*_{\mathrm{soft}}(s,s')-V^*_{\mathrm{soft}}(s)}{\alpha}\right\},$$0, or stepwise clipping of policy updates (cf. PPO);
• Importance sampling or self-normalized IS if $$\pi^*(s'|s)\;\propto\;\pi_{\mathrm{prior}}(s'|s)\; \exp\left\{\frac{Q^*_{\mathrm{soft}}(s,s')-V^*_{\mathrm{soft}}(s)}{\alpha}\right\},$$1;
• Baseline subtraction is automatically handled via $$\pi^*(s'|s)\;\propto\;\pi_{\mathrm{prior}}(s'|s)\; \exp\left\{\frac{Q^*_{\mathrm{soft}}(s,s')-V^*_{\mathrm{soft}}(s)}{\alpha}\right\},$$2;
• For finite horizon, set discount $$\pi^*(s'|s)\;\propto\;\pi_{\mathrm{prior}}(s'|s)\; \exp\left\{\frac{Q^*_{\mathrm{soft}}(s,s')-V^*_{\mathrm{soft}}(s)}{\alpha}\right\},$$3 (Deleu et al., 1 Sep 2025).

A typical Trust-PCL training loop involves rolling out $$\pi^*(s'|s)\;\propto\;\pi_{\mathrm{prior}}(s'|s)\; \exp\left\{\frac{Q^*_{\mathrm{soft}}(s,s')-V^*_{\mathrm{soft}}(s)}{\alpha}\right\},$$4 trajectories under $$\pi^*(s'|s)\;\propto\;\pi_{\mathrm{prior}}(s'|s)\; \exp\left\{\frac{Q^*_{\mathrm{soft}}(s,s')-V^*_{\mathrm{soft}}(s)}{\alpha}\right\},$$5, storing them in a buffer, sampling $$\pi^*(s'|s)\;\propto\;\pi_{\mathrm{prior}}(s'|s)\; \exp\left\{\frac{Q^*_{\mathrm{soft}}(s,s')-V^*_{\mathrm{soft}}(s)}{\alpha}\right\},$$6 for updates, computing the residuals and squared-error loss, and applying gradient updates separately to $$\pi^*(s'|s)\;\propto\;\pi_{\mathrm{prior}}(s'|s)\; \exp\left\{\frac{Q^*_{\mathrm{soft}}(s,s')-V^*_{\mathrm{soft}}(s)}{\alpha}\right\},$$7 and $$\pi^*(s'|s)\;\propto\;\pi_{\mathrm{prior}}(s'|s)\; \exp\left\{\frac{Q^*_{\mathrm{soft}}(s,s')-V^*_{\mathrm{soft}}(s)}{\alpha}\right\},$$8.

5. Theoretical Guarantees and Pathwise Consistency

Trust-PCL leverages the contractive property of the soft-Bellman operator for existence and uniqueness of the fixed point $$\pi^*(s'|s)\;\propto\;\pi_{\mathrm{prior}}(s'|s)\; \exp\left\{\frac{Q^*_{\mathrm{soft}}(s,s')-V^*_{\mathrm{soft}}(s)}{\alpha}\right\},$$9 under bounded rewards and compact policy classes (Nachum et al., 2017). Minimizing the squared pathwise consistency error by gradient descent converges to a local stationary point according to standard SGD guarantees for nonconvex objectives. By annealing $$Q^*_{\mathrm{soft}}(s,s') = r(s,s') + V^*_{\mathrm{soft}}(s'),\qquad V^*_{\mathrm{soft}}(s) = \alpha \log \sum_{s'} \pi_{\mathrm{prior}}(s'|s) \exp\left(\frac{Q^*_{\mathrm{soft}}(s,s')}{\alpha}\right).$$0 to satisfy a fixed KL constraint, Trust-PCL achieves trust-region-like stability corresponding to a chosen KL radius $$Q^*_{\mathrm{soft}}(s,s') = r(s,s') + V^*_{\mathrm{soft}}(s'),\qquad V^*_{\mathrm{soft}}(s) = \alpha \log \sum_{s'} \pi_{\mathrm{prior}}(s'|s) \exp\left(\frac{Q^*_{\mathrm{soft}}(s,s')}{\alpha}\right).$$1.

No importance sampling weights are required in the standard KL-regularized objective when using off-policy data, as pathwise consistency holds for all samples. The use of a recency-weighted replay buffer (with priority proportional to $$Q^*_{\mathrm{soft}}(s,s') = r(s,s') + V^*_{\mathrm{soft}}(s'),\qquad V^*_{\mathrm{soft}}(s) = \alpha \log \sum_{s'} \pi_{\mathrm{prior}}(s'|s) \exp\left(\frac{Q^*_{\mathrm{soft}}(s,s')}{\alpha}\right).$$2 for sub-episode insertion time $$Q^*_{\mathrm{soft}}(s,s') = r(s,s') + V^*_{\mathrm{soft}}(s'),\qquad V^*_{\mathrm{soft}}(s) = \alpha \log \sum_{s'} \pi_{\mathrm{prior}}(s'|s) \exp\left(\frac{Q^*_{\mathrm{soft}}(s,s')}{\alpha}\right).$$3) further accelerates learning.

6. Empirical Performance and Applications

On standard continuous control benchmarks (Acrobot, HalfCheetah, Swimmer, Hopper, Walker2d, Ant in MuJoCo), Trust-PCL matches or exceeds the final reward achieved by TRPO while requiring $$Q^*_{\mathrm{soft}}(s,s') = r(s,s') + V^*_{\mathrm{soft}}(s'),\qquad V^*_{\mathrm{soft}}(s) = \alpha \log \sum_{s'} \pi_{\mathrm{prior}}(s'|s) \exp\left(\frac{Q^*_{\mathrm{soft}}(s,s')}{\alpha}\right).$$4–$$Q^*_{\mathrm{soft}}(s,s') = r(s,s') + V^*_{\mathrm{soft}}(s'),\qquad V^*_{\mathrm{soft}}(s) = \alpha \log \sum_{s'} \pi_{\mathrm{prior}}(s'|s) \exp\left(\frac{Q^*_{\mathrm{soft}}(s,s')}{\alpha}\right).$$5 fewer environment steps (Nachum et al., 2017). Ablation studies show that removing the KL penalty ($$Q^*_{\mathrm{soft}}(s,s') = r(s,s') + V^*_{\mathrm{soft}}(s'),\qquad V^*_{\mathrm{soft}}(s) = \alpha \log \sum_{s'} \pi_{\mathrm{prior}}(s'|s) \exp\left(\frac{Q^*_{\mathrm{soft}}(s,s')}{\alpha}\right).$$6, $$Q^*_{\mathrm{soft}}(s,s') = r(s,s') + V^*_{\mathrm{soft}}(s'),\qquad V^*_{\mathrm{soft}}(s) = \alpha \log \sum_{s'} \pi_{\mathrm{prior}}(s'|s) \exp\left(\frac{Q^*_{\mathrm{soft}}(s,s')}{\alpha}\right).$$7) triggers instability, while strict on-policy sampling drives up sample cost. Off-policy Trust-PCL with recency-weighted buffer achieves both stability and superior sample efficiency.

In generative modeling contexts, Trust-PCL’s equivalence to RTB provides a unified perspective. For example, in the 2D Gaussian mixture experiment of (Deleu et al., 1 Sep 2025), both RTB and properly configured off-policy Trust-PCL recover the full multimodal distribution, provided the reward matches the target energy function $$Q^*_{\mathrm{soft}}(s,s') = r(s,s') + V^*_{\mathrm{soft}}(s'),\qquad V^*_{\mathrm{soft}}(s) = \alpha \log \sum_{s'} \pi_{\mathrm{prior}}(s'|s) \exp\left(\frac{Q^*_{\mathrm{soft}}(s,s')}{\alpha}\right).$$8. This demonstrates that prior reports of RTB’s superiority reflected reward mismatch and sampling protocol, not an intrinsic algorithmic difference.

7. Impact, Implications, and Research context

Trust-PCL unifies off-policy sample efficiency with the robust convergence properties of trust-region optimization, making it a central tool in KL-regularized RL for both RL control and generative modeling applications. Its equivalence to the RTB objective situates it as the canonical off-policy squared-error algorithm for energy-based marginal matching, directly extending to GFlowNets and beyond (Deleu et al., 1 Sep 2025). The use of off-policy data, automatic KL control, and pathwise consistency positions Trust-PCL as a foundational reference in both continuous RL and sequential generative model fine-tuning.

A plausible implication is that, under proper reward and policy update design, off-policy KL-regularized RL objectives provide a flexible, expressive, and stable optimization family for energy-based modeling and generative flow networks, obviating the need for distinct new objectives for finetuning in sequential settings.