---
title: Relative Trajectory Balance (RTB)
url: https://www.emergentmind.com/topics/relative-trajectory-balance-rtb
type: topic
---

# Relative Trajectory Balance (RTB)

Relative Trajectory Balance (RTB) is a trajectory-level, off-policy squared-residual objective for sequential generative models in which a learned trajectory distribution is trained **relative to** a reference or prior trajectory distribution, with terminal weighting provided by a reward, likelihood, or energy term. In the diffusion-posterior formulation, RTB enforces the pathwise relation \(Z_\phi\,p_\phi(\tau)\approx r(x_1)\,p_\theta(\tau)\), so that marginalizing trajectories yields \(p_\phi(x_1)\propto p_\theta(x_1)\,r(x_1)\); in the sequential fine-tuning formulation, it targets a terminal marginal of the form \(P_\phi^\top(s_T)\propto \pi_{\mathrm{prior}}^\top(s_T)\exp(-E(s_T)/\alpha)\) [2503.09746], [2509.01632]. A later theoretical analysis places RTB inside KL-regularized reinforcement learning by proving an exact equivalence to Trust-PCL up to a positive constant factor, reframing RTB as a path-consistency objective rather than a fundamentally separate paradigm [2509.01632].

## 1. Core formulation

In the diffusion-based presentation, RTB is defined on a denoising trajectory
\[
\tau = (x_0 \rightarrow x_{\Delta t}\rightarrow \cdots \rightarrow x_1),
\qquad \Delta t=\frac{1}{T},
\]
with Markov transitions
\[
p(x_{t+\Delta t}\mid x_t)=\mathcal N\!\left(x_{t+\Delta t}\mid x_t+u_t(x_t)\Delta t,\sigma_t^2\Delta t\,I\right),
\]
and trajectory density
\[
p(x_0,x_{\Delta t},\dots,x_1)=p(x_0)\prod_{i=1}^T p(x_{i\Delta t}\mid x_{(i-1)\Delta t}).
\]
Given a pretrained prior diffusion model \(p_\theta\), a learned posterior model \(p_\phi\), a positive terminal reward \(r(x_1)\), and a learnable scalar normalizer \(Z_\phi\), the RTB loss is
\[
\mathcal{L}_{\mathrm{RTB}}(\tau;\phi)
:=
\left(
\log
\frac{
Z_\phi\,p_\phi(\tau)
}{
r(x_1)\,p_\theta(\tau)
}
\right)^2.
\]
The zero-loss condition implies
\[
p_\phi(x_1)\propto p_\theta(x_1)\,r(x_1),
\]
so the objective is designed to recover a reward-tilted posterior rather than merely a high-reward policy [2503.09746].

The same paper gives a conditional, amortized version for inverse problems:
\[
\mathcal{L}_{\mathrm{RTB}}(\tau;y,\phi)
:=
\left(
\log
\frac{
Z_\phi(y)\,p_\phi(\tau\mid y)
}{
r(x_1,y)\,p_\theta(\tau\mid y)
}
\right)^2.
\]
Here the learned sampler is conditioned on an observation \(y\), and the scalar normalizer becomes a function \(Z_\phi(y)\). This turns RTB into an amortized posterior-training objective: a single model can be trained to generalize across measurements rather than solve each posterior at sampling time from scratch [2503.09746].

In the finite-horizon sequential fine-tuning presentation, RTB is written as an expectation over an arbitrary behavior distribution \(\pi_b\):
\[
\mathcal{L}_{\mathrm{RTB}}(\phi,\psi)
=
\frac12
\mathbb E_{\tau\sim \pi_b}
\big[
\Delta_{\mathrm{RTB}}^2(\tau;\phi,\psi)
\big],
\]
with residual
\[
\Delta_{\mathrm{RTB}}(\tau;\phi,\psi)
=
\log
\frac{
\pi_{\mathrm{prior}}(\tau)
}{
Z_\psi\,P_\phi(\tau)
}
-
\frac{E(s_T)}{\alpha}.
\]
This formulation emphasizes the same structure in a different notation: a learned trajectory law \(P_\phi\) is matched to a reference trajectory law \(\pi_{\mathrm{prior}}\) tilted by terminal energy \(E(s_T)\) and normalized by \(Z_\psi\) [2509.01632].

## 2. From trajectory balance to relative trajectory balance

RTB inherits its basic pathwise logic from Trajectory Balance (TB), introduced for GFlowNets as a trajectory-level alternative to flow matching and detailed balance. In the original TB formulation, for a complete trajectory \(\tau=(s_0,\dots,s_n=x)\),
\[
\mathcal L_{\mathrm{TB}}(\tau)
=
\left(
\log
\frac{
Z_\theta \prod_{t=1}^n P_F(s_t\mid s_{t-1};\theta)
}{
R(x)\prod_{t=1}^n P_B(s_{t-1}\mid s_t;\theta)
}
\right)^2,
\]
and any global minimizer defines a policy that samples exactly from the reward-proportional target distribution [2201.13259]. TB is therefore an absolute balance condition: forward trajectory mass, multiplied by a learned partition function, is matched directly to terminal reward times backward path mass.

RTB is a relative version of this construction. In the diffusion derivation, the target unnormalized density is \(p^{\text{prior}}(x_1)\,r(x_1)\), and the appendix shows that RTB arises from a TB-style objective in which posterior and prior share the same fixed backward process, causing the backward terms to cancel. The resulting balance equation is
\[
Z_\phi\,p_\phi(\tau)\approx r(x_1)\,p_\theta(\tau),
\]
so the reference process appears explicitly inside the numerator-denominator ratio rather than only through a terminal reward [2503.09746].

A later theoretical paper sharpens this relation. It studies RTB as a method for fine-tuning sequential generative models toward
\[
P_\phi^\top(s_T)\propto \pi_{\mathrm{prior}}^\top(s_T)\exp(-E(s_T)/\alpha),
\]
and proves that RTB is exactly equivalent to Trust-PCL when the reward design satisfies
\[
\sum_{t=0}^{T}r(s_t,s_{t+1})=-E(s_T),
\]
with the correspondences
\[
V^\psi_{\mathrm{soft}}(s_0)=\alpha\log Z_\psi,
\qquad
\pi_\phi(s'\mid s)=P_\phi(s'\mid s).
\]
Under these identifications,
\[
\mathcal{L}_{T}(\phi,\psi)=\alpha^2\mathcal{L}_{\mathrm{RTB}}(\phi,\psi).
\]
This is an exact objective equivalence up to the multiplicative factor \(\alpha^2\), not merely an analogy at the level of fixed points or asymptotic optima [2509.01632].

The same paper also shows an on-policy gradient relation. If one differentiates RTB while ignoring derivatives through the sampling distribution, then the resulting gradient equals a REINFORCE-with-KL gradient up to a factor \(1/\alpha\), provided the REINFORCE baseline is chosen as
\[
b=\alpha\log Z_\psi.
\]
This places RTB even more directly inside the family of KL-regularized path-consistency methods rather than outside it [2509.01632].

## 3. Off-policy character and optimization mechanics

A defining property of RTB is that it is explicitly formulated as an off-policy objective. In the sequential formulation, the expectation is taken under an arbitrary behavior distribution \(\pi_b\), and in the diffusion formulation the gradient with respect to \(\phi\) does not require backpropagation through the trajectory-generation process. This permits training trajectories to come from distributions other than the current model, including replay-buffer policies and backtracking-based behavior policies [2509.01632], [2503.09746].

The diffusion paper makes this operational with replay and backtracking exploration. It defines a replay buffer
\[
\mathcal D=\{(x_1,r(x_1))\},
\]
and a behavior distribution
\[
P_\beta(\tau)=P_B(\tau\mid x_1)\,P(x_1;\mathcal D),
\]
where \(P(x_1;\mathcal D)\) is a prioritization distribution over stored terminal samples. The intended use is to obtain high-target-density terminal states \(x_1\), sample reverse/noising trajectories
\[
x_1 \leftarrow x_{1-\Delta t}\leftarrow \cdots \leftarrow x_0,
\]
and then train RTB on those trajectories even if the current sampler \(p_\phi\) cannot yet reach those modes on-policy. The paper presents this as a direct mechanism for mode discovery and for preventing catastrophic forgetting once good terminal samples have been discovered [2503.09746].

The same source describes two practical stabilizations. First, it replaces the squared loss by the empirical variance over a minibatch of the quantity inside the square, thereby removing dependence on \(\log Z_\phi\); this is described as a relative variant of VarGrad. Second, it applies loss clipping by skipping updates on trajectories whose loss is already close to \(0\). Both modifications are presented as numerical stabilizers, especially in conditional settings [2503.09746].

The equivalence paper uses off-policy capability to revisit an illustrative example from the RTB literature. It argues that earlier negative results for KL-regularized RL were partly caused by a reward mismatch, and reports that off-policy REINFORCE with self-normalized importance sampling (SNIS), when given the correct reward design and the same exploratory behavior policy, can recover the expected tilted target distribution as accurately as RTB on the toy task [2509.01632]. This does not negate RTB’s off-policy utility; rather, it narrows the dispute to formulation and reward specification.

## 4. RTB for Bayesian inverse problems with diffusion priors

The first major applied use of RTB in the supplied literature is posterior training for Bayesian inverse problems with pretrained diffusion priors. The inferential target is
\[
p(x\mid y)\propto p(y\mid x)\,p(x),
\]
with \(p(x)\) represented implicitly by a diffusion model rather than a tractable closed-form density. RTB addresses the intractability of \(p(x)\) by working on path space: it compares entire posterior trajectories to prior trajectories and uses the likelihood \(p(y\mid x)\) as terminal reward \(r(x)\) [2503.09746].

The paper instantiates this framework on several concrete inverse problems. For inpainting,
\[
p(y\mid x)\propto \exp\left(-\frac{\|Px-y\|^2}{2\sigma^2}\right).
\]
For Fourier phase retrieval,
\[
p(y\mid x)\propto \exp\left(-\frac{\|\,y-|\mathcal F(x)|\,\|^2}{2\sigma^2}\right).
\]
For nonlinear deblur,
\[
p(y\mid x)\propto \exp\left(-\frac{\|A(x)-y\|^2}{2\sigma^2}\right).
\]
For gravitational lensing,
\[
p(x\mid y)\propto p_\theta(x)\,\mathcal N(y;Ax,\sigma^2 I).
\]
The conditional RTB objective then trains an amortized posterior sampler conditioned on \(y\) rather than a training-free guided sampler specialized to a single observation [2503.09746].

A practical modification introduces a Langevin-biased posterior drift:
\[
u_{\phi}(x_t,t)=\mathrm{NN}_1(x_t,t;\phi)+\mathrm{NN}_2(x_t,t,\phi)\,\nabla_{x_t}\log r(x_t).
\]
This is described as an inductive bias that allows reward information to influence intermediate diffusion states rather than only the terminal state. The paper also notes that one may use parameter-efficient fine-tuning, that full backpropagation through the sampling process is not required, and that one may backpropagate only through a subset of trajectory summands when full trajectory memory is too expensive [2503.09746].

Empirically, the paper compares RTB with training-free posterior methods such as DPS, FPS, FPS-SMC, classifier-guidance-style methods, CLA in gravitational lensing, and hybrids such as RTB+DPS and RTB+DPS+CLA. Its main reported pattern is that guidance-based methods often achieve stronger likelihood or reconstruction metrics, whereas RTB tends to achieve much better \(\log Z\), which the authors interpret as evidence of greater posterior faithfulness. The reported metrics include \(E[\log r(x)]\), LPIPS, FID, and \(\log Z\). In gravitational lensing, RTB used \(300\) diffusion steps, while CLA required \(2000\) steps for reasonable samples; at the same time, the paper notes that training can be unstable for very peaky rewards, and in lensing about \(30\%\) of runs diverged [2503.09746].

The paper’s abstract states that existing training-free diffusion posterior methods struggle to perform effective posterior inference in latent space due to inherent biases. The supplied body text does not provide a formal latent-variable derivation of this bias, but it does present the broader empirical claim that heuristic posterior-guidance dynamics can shift samples away from the true posterior, often yielding artificially high likelihood values at the cost of reduced diversity [2503.09746].

## 5. Related objectives and neighboring developments

RTB is closely connected to, but distinct from, several later or adjacent trajectory-balance methods. The most direct precursor is standard TB in GFlowNets, whose motivation was improved long-range credit assignment relative to flow matching and detailed balance. TB imposes a single global constraint on a complete sampled trajectory and was empirically shown to improve convergence, diversity of generated samples, and robustness to long action sequences and large action spaces [2201.13259].

Not every trajectory-balance method built around a reference model is RTB. The paper “Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training” explicitly does **not** introduce or discuss Relative Trajectory Balance. Its method, TBA, uses standard TB—implemented through a batch-estimated “VarGrad variant of trajectory balance”—inside an asynchronous actor–learner system with replay buffers, and its key systems claim is that TB is off-policy. The paper states that there is no objective named RTB, no derivation of an RTB loss, and no comparison between RTB and standard TB [2503.18929].

A conceptually close but terminologically distinct development appears in “Beyond Mode-Seeking RL: Trajectory-Balance Post-Training for Diffusion Language Models.” That paper does not mention RTB by name, but its TraFL objective defines a reward-tilted target relative to a frozen reference trajectory distribution,
\[
p^*(\tau\mid \mathbf{x})
=
\frac{1}{Z(\mathbf{x})}
p_{\mathrm{ref}}(\tau\mid \mathbf{x})
\exp\bigl(\beta r(\mathbf{x},\mathbf{y})\bigr),
\]
and minimizes a squared residual against that reference-anchored target. The paper presents TraFL as a response to “trajectory locking,” a failure mode in diffusion-language-model post-training in which sampled reward-driven updates over-concentrate probability mass on a narrow set of denoising paths [2605.13935]. A plausible implication is that RTB-style reference-relative balance principles had already become a reusable design pattern beyond the original diffusion-posterior setting.

Another related method is Rooted absorbed Prefix Trajectory Balance (RapTB). That paper is not presented as RTB, but it introduces a root-relative auxiliary residual
\[
\bar\Delta_k(\xi)=\Delta_k^{\mathrm{TB}}(\xi)-\Delta_0^{\mathrm{TB}}(\xi),
\]
which cancels the global normalizer and provides dense prefix-level supervision. The paper is explicit that standard terminal TB remains the only exact balance condition whose optimum matches the reward-proportional target, while the rooted relative term is an auxiliary variance-reducing regularizer layered on top [2603.00454]. This suggests a lineage in which “relative” constructions increasingly serve credit-assignment and normalization-cancellation roles even when full RTB is not used.

## 6. Terminological boundaries and recurrent misconceptions

The term “RTB” is not unique to machine learning. In cosmology, RTB denotes **Ricci-trace-based gravity**, the subclass \(f(R,T)=R+f(T)\) of \(f(R,T)\) theories, where the gravitational action depends linearly on the Ricci scalar and nontrivially on the trace \(T\) of the matter energy-momentum tensor [2409.14443]. This usage is entirely unrelated to Relative Trajectory Balance in generative modeling and RL.

Within generative-model post-training, another recurring misconception is to treat every TB-derived, reference-anchored, or replay-compatible method as RTB. The available papers do not support that conflation. TBA is explicitly standard TB in an asynchronous system rather than RTB [2503.18929]. TraFL is RTB-like in the sense of being reference-relative and reward-tilted, but it is not introduced as RTB and uses a diffusion-compatible sequence-level surrogate rather than exact pathwise log-probabilities [2605.13935]. RapTB employs a rooted relative auxiliary residual, but it retains standard TB as the exact global anchor [2603.00454].

A further point of debate concerns novelty. One line of work presents RTB as a recently introduced off-policy objective that is asymptotically unbiased for posterior inference in diffusion inverse problems [2503.09746]. A later theoretical analysis argues that RTB is not a fundamentally new alternative to KL-regularized RL because it is exactly equivalent to Trust-PCL under a specific reward design and parameter identification [2509.01632]. The literature therefore supports two simultaneously true statements: RTB is a useful and practically distinctive pathwise objective for posterior and generative-model fine-tuning, and RTB can also be understood as a re-expression of pre-existing KL-regularized path-consistency machinery rather than a separate theoretical family.

In that sense, the most stable definition of RTB is not institutional or historical but structural: it is a **relative trajectory-balance objective** in which a learned trajectory distribution is matched, via a squared log-ratio residual, to a reward- or likelihood-tilted reference trajectory distribution, with a learned normalization term and explicit off-policy compatibility.

Source: https://www.emergentmind.com/topics/relative-trajectory-balance-rtb