Trust Region Policy Distillation (TOP-D)

• TOP-D is a family of methods that distills a student policy from a teacher by applying explicit trust regions to ensure reliable updates.
• It uses a proximal teacher operator that interpolates between student and teacher policies, bounding token rewards to mitigate update instability.
• The approach leverages PPO-style clipping and trust-region constraints to enhance sample efficiency and support monotonic policy improvement.

Trust Region Policy Distillation (TOP-D) denotes a family of distillation methods in which a student policy is moved toward a teacher under an explicit trust region, typically expressed through KL control, PPO-style clipping, or both. In the direct formulation, TOP-D replaces unstable On-Policy Distillation (OPD) with a proximal teacher that interpolates between the current student and the true teacher, yielding lower-bounded token rewards and a PPO-like update. Closely related formulations apply the same principle to offline-to-online RL fine-tuning, teacher–student mixture anchors, trust-region behavior policies, or teacher-verified supervision regions. Across these variants, the common objective is to preserve dense teacher signal while constraining policy movement or supervision to regions where optimization remains reliable (Xie et al., 6 Jul 2026, Chen et al., 25 May 2026, Zhang et al., 26 Feb 2026, Plyusov et al., 29 May 2026, Xing et al., 31 May 2026).

1. Foundations in trust-region optimization and on-policy distillation

TOP-D sits at the intersection of trust-region RL and policy distillation. In Trust Region Policy Optimization (TRPO), the canonical update maximizes a surrogate objective under a KL constraint around the old policy,

$$\begin{aligned} \max_{\theta}\quad & L_{\theta'}(\theta) \ \text{s.t.}\quad & \mathbb{E}_{s\sim\rho_{\theta'}}\!\left[ D_{\mathrm{KL}}\big(\pi_{\theta'}(\cdot\mid s)\Vert\pi_\theta(\cdot\mid s)\big) \right]\le \delta, \end{aligned}$$

with the surrogate based on advantages under the reference policy. The trust region is the region in policy space within which the surrogate is trusted to approximate the true objective (Schulman et al., 2015).

In the direct TOP-D formulation, OPD is cast as reverse-KL optimization from student to teacher. For prompts $x$ and student samples $y\sim\pi_\theta(\cdot\mid x)$,

$$\theta^*=\arg\min_\theta \mathbb{E}_{x}\!\left[\mathrm{KL}(\pi_\theta(\cdot\mid x),\pi^*(\cdot\mid x))\right] =\arg\max_\theta J(\theta),$$

with

$$J(\theta)=\mathbb{E}_{x\sim\mathcal{D}_x,\,y\sim\pi_\theta(\cdot\mid x)} \left[\log\frac{\pi^*(y\mid x)}{\pi_\theta(y\mid x)}\right].$$

The corresponding token-level policy gradient is

$$\nabla_\theta J(\theta)=\mathbb{E}_{x,y}\left[\sum_{t=1}^{|y|} \nabla_\theta\log\pi_\theta(y_t\mid x,y_{<t}) \Big(\sum_{k=t}^{|y|}\log\rho_k\Big)\right], \quad \rho_k=\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}.$$

This objective makes explicit why vanilla OPD is brittle: the token reward

$$r_k=\log\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}$$

is unbounded below, so large teacher–student disagreements can generate extreme negative rewards and high-variance gradients. The paper also emphasizes that standard OPD is strictly on-policy, so old samples are discarded after each update, which hurts sample efficiency (Xie et al., 6 Jul 2026).

2. Canonical TOP-D mechanism

The defining construction in the direct TOP-D paper is the proximal teacher. At global step $k$, with student policy $\pi_k$, the proximal teacher operator is

$\mathcal{T}(\pi)=\alpha\pi^*+(1-\alpha)\pi,$

so that

$x$0

Token-wise, at state $x$1,

$x$2

This construction induces the TOP-D reward

$x$3

A crucial property is the lower bound

$x$4

which removes the $x$5 penalties that destabilize OPD. The paper explicitly contrasts this with log-space interpolation, where $x$6, which merely rescales rewards and does not bound them.

TOP-D then constructs a token-level return

$x$7

and group-normalized advantages

$x$8

where $x$9 and $y\sim\pi_\theta(\cdot\mid x)$0 are the mean and standard deviation over all tokens in the group of $y\sim\pi_\theta(\cdot\mid x)$1 sampled responses. Trajectories are generated by the frozen behavior policy $y\sim\pi_\theta(\cdot\mid x)$2, and the target policy $y\sim\pi_\theta(\cdot\mid x)$3 is updated with the importance ratio

$y\sim\pi_\theta(\cdot\mid x)$4

The practical TOP-D objective is the PPO-style clipped surrogate

$y\sim\pi_\theta(\cdot\mid x)$5

Operationally, TOP-D combines an external trust region on the supervising distribution with an internal trust region on the student update. The paper further states that the proximal teacher is not sampled from; only its log-ratio with respect to the student is needed, so implementation requires no extra forward passes relative to OPD. Reported training hyperparameters include global batch size 512 prompts, group size $y\sim\pi_\theta(\cdot\mid x)$6, mini-batch size 32 prompts, max prompt length 2048 tokens, max rollout length 16384 tokens, interpolation $y\sim\pi_\theta(\cdot\mid x)$7 or $y\sim\pi_\theta(\cdot\mid x)$8, PPO clip $y\sim\pi_\theta(\cdot\mid x)$9 and $$\theta^*=\arg\min_\theta \mathbb{E}_{x}\!\left[\mathrm{KL}(\pi_\theta(\cdot\mid x),\pi^*(\cdot\mid x))\right] =\arg\max_\theta J(\theta),$$0, off-policy epoch $$\theta^*=\arg\min_\theta \mathbb{E}_{x}\!\left[\mathrm{KL}(\pi_\theta(\cdot\mid x),\pi^*(\cdot\mid x))\right] =\arg\max_\theta J(\theta),$$1, and AdamW learning rate $$\theta^*=\arg\min_\theta \mathbb{E}_{x}\!\left[\mathrm{KL}(\pi_\theta(\cdot\mid x),\pi^*(\cdot\mid x))\right] =\arg\max_\theta J(\theta),$$2 (Xie et al., 6 Jul 2026).

3. Theoretical guarantees and control of optimization error

The direct TOP-D analysis is built around bounded variance, global convergence, and monotonic improvement. Under the assumption that there exists $$\theta^*=\arg\min_\theta \mathbb{E}_{x}\!\left[\mathrm{KL}(\pi_\theta(\cdot\mid x),\pi^*(\cdot\mid x))\right] =\arg\max_\theta J(\theta),$$3 such that

$$\theta^*=\arg\min_\theta \mathbb{E}_{x}\!\left[\mathrm{KL}(\pi_\theta(\cdot\mid x),\pi^*(\cdot\mid x))\right] =\arg\max_\theta J(\theta),$$4

the paper defines the per-token OPD and TOP-D gradients

$$\theta^*=\arg\min_\theta \mathbb{E}_{x}\!\left[\mathrm{KL}(\pi_\theta(\cdot\mid x),\pi^*(\cdot\mid x))\right] =\arg\max_\theta J(\theta),$$5

Its bounded-variance theorem states that, for any $$\theta^*=\arg\min_\theta \mathbb{E}_{x}\!\left[\mathrm{KL}(\pi_\theta(\cdot\mid x),\pi^*(\cdot\mid x))\right] =\arg\max_\theta J(\theta),$$6,

$$\theta^*=\arg\min_\theta \mathbb{E}_{x}\!\left[\mathrm{KL}(\pi_\theta(\cdot\mid x),\pi^*(\cdot\mid x))\right] =\arg\max_\theta J(\theta),$$7

where $$\theta^*=\arg\min_\theta \mathbb{E}_{x}\!\left[\mathrm{KL}(\pi_\theta(\cdot\mid x),\pi^*(\cdot\mid x))\right] =\arg\max_\theta J(\theta),$$8 is vocabulary size and $$\theta^*=\arg\min_\theta \mathbb{E}_{x}\!\left[\mathrm{KL}(\pi_\theta(\cdot\mid x),\pi^*(\cdot\mid x))\right] =\arg\max_\theta J(\theta),$$9 is a universal constant. The asymmetry of the operator is important: large negative signals are hard-bounded by $$J(\theta)=\mathbb{E}_{x\sim\mathcal{D}_x,\,y\sim\pi_\theta(\cdot\mid x)} \left[\log\frac{\pi^*(y\mid x)}{\pi_\theta(y\mid x)}\right].$$0, while large positive signals are linearly damped in $$J(\theta)=\mathbb{E}_{x\sim\mathcal{D}_x,\,y\sim\pi_\theta(\cdot\mid x)} \left[\log\frac{\pi^*(y\mid x)}{\pi_\theta(y\mid x)}\right].$$1.

The global convergence analysis models actual optimization as

$$J(\theta)=\mathbb{E}_{x\sim\mathcal{D}_x,\,y\sim\pi_\theta(\cdot\mid x)} \left[\log\frac{\pi^*(y\mid x)}{\pi_\theta(y\mid x)}\right].$$2

where $$J(\theta)=\mathbb{E}_{x\sim\mathcal{D}_x,\,y\sim\pi_\theta(\cdot\mid x)} \left[\log\frac{\pi^*(y\mid x)}{\pi_\theta(y\mid x)}\right].$$3 captures optimization error. With the policy-space distance

$$J(\theta)=\mathbb{E}_{x\sim\mathcal{D}_x,\,y\sim\pi_\theta(\cdot\mid x)} \left[\log\frac{\pi^*(y\mid x)}{\pi_\theta(y\mid x)}\right].$$4

the iterates satisfy

$$J(\theta)=\mathbb{E}_{x\sim\mathcal{D}_x,\,y\sim\pi_\theta(\cdot\mid x)} \left[\log\frac{\pi^*(y\mid x)}{\pi_\theta(y\mid x)}\right].$$5

If

$$J(\theta)=\mathbb{E}_{x\sim\mathcal{D}_x,\,y\sim\pi_\theta(\cdot\mid x)} \left[\log\frac{\pi^*(y\mid x)}{\pi_\theta(y\mid x)}\right].$$6

then

$$J(\theta)=\mathbb{E}_{x\sim\mathcal{D}_x,\,y\sim\pi_\theta(\cdot\mid x)} \left[\log\frac{\pi^*(y\mid x)}{\pi_\theta(y\mid x)}\right].$$7

This makes $$J(\theta)=\mathbb{E}_{x\sim\mathcal{D}_x,\,y\sim\pi_\theta(\cdot\mid x)} \left[\log\frac{\pi^*(y\mid x)}{\pi_\theta(y\mid x)}\right].$$8 an explicit trade-off parameter: larger $$J(\theta)=\mathbb{E}_{x\sim\mathcal{D}_x,\,y\sim\pi_\theta(\cdot\mid x)} \left[\log\frac{\pi^*(y\mid x)}{\pi_\theta(y\mid x)}\right].$$9 gives faster contraction and a smaller asymptotic ball, but also higher gradient variance.

For monotonic improvement, the paper adapts trust-region analysis to the undiscounted episodic setting with finite maximum response length $$\nabla_\theta J(\theta)=\mathbb{E}_{x,y}\left[\sum_{t=1}^{|y|} \nabla_\theta\log\pi_\theta(y_t\mid x,y_{<t}) \Big(\sum_{k=t}^{|y|}\log\rho_k\Big)\right], \quad \rho_k=\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}.$$0. It defines the normalized state distribution

$$\nabla_\theta J(\theta)=\mathbb{E}_{x,y}\left[\sum_{t=1}^{|y|} \nabla_\theta\log\pi_\theta(y_t\mid x,y_{<t}) \Big(\sum_{k=t}^{|y|}\log\rho_k\Big)\right], \quad \rho_k=\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}.$$1

and the surrogate

$$\nabla_\theta J(\theta)=\mathbb{E}_{x,y}\left[\sum_{t=1}^{|y|} \nabla_\theta\log\pi_\theta(y_t\mid x,y_{<t}) \Big(\sum_{k=t}^{|y|}\log\rho_k\Big)\right], \quad \rho_k=\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}.$$2

It then proves the lower bound

$$\nabla_\theta J(\theta)=\mathbb{E}_{x,y}\left[\sum_{t=1}^{|y|} \nabla_\theta\log\pi_\theta(y_t\mid x,y_{<t}) \Big(\sum_{k=t}^{|y|}\log\rho_k\Big)\right], \quad \rho_k=\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}.$$3

where

$$\nabla_\theta J(\theta)=\mathbb{E}_{x,y}\left[\sum_{t=1}^{|y|} \nabla_\theta\log\pi_\theta(y_t\mid x,y_{<t}) \Big(\sum_{k=t}^{|y|}\log\rho_k\Big)\right], \quad \rho_k=\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}.$$4

This places TOP-D in direct continuity with TRPO-style performance-difference analysis, while the internal clipped updates are used to control the optimization error term $$\nabla_\theta J(\theta)=\mathbb{E}_{x,y}\left[\sum_{t=1}^{|y|} \nabla_\theta\log\pi_\theta(y_t\mid x,y_{<t}) \Big(\sum_{k=t}^{|y|}\log\rho_k\Big)\right], \quad \rho_k=\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}.$$5 (Xie et al., 6 Jul 2026).

4. Related trust-region distillation formulations

A closely related formulation appears in "Extreme Region Policy Distillation" (Chen et al., 25 May 2026). That work separates aggressive signal extraction from KL-efficient projection. Its second stage is explicitly described as what one would call Trust Region Policy Distillation: given an extreme-region teacher $$\nabla_\theta J(\theta)=\mathbb{E}_{x,y}\left[\sum_{t=1}^{|y|} \nabla_\theta\log\pi_\theta(y_t\mid x,y_{<t}) \Big(\sum_{k=t}^{|y|}\log\rho_k\Big)\right], \quad \rho_k=\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}.$$6, find a student $$\nabla_\theta J(\theta)=\mathbb{E}_{x,y}\left[\sum_{t=1}^{|y|} \nabla_\theta\log\pi_\theta(y_t\mid x,y_{<t}) \Big(\sum_{k=t}^{|y|}\log\rho_k\Big)\right], \quad \rho_k=\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}.$$7 by solving

$$\nabla_\theta J(\theta)=\mathbb{E}_{x,y}\left[\sum_{t=1}^{|y|} \nabla_\theta\log\pi_\theta(y_t\mid x,y_{<t}) \Big(\sum_{k=t}^{|y|}\log\rho_k\Big)\right], \quad \rho_k=\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}.$$8

with a PPO-style surrogate driven by the fixed token-level signal

$$\nabla_\theta J(\theta)=\mathbb{E}_{x,y}\left[\sum_{t=1}^{|y|} \nabla_\theta\log\pi_\theta(y_t\mid x,y_{<t}) \Big(\sum_{k=t}^{|y|}\log\rho_k\Big)\right], \quad \rho_k=\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}.$$9

The same paper argues that aggressive off-policy optimization spends substantial KL on unnecessary drift, and that the distillation stage improves KL efficiency by filtering harmful drift while preserving useful signals. It also extends the trust-region view to weak-teacher settings, including signals constructed from “unlearned” or older checkpoints.

In "Reinforcement-aware Knowledge Distillation for LLM Reasoning" (Zhang et al., 26 Feb 2026), the corresponding mechanism is Trust Region Ratio Distillation (TRRD). The central ratio is

$$r_k=\log\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}$$0

which is equivalent to a PPO/GRPO-style ratio around a teacher–old-policy mixture anchor. Because it is multiplied by the group-relative advantage and clipped exactly as in PPO/GRPO, teacher influence is selective rather than additive: imitation enters only through the advantage-weighted trust-region update, not as a separate KL penalty. The paper presents this as “advantage-aware, trust-region-bounded distillation on student rollouts.”

"Trust-Region Behavior Blending for On-Policy Distillation" (Plyusov et al., 29 May 2026) shifts the trust-region mechanism from the update to the rollout policy. At each prefix $$r_k=\log\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}$$1, it defines the behavior policy $$r_k=\log\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}$$2 by

$$r_k=\log\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}$$3

The optimizer lies in the geometric family

$$r_k=\log\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}$$4

and the KL budget is annealed to zero during warmup, so training returns to pure student rollouts after warmup. Here the trust region governs which prefixes are visited, while the per-prefix reverse-KL OPD loss remains unchanged.

"Trust Region On-Policy Distillation" (Xing et al., 31 May 2026) uses trust regions at the token level. For a student-sampled token $$r_k=\log\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}$$5, it defines

$$r_k=\log\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}$$6

Inside the trust region, it uses reverse-KL supervision; outside it explores clipping, masking, and, most effectively, a top-$$r_k=\log\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}$$7 forward-KL estimator. It also introduces off-policy teacher-prefix guidance, combining teacher-generated prefixes with student continuations. Taken together, these papers suggest that TOP-D is not a single objective so much as a design pattern: trust regions can be imposed on the teacher, the student update, the rollout policy, the mixture anchor, or the set of reliable supervised tokens.

5. Empirical performance and reported benchmarks

Across the direct TOP-D paper and closely related trust-region distillation variants, the reported empirical pattern is consistent: trust-region control improves stability, sample efficiency, or KL efficiency relative to baseline OPD, KL-regularized distillation, or unconstrained off-policy reuse.

Framework	Setting	Reported outcome
TOP-D (Xie et al., 6 Jul 2026)	Qwen3-8B-Base student, Qwen3-30B-A3B teacher	AIME24 avg@32 50.42% vs OPD 24.58%; AMC23 88.13% vs 76.88%; MATH-500 91.23% vs 87.98%
TOP-D (Xie et al., 6 Jul 2026)	Qwen3-1.7B-Base student, Qwen3-30B-A3B teacher	AIME24 20.31% vs OPD 8.96%; AIME25 17.71% vs 7.50%; AIME26 13.75% vs 5.94%
ERPD stage-2 TOP-D view (Chen et al., 25 May 2026)	Qwen3-4B with CE teachers	Distilled students achieve comparable or better accuracy at lower KL; some students surpass the teacher
RLAD/TRRD (Zhang et al., 26 Feb 2026)	Qwen3-8B-Base, 30K context	RLAD Avg 66.5 vs GRPO 61.0 and KDRL 64.9
TRB (Plyusov et al., 29 May 2026)	Qwen3-1.7B $$r_k=\log\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}$$8 8B and Qwen3-0.6B $$r_k=\log\frac{\pi^*(y_k\mid x,y_{<k})}{\pi_\theta(y_k\mid x,y_{<k})}$$9 4B	Strongest average among compared methods: 33.2 and 44.4
TrOPD (Xing et al., 31 May 2026)	Qwen3-Nemotron-4B teacher, Qwen3-SFT-1.7B student	51.73 vs OPD 48.29

The direct TOP-D paper emphasizes three specific outcomes. First, it states that TOP-D dramatically enhances training stability, sample efficiency, and final performance on mathematical reasoning tasks. Second, it reports that these gains come with zero additional computational overhead relative to OPD. Third, ablations show that removing the proximal teacher ($k$0) recovers OPD-like instability, while removing the off-policy internal trust region yields slower and less sample-efficient training (Xie et al., 6 Jul 2026).

The ERPD results point to a different but complementary empirical claim: strong performance improvements can coexist with substantially smaller KL divergence after trust-region distillation than in the aggressively optimized teacher. That evidence motivates the notion of “wasted KL,” namely policy movement that does not contribute to actual performance improvement (Chen et al., 25 May 2026).

The other formulations reinforce the same pattern in distinct regimes. RLAD/TRRD outperforms offline distillation, standard GRPO, and KL-based on-policy teacher–student knowledge distillation on logic reasoning and math benchmarks (Zhang et al., 26 Feb 2026). TRB attains the strongest average among the compared methods in two math-reasoning distillation settings, and its continuation diagnostics indicate that early trust-region behavior blending changes the visited prefix distribution in a way that leaves easier paths to correct solutions (Plyusov et al., 29 May 2026). TrOPD consistently outperforms SoTA OPD baselines, including OPD, EOPD, and REOPOLD, across mathematical reasoning, code generation, and general-domain benchmarks (Xing et al., 31 May 2026).

6. Scope, limitations, and interpretive boundaries

The direct TOP-D analysis carries explicit assumptions: bounded score-function norm, finite maximum sequence length $k$1, and bounded asymptotic optimization error $k$2. Its empirical validation is reported up to 8B parameters for the student; larger scales such as $k$3B students are not yet studied, and the training curves over roughly 200–400 update steps do not saturate. The paper also notes regimes where TOP-D may be less consequential, including tasks where teacher and student are already very close and tasks with extremely noisy or misaligned teachers (Xie et al., 6 Jul 2026).

Related trust-region variants expose further trade-offs. ERPD explicitly shows that the teacher need not be stronger than the base policy for trust-region distillation to remain useful, because relative token preferences can be extracted from weak or degenerate teachers through alternative signal construction strategies. This suggests a broader interpretation of TOP-D in which the key object is not always a strong teacher policy, but a structured supervisory signal that can be projected safely back into a trust region around the base model (Chen et al., 25 May 2026).

Not all trust-region formulations have the same computational profile. TRB requires online teacher decoding during warmup, with teacher weights plus teacher KV cache resident in memory along with the student; this overhead is temporary but substantial for large teachers (Plyusov et al., 29 May 2026). TrOPD is slightly more expensive than pure OPD because of the top-$k$4 forward-KL branch, although it remains feasible for long sequences with $k$5 (Xing et al., 31 May 2026). By contrast, the direct TOP-D paper’s “zero additional computational overhead” claim applies specifically to its proximal-teacher construction, which modifies the scalar reward algebraically without extra forward passes (Xie et al., 6 Jul 2026).

A plausible implication is that TOP-D is best understood as a trust-region family rather than a single canonical algorithm. In one form, the trust region is centered on the old student and the teacher is made proximal; in another, the center is a teacher–student mixture; in another, the behavior policy is projected toward the teacher within a student-centered KL ball; in another, supervision is admitted only on teacher-verified tokens. What remains invariant is the organizing principle inherited from trust-region RL: dense policy guidance is most useful when the update or the supervision is restricted to a region in which the optimization surrogate remains trustworthy (Schulman et al., 2015).