Teacher–Student Reinforcement Learning

• Teacher–Student Reinforcement Learning (TSRL) is a framework where a teacher model guides a student using advice, reward signals, curricula, or latent alignment to boost learning efficiency.
• TSRL techniques include action advice under a budget, meta-control over optimization, and latent alignment, which together accelerate early learning and enhance deployability.
• Empirical studies demonstrate TSRL's ability to improve sample efficiency, robustness, and transferability while mitigating challenges such as teacher-student mismatch and non-stationarity.

Searching arXiv for recent and foundational papers on Teacher–Student Reinforcement Learning and closely related teacher-guided RL formulations. Teacher–Student Reinforcement Learning (TSRL) denotes a family of learning frameworks in which a “teacher” model guides a “student” learner during reinforcement learning or during closely coupled optimization processes that are cast in reinforcement-learning terms. Across the literature, the teacher may provide action advice, reward augmentation, curricula, loss weights, latent representations, or privileged features; the student may be an RL agent, a supervised learner controlled by an RL teacher, or a deployable control policy distilled from a privileged counterpart. The unifying theme is asymmetric information or capability: the teacher possesses knowledge, observations, representational capacity, or meta-control authority that the student lacks, and uses that asymmetry to improve learning dynamics, sample efficiency, generalization, robustness, or deployability. The resulting formulations range from action-level advising under a budget (Fachantidis et al., 2017), to reward-level teacher guidance embedded directly into the MDP (Reid, 2020), to joint teacher–student optimization of latent spaces or curricula (Muslimani et al., 2022, Schraner, 2022).

1. Scope and conceptual variants

TSRL is not a single algorithmic template but a class of asymmetric learning mechanisms. One major line treats the teacher as an advisor or controller over the student’s learning process. In “Learning to Teach Reinforcement Learning Agents” the teacher intermittently suggests actions under a finite advice budget, and must learn both what to advise and when to advise (Fachantidis et al., 2017). In “Student/Teacher Advising through Reward Augmentation” the teacher’s fixed action-value function is converted into a penalty term subtracted from the student’s reward, so the student can be trained by an otherwise unchanged RL algorithm (Reid, 2020). In “Teacher-student curriculum learning for reinforcement learning” the teacher is a task selector in a Curriculum MDP, choosing which source task the student should train on next while the student itself learns with PPO (Schraner, 2022).

A second line treats the teacher as a provider of privileged observations or higher-capacity representations. “Teacher-Student Reinforcement Learning for Mapless Navigation using a Planetary Space Rover” uses a privileged teacher trained in simulation with noise-free exteroception, followed by a student trained offline on noisy observations through behavior cloning and a recurrent belief encoder (Mortensen et al., 2023). “CTS: Concurrent Teacher-Student Reinforcement Learning for Legged Locomotion” trains teacher and student concurrently under PPO, with a privileged encoder for the teacher and a proprioceptive encoder for the student coupled by latent reconstruction (Wang et al., 2024). “VMTS” similarly employs a privileged teacher and a deployable student, but adds a mixture-of-experts teacher and a Barlow Twins-style alignment loss between teacher and student representations (2503.07049).

A third line uses RL to control teaching signals in nonstandard student settings. “Reinforcement Teaching” models the student’s learning process as a Teaching MDP and trains a teacher policy to choose interventions such as start states or optimizer step sizes, using a learned parametric-behavior embedder to represent the student (Muslimani et al., 2022). “Student-Teacher Curriculum Learning via Reinforcement Learning” uses an RL teacher that observes a representation of the student’s weights and selects entropy-ordered training batches for supervised students on hospital admission, MIMIC-III, and CIFAR-10 tasks (El-Bouri et al., 2020). “Tutor-Student Reinforcement Learning” for deepfake detection casts per-sample loss weighting as a PPO-controlled teaching policy with dense rewards tied to incorrect-to-correct prediction flips (Lei et al., 25 Mar 2026).

This diversity suggests a broad editor’s term, “teaching channel”: the interface through which the teacher influences the student. In the cited literature, the teaching channel may be the action stream, reward signal, task distribution, observation representation, or loss weighting. A plausible implication is that TSRL is better characterized by asymmetry plus adaptive coupling than by any particular architectural choice.

2. Formalizations of teacher influence

A recurring formal structure is to embed teacher guidance into a standard RL objective while preserving the student’s own policy optimization loop. In reward augmentation TSRL, the student remains a tabular Q-learning agent in a one-player stochastic game with state $s=\langle h_x,h_y,p_x,p_y\rangle$, action set $\{0,1,2,3\}$, reward $-1$ per step and $0$ at capture, and update

$$Q(S,a)\leftarrow Q(S,a)+\alpha\left[R+\gamma \max_{a'}Q(S',a')-Q(S,a)\right].$$

Teacher guidance enters through a stationary punishment function $pun_x(s,a)$ derived from a fixed $Q_{\text{teacher}}(s,a)$, giving the augmented reward

$\hat R_{\text{agent}}=R_{\text{agent}}-pun_x(s,a),$

and the same Q-learning update is then applied with $\hat R_{\text{agent}}$ in place of $R$ (Reid, 2020). The three schedules are punishing sub-optimal actions, anti-optimal actions, and continuous punishment by severity.

A second formal structure is constrained or regularized optimization against a teacher policy. “Don’t Forget Your Teacher” formulates the student as minimizing expected discounted cost subject to a divergence constraint between student and teacher policies, using either reverse or forward KL between trajectory distributions and solving the resulting primal–dual problem with projected policy-gradient updates (Nazari et al., 2019). By contrast, “TGRL” uses a reward-plus-imitation objective,

$\{0,1,2,3\}$0

but dynamically adjusts the effective supervision strength by comparing the main student to a counterfactual RL-only learner through a dual update on $\{0,1,2,3\}$1, so teacher supervision increases only when it improves task performance over the RL-only baseline (Shenfeld et al., 2023).

A third formal structure is representation-level coupling. In LATS, each traffic-signal agent follows

$\{0,1,2,3\}$2

where $\{0,1,2,3\}$3 is a distilled semantic feature obtained by aligning student and teacher latent posteriors through VAEs and a KL term inside a teacher–student objective $\{0,1,2,3\}$4 (Zhang et al., 25 Mar 2026). In CTS, teacher and student share a policy network but use different encoders, with teacher and student PPO surrogates optimized concurrently and the student encoder trained by reconstruction of the teacher latent (Wang et al., 2024). In VMTS, the student is trained with mean squared error on teacher actions plus an explicit cross-correlation alignment loss between teacher and student encodings (2503.07049). In representational-alignment TSRL for imitation learning, the teacher policy is deliberately made “imitable” by forcing teacher and student inputs into a shared contrastively learned latent space while preventing policy gradients from reintroducing private teacher-only information into the encoder (Mammadov et al., 27 May 2026).

These formulations are heterogeneous, but they all instantiate a common asymmetry: the teacher has access to either superior policy estimates, privileged observations, a larger semantic model, or meta-level control over training.

3. Teacher roles: advice, curriculum, and meta-control

Action advising is the most direct teacher role. In the advice-under-budget formulation, the teacher action space is $\{0,1,2,3\}$5 while budget remains, with $\{0,1,2,3\}$6 denoting “no advice,” and the teacher’s augmented state includes the remaining budget and the teacher’s acting value function $\{0,1,2,3\}$7 (Fachantidis et al., 2017). The paper distinguishes advice production from advice distribution: a teacher policy may know which action is good, yet still need a separate policy for deciding when advice is worth spending. The proposed Q-Teaching algorithm learns this timing policy with reward proportional to the teacher’s value gap between the greedy advised action and a baseline action, either the student’s intended action or the worst-valued action.

Curriculum control is a broader teaching role. In teacher-student curriculum learning for RL, the teacher’s action is task selection from a predefined task set, and the teacher is optimized in a Curriculum MDP using observation summaries such as Reward History, Previous Task Reward, Learning Progress, Absolute Learning Progress, and exponential moving averages (Schraner, 2022). In the hospital-admission curriculum work, the teacher observes a representation of the student’s weights and selects the next batch of entropy-sorted training data, with actions controlling the central position and width of a selected region along an entropy axis (El-Bouri et al., 2020). In the deepfake-detection tutor framework, the curriculum is finer grained: the tutor assigns each sample a continuous weight in $\{0,1,2,3\}$8, thereby re-weighting the student’s per-sample loss online based on visual features, confidence, EMA loss, and forgetting counts (Lei et al., 25 Mar 2026).

Meta-control over optimization itself appears in “Reinforcement Teaching,” where the teacher acts on components of the student learning process such as the student’s start state or optimizer step size, and uses learning-progress reward shaping

$\{0,1,2,3\}$9

to accelerate teacher learning while preserving the optimal teaching policy under potential-based shaping (Muslimani et al., 2022). In MTKD-RL, the RL agent acts by assigning sample-wise weights to multiple teachers in a knowledge-distillation setting, using both teacher performance and teacher–student gap features as state (Yang et al., 22 Feb 2025). This suggests that TSRL extends naturally beyond conventional environment action spaces into “learning-process action spaces.”

4. Privileged information, representation transfer, and latent alignment

A large subset of TSRL is organized around privileged information. In the rover navigation pipeline, the teacher receives noise-free proprioceptive and exteroceptive inputs, including dense and sparse simulation heightmaps, and is trained with PPO in Isaac Sim. The student then imitates the teacher from realistic noisy observations $-1$0, using a GRU-based recurrent belief model and attention gating to reconstruct teacher-like latent terrain states (Mortensen et al., 2023). The method is explicitly framed as “learning by cheating”: teacher competence is optimized under privileged information, while robustness is pushed into the student via noise-aware imitation.

CTS generalizes this privileged/deployable split into concurrent RL. The teacher latent is $-1$1, the student latent is $-1$2, and both teacher and student groups optimize the same policy through separate PPO surrogates, while the student encoder minimizes

$-1$3

to reconstruct the teacher latent (Wang et al., 2024). The coupling occurs in latent space rather than at the action level, and both groups contribute trajectories to the policy update.

VMTS uses a privileged teacher observation $-1$4, where $-1$5 contains simulator physical parameters and $-1$6 is a local height map within a $-1$7 m radius. Its teacher is terrain-aware through a four-expert mixture-of-experts module with weights

$-1$8

and a weighted expert encoding $-1$9 (2503.07049). The student receives only historical proprioception and depth images, and is trained with action imitation plus a Barlow Twins-style alignment term. The reported ablations show that removing either the MoE teacher or the alignment loss degrades tracking and height-error performance.

LATS pushes the representation-transfer formulation further by making the teacher a pretrained embedding LLM rather than a policy. For each phase $0$0 of each intersection, the LLM embeds a textual prompt $0$1 into a $0$2-dimensional semantic feature $0$3, and teacher/student VAEs map teacher and student inputs into $0$4-dimensional latent variables aligned through KL minimization (Zhang et al., 25 Mar 2026). The teacher is used only during training, and the distilled student replaces it at inference, eliminating the reported latency gap between pure LLM control and the deployed policy.

The representational-alignment imitation-learning paper offers the clearest articulation of the “imitation gap.” It defines an irreducible gap $0$5 when the teacher uses private variables unavailable to the student, and addresses it by learning encoders $0$6 and $0$7 into a shared latent through symmetric InfoNCE plus alignment and stability losses, while blocking RL gradients from encoding teacher-private information (Mammadov et al., 27 May 2026). This suggests that one branch of TSRL is no longer primarily about policy transfer, but about making the teacher’s policy itself student-realizable.

5. Empirical patterns, gains, and observed trade-offs

A recurrent empirical result is accelerated early learning. Reward augmentation in the hunter–prey gridworld “can significantly improve the rate of learning,” with sub-optimal punishment producing a large early speedup and anti-optimal punishment producing a smaller early speedup without the same long-run degradation (Reid, 2020). LLM4Teach reports markedly improved early-stage learning relative to tabula rasa PPO and hierarchical RL in sparse-reward MiniGrid, with high success rates reached in fewer than $0$8 iterations (Zhou et al., 2023). In LATS, the distilled semantic module yields improved overall performance and generalization over both traditional RL and LLM-only approaches on Grid $0$9 and Monaco traffic networks, with lowest ATD on both and faster inference than online LLM policies (Zhang et al., 25 Mar 2026).

Another recurrent pattern is improved deployability relative to the teacher. In rover navigation, the privileged teacher slightly outperforms the student under clean simulation inputs, but degrades sharply under noise, whereas the student remains more robust under noisy heightmaps and transfers better to the physical rover (Mortensen et al., 2023). LATS reports deployment without online LLM inference, giving less than $$Q(S,a)\leftarrow Q(S,a)+\alpha\left[R+\gamma \max_{a'}Q(S',a')-Q(S,a)\right].$$0 s per step per agent versus approximately $$Q(S,a)\leftarrow Q(S,a)+\alpha\left[R+\gamma \max_{a'}Q(S',a')-Q(S,a)\right].$$1 s for pure LLM-based policies (Zhang et al., 25 Mar 2026). In VMTS, the full student achieves lower velocity tracking error and height error than blind, PIE, and standard TS baselines in mixed-terrain simulation, while real-world tests show stable locomotion on grassland and obstacle negotiation with depth input (2503.07049).

Curriculum-based TSRL commonly improves generality rather than merely final single-task score. The PPO-based teacher in the curriculum-learning thesis improves total mean return and percentage of environments solved on MiniGrid relative to Uniform, LP, Thompson, and Window baselines, while also showing broader transfer across tasks (Schraner, 2022). In the hospital-admission curriculum work, teachers discover distinct curricula across domains: entropy spikes in Ward Admission, small-batch initialization followed by batch expansion in MIMIC-III, and repeated low-entropy then high-entropy batches in CIFAR-10 (El-Bouri et al., 2020).

In constrained-teacher formulations, the advantage is controlled deviation rather than unconstrained exploitation. “Don’t Forget Your Teacher” demonstrates in GridWorld that the student can improve performance while remaining close to a given teacher policy under a KL constraint, and that percentile KL clipping plus entropy control are needed in practice to avoid convergence to highly stochastic policies (Nazari et al., 2019). TGRL reports that dynamic supervision weighting allows the student to exceed suboptimal or privileged teachers in partially observable tasks, while avoiding the brittleness of fixed imitation coefficients (Shenfeld et al., 2023).

A further pattern is that richer teaching signals can outperform pure imitation. MTKD-RL shows that combining teacher performance with teacher–student gap in the agent state yields better results than either alone, across CIFAR-100 and ImageNet classification and in downstream detection and segmentation from pretrained backbones (Yang et al., 22 Feb 2025). The deepfake-detection tutor shows substantially larger gains from dynamic RL-based sample weighting than from static curriculum learning on the CORE backbone (Lei et al., 25 Mar 2026). These results suggest that the strongest empirical gains often arise when the teacher controls where and how strongly to supervise rather than merely what action to take.

6. Failure modes, guarantees, and design tensions

A central tension in TSRL is whether teacher guidance preserves the original task objective. Reward augmentation makes this explicit: because the shaping term $$Q(S,a)\leftarrow Q(S,a)+\alpha\left[R+\gamma \max_{a'}Q(S',a')-Q(S,a)\right].$$2 is not potential-based reward shaping, the paper does not claim policy invariance, and large $$Q(S,a)\leftarrow Q(S,a)+\alpha\left[R+\gamma \max_{a'}Q(S',a')-Q(S,a)\right].$$3 or indiscriminate punishment can degrade asymptotic performance by altering the optimal policy of the original MDP (Reid, 2020). The paper further notes that positive bonuses for teacher-optimal actions can create perverse incentives, causing the student to seek “bonus states” rather than optimize capture speed.

Teacher quality and teacher mismatch are pervasive concerns. Action-advice under budget shows that high average acting performance alone is not a sufficient criterion for teacher usefulness, and that coefficient of variation is more predictive of advising quality than mean score alone (Fachantidis et al., 2017). TGRL is built around the observation that pure imitation can be suboptimal when the teacher is privileged or suboptimal, because the student may need information-gathering behavior the teacher never performs (Shenfeld et al., 2023). The representational-alignment imitation paper formalizes the same issue as an irreducible imitation gap when private teacher variables are not inferable from student observations (Mammadov et al., 27 May 2026).

Another tension lies between fixed and adaptive teachers. Reward augmentation avoids non-stationarity by freezing $$Q(S,a)\leftarrow Q(S,a)+\alpha\left[R+\gamma \max_{a'}Q(S',a')-Q(S,a)\right].$$4 before student learning (Reid, 2020). The same recommendation appears in the detailed design guidance there: if $$Q(S,a)\leftarrow Q(S,a)+\alpha\left[R+\gamma \max_{a'}Q(S',a')-Q(S,a)\right].$$5 changes online, the effective reward becomes a moving target. CTS takes the opposite path, but controls instability by sharing a policy and critic, separating teacher and student encoders, and reconstructing the teacher latent rather than chasing a fixed set of teacher actions (Wang et al., 2024). This suggests two incompatible but successful design philosophies: freeze the teacher to preserve stationarity, or co-train teacher and student under a single optimization procedure that absorbs the non-stationarity into joint policy learning.

Curriculum TSRL introduces its own instability through non-stationary students. The supervised curriculum paper reports that the teacher state is “based on the weights of the student,” and that robustness must be checked by perturbing this state with Gaussian noise; nevertheless, learned policies remain consistent under small perturbations and across teacher seeds (El-Bouri et al., 2020). In SimEdu, the classroom ITS setting is explicitly a POMDP because the student’s knowledge state is only partially observable, and both heuristic and DQN teachers benefit from probing interventions but face trade-offs between information gain and disruption cost (Jiang et al., 19 Nov 2025). A plausible implication is that partial observability is not incidental in TSRL; it is intrinsic whenever the teacher must diagnose the student.

Theoretical guarantees, where present, are highly local to particular formulations. The corrective-RL framework proves asymptotic convergence of the primal–dual policy-gradient method under two-time-scale assumptions (Nazari et al., 2019). Reinforcement Teaching justifies learning-progress shaping as potential-based, so the optimal teaching policy is preserved (Muslimani et al., 2022). Reward augmentation establishes convergence only to the optimal policy of the modified MDP, not the original one (Reid, 2020). Cooperative-game analyses of TSCL reinterpret curriculum rewards as marginal contributions of experiences and argue that order-sensitive values such as Nowak–Radzik can explain when curriculum learners fail under negatively interacting experiences (Diaz et al., 2024). Taken together, these works suggest that TSRL has no single global theory; guarantees are channel-specific.

7. Research directions and synthesis

Several trajectories are evident. One is the move from explicit action advice toward latent or semantic guidance. LATS replaces action guidance with semantic-feature transfer from an embedding LLM to compact MARL agents (Zhang et al., 25 Mar 2026), and representational-alignment TSRL makes “imitable-by-construction” teacher policies the primary design goal (Mammadov et al., 27 May 2026). Another is the move from fixed heuristics to learned meta-teaching. Reinforcement Teaching, MTKD-RL, the deepfake tutor, and the supervised curriculum teacher all place the teacher itself inside an RL loop that optimizes over the student’s learning dynamics (Muslimani et al., 2022, Yang et al., 22 Feb 2025, Lei et al., 25 Mar 2026, El-Bouri et al., 2020).

A second trajectory is toward explicit handling of heterogeneity. LgTS uses an LLM-generated DAG of sub-goals and a teacher that samples which sub-task the student should train next, minimizing environmental interactions under unknown transition dynamics (Shukla et al., 2023). VMTS uses a terrain-aware MoE teacher to handle diverse locomotion terrains (2503.07049). SimEdu models heterogeneity at the population level through different student distributions and course structures, showing that RL policies struggle when shifted from easier to harder cohorts unless trained accordingly (Jiang et al., 19 Nov 2025). This suggests that TSRL increasingly treats heterogeneity not as nuisance variation but as the reason a teacher is needed.

A third trajectory is toward sparse, costly, or budgeted teaching. Advice-under-budget explicitly formulates a finite intervention budget $$Q(S,a)\leftarrow Q(S,a)+\alpha\left[R+\gamma \max_{a'}Q(S',a')-Q(S,a)\right].$$6 and shows that learning when to advise is a constrained exploitation problem (Fachantidis et al., 2017). Reward augmentation notes that continuous teacher input can be expensive because it requires advice at every step, whereas anti-optimal punishment fires less frequently as the student improves (Reid, 2020). GP-based diagnosis before teaching in supervised learning and offline RL frames the teacher’s problem as first inferring what the student lacks, then providing only non-redundant teaching data (Wang et al., 2022). A plausible implication is that future TSRL systems will increasingly optimize teaching efficiency rather than only student return.

Across the cited literature, TSRL can therefore be read as a general asymmetric optimization principle: use a teacher to alter the student’s effective learning process, but do so in a way that respects deployability, mismatch, and cost. The concrete mechanism may be action advice, reward augmentation, latent alignment, task scheduling, or loss weighting, yet the core design problem remains the same: how to exploit asymmetry without locking the student into the teacher’s limitations.