Dynamic Teaching in Sequential Decision-Making

• Dynamic Teaching is a framework that adaptively selects instructional sequences in MDPs to optimize student learning through continual feedback and curriculum design.
• It leverages methods like adaptive demonstration scheduling, difficulty ratio curricula, and segmented instruction to boost sample efficiency and accelerate policy convergence.
• Empirical studies show that these dynamic methods improve performance in tasks such as robotic manipulation, navigation, and text-based decision making by reducing demonstration costs.

Dynamic teaching in sequential decision making environments denotes a set of algorithms, theoretical frameworks, and empirical methods wherein a teacher agent actively and adaptively selects instructional sequences to optimize the learning of a student policy in complex Markov decision processes (MDPs), bandit processes, or related sequential settings. Distinguishing itself from static demonstration or passive imitation, dynamic teaching leverages continual policies, curriculum construction, feedback, and model inference to accelerate convergence, promote generalization, and achieve sample-efficient learning—even in the presence of stochasticity, limited feedback, or evolving learner states. This article synthesizes the core definitions, methodologies, theoretical contributions, and empirical findings of dynamic teaching, referencing principal works in interactive RL, imitation learning, inverse RL, meta-teaching, and teaching dimension theory.

1. Core Principles and Formal Setting

Dynamic teaching considers a sequential decision environment formalized as an MDP or related structure:

• MDP tuple: $(\mathcal{S}, \mathcal{A}, P, r, \gamma)$, with state space $\mathcal{S}$, action space $\mathcal{A}$, transition kernel $P$, reward $r$, and discount $\gamma$.
• Teacher and learner roles: The teacher possesses additional knowledge (target reward, optimal policy, true transition model, or "desired" trajectories) and adaptively steers the instructional process, viewing demonstration as a control problem over trajectories and feedback.
• Student policy learning: The student is a RL, IRL, or imitation learner with access to the teacher’s feedback, but possibly with limited state/action coverage, partial observability, or evolving inner state.

Traditional static-policy or imitation-learning approaches present the learner with a fixed set of expert demonstrations or a single expert policy. Dynamic teaching extends this by constructing policies that adaptively select the sequence, content, or granularity of teaching signals in order to minimize sample complexity, account for stochastic labels, facilitate coverage beyond the expert's manifold, and integrate active querying or structured curricula (Yin et al., 2020, Walsh et al., 2012).

2. Teaching Dimension, Subset Teaching, and Sequential Extensions

Dynamic teaching requirements are rigorously captured by extensions of classical teaching dimension (TD) theory. In the MDP or stochastic sequence setting, the following generalizations are central (Walsh et al., 2012):

• Teaching Dimension (TD): Minimum number of labeled examples required to uniquely identify a target concept in a supervised context.
• Subset Teaching Dimension (STD): Reduces to minimum size of a set that, by mutual teacher-learner awareness, eliminates all candidate hypotheses except the target, often with orders-of-magnitude reduction relative to TD.
• Noisy Teaching Dimension (NTD/NSTD): Extends TD/STD to stochastic concept classes; focuses on convergence in total-variation or error probability and early stopping rules under empirical feedback.
• Sequential/MDP teaching (TD$^s$/SSTD): Introduces path and transition constraints; the teacher constructs sequences in the reachable space of the MDP, often requiring shortest teaching tours analogous to TSP.

Key theoretical results demonstrate that optimal dynamic teaching in MDPs is NP-hard, but greedy heuristics yield practical, sample-efficient solutions—sometimes achieving sample complexity reductions from $O(\epsilon^{-2}\log(1/\delta))$ to $O(\epsilon^{-2})$ or less for certain concept classes. For example, for teaching monotone conjunctions, STD reduces teaching cost from $n$ to $1$ (Walsh et al., 2012).

3. Teaching Algorithms and Adaptive Curriculum Construction

Modern dynamic teaching frameworks instantiate these principles via explicit algorithms for curriculum design, adaptive demonstration, or interactive feedback. Methods include:

• Teacher-student imitation with full Q-table curricula: The teacher, trained via RL (e.g., DQN with Bellman error loss), generates a curriculum of rich demonstration trajectories annotated with Q-values, enabling the student (e.g., a contextualized LLM like BERT) to learn via regression or cross-entropy/KL objectives on the entire admissible action set (Yin et al., 2020). Richer curricula provide denser gradient signals than single-action imitation, yielding rapid convergence and the ability to leverage large function approximators.
• Scheduling and allocation of demonstrations: Dynamic teaching involves shaping a curriculum by ranking demonstrations according to learner-centric difficulty or maximizing the ratio of difficulty under the learner policy to difficulty under the expert policy. Formally, for demonstration $\xi$, $$\Psi^L_t(\xi) / \Psi^E(\xi) = \prod_\tau \pi^E(a_\tau|s_\tau) / \pi^L_t(a_\tau|s_\tau)$$. This ratio-based curriculum accelerates convergence for both MaxEnt-IRL and Cross-Ent-BC learners, with provable linear convergence (Yengera et al., 2021, Zayanov et al., 2023).
• Active state selection and demonstration: In limited feedback scenarios, the teacher actively queries learner trajectories from selected states (maximizing value-at-risk reductions), infers the learner’s latent policy via causal-entropy IRL from minimal feedback, and selects subsequent demonstrations via algorithms such as the "difficulty score ratio." Joint planning of state selection and demonstration maximizes policy improvement per feedback round (Zayanov et al., 2023).
• Segmented/sequential demonstration (ST$^2$): For complex, long-horizon robotic tasks, the teacher segments the demonstration into sub-tasks via key-points. Each segment is learned as an independent policy, reducing compounding error, localizing drift, and alleviating demonstration fatigue in kinesthetic teaching (Ajanović et al., 23 Oct 2025).

4. Dynamic Teaching for Environment Dynamics, Meta-Learning, and Model Transfer

Beyond direct policy instruction, dynamic teaching frameworks encompass teacher-guided learning of transition models and meta-learning of learner states:

• Environment dynamics modeling: Behavior Aware Modeling (BAM) jointly infers task cost/reward and transition dynamics $T_\theta$ by fusing teacher demonstrations, evaluative feedback, and observed transitions. Maximizing the likelihood of both behavior and transitions, with gradient propagation through soft value iteration, allows for more rapid and reusable inference of shared environment structure than policy cloning or task-specific IRL (Loftin et al., 2019).
• Teaching learners with inner/latent states: In settings where the learner’s meta-parameters or algorithm itself evolves (meta-learning), dynamic teaching formalizes the process as an optimal control problem over the joint state $(\theta_t, z_t)$, with objectives trading off final model performance and generalization to future tasks. Non-manipulative teaching policies induce "enlightenment" of the inner state, promoting generalization outside the training domain and avoiding indoctrination via withheld information (Celikok et al., 2020).
• Knowledge transfer in batch/offline RL: In safety-constrained domains, dynamic application of various teacher signals—demonstrations, expert-actions at queried states, and teacher-supplied action gradients—are blended into actor-critic updates. Scheduling or filtering (e.g., via Q-filters) ensures reliance on the teacher only when the teacher's action is truly better, phasing out as the learner surpasses the teacher (Emedom-Nnamdi et al., 2023).

5. Empirical Results and Practical Implications

Empirical studies substantiate the theoretical advantages of dynamic teaching, demonstrating significant improvements in sample efficiency, generalization, and user experience across diverse domains:

• Text-based sequential decision tasks: Student models trained with dynamic teacher Q-table curricula and contextual representations achieve +7% in-domain and +24% out-of-domain improvement relative to RL teachers, converging in hundreds of thousands of updates versus millions for RL alone (Yin et al., 2020).
• Long-horizon robotic manipulation: Segmented demonstration in ST$^2$ increases task success rate (65.5% vs 51.1% for monolithic), reduces motion and torque jerk, and alleviates perceived temporal/physical demand without sacrificing simplicity for users preferring monolithic flows (Ajanović et al., 23 Oct 2025).
• Car driving, navigation, and TSP tasks: Difficulty-ratio curricula and interactive teaching frameworks provide accelerated policy convergence and require fewer demonstrations or episodes, outperforming static demonstration baselines (Yengera et al., 2021, Zayanov et al., 2023).
• Employer-labeled batch RL: Carefully scheduled and adaptively blended teacher signals achieve rapid, safe policy improvement in continuous-control tasks, overcoming coverage and covariate shift limitations of pure behavioral cloning or RL (Emedom-Nnamdi et al., 2023).
• Environment dynamics transfer: BAM reduces teacher effort by 20–50% versus baseline methods and generalizes dynamics knowledge across tasks, especially when environments share latent structure (Loftin et al., 2019).

6. Limitations, Open Problems, and Future Directions

While dynamic teaching frameworks exhibit substantial strengths, several theoretical and practical challenges remain:

• The sample and computational complexity of optimal teaching remains high for large or continuous state spaces; greedy heuristics are effective but not universally optimal (Walsh et al., 2012).
• Coverage limitations persist when the teacher support does not sufficiently overlap with optimal policy regions; interleaving teacher-driven and student exploration or fusing multiple teacher sources are open avenues (Yin et al., 2020).
• Real-world user studies indicate cognitive overhead in segmenting tasks or interpreting instructional strategies, suggesting further interface and automation research for key-point selection, demonstration scheduling, and sub-policy composition (Ajanović et al., 23 Oct 2025).
• Scalable approaches to joint dynamics and reward/behavior inference, richer modalities of feedback (e.g., natural language), and adaptation to imperfect, partial, or uncertain teachers and learners are open research areas (Loftin et al., 2019, Celikok et al., 2020, Peltola et al., 2018).
• Formal sample complexity guarantees in partially observable, nonstationary, or multi-agent domains are incomplete.

Dynamic teaching provides a principled, theoretically grounded, and empirically validated framework for active, adaptive, and structured instruction in sequential decision making environments, subsuming classical teaching dimension, curriculum learning, interactive demonstration, and meta-level teaching of evolving learners.

Cited Works

• (Yin et al., 2020) Learning to Generalize for Sequential Decision Making
• (Ajanović et al., 23 Oct 2025) Sequentially Teaching Sequential Tasks $(ST)^2$: Teaching Robots Long-horizon Manipulation Skills
• (Zayanov et al., 2023) Interactively Teaching an Inverse Reinforcement Learner with Limited Feedback
• (Loftin et al., 2019) Interactive Learning of Environment Dynamics for Sequential Tasks
• (Celikok et al., 2020) Teaching to Learn: Sequential Teaching of Agents with Inner States
• (Peltola et al., 2018) Machine Teaching of Active Sequential Learners
• (Emedom-Nnamdi et al., 2023) Knowledge Transfer from Teachers to Learners in Growing-Batch Reinforcement Learning
• (Yengera et al., 2021) Curriculum Design for Teaching via Demonstrations: Theory and Applications
• (Walsh et al., 2012) Dynamic Teaching in Sequential Decision Making Environments