TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training
Abstract: On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD.
Summary
- The paper demonstrates that adaptive rollout-depth budgeting and progressive turn-normalized loss allocation significantly enhance training efficiency and accuracy.
- It corrects shallow-bias by reallocating the KL supervision signal based on fine-grained turn-level analysis.
- Empirical evaluations on ALFWorld, WebShop, and Multi-Hop Search show notable speedups and accuracy improvements over standard on-policy distillation.
TurnOPD: Turn-Aware On-Policy Distillation for Long-Horizon Language Agents
Introduction and Motivation
On-policy distillation (OPD) frameworks, wherein a student model is trained by matching a frozen, stronger teacher along the student’s own trajectories, provide dense, on-policy feedback critical for training language-based agents. However, when scaling to long-horizon tasks—where agents interact with environments over many decision steps—the naïve OPD paradigm is subject to key inefficiencies. In particular, standard OPD with fixed rollout depth and trajectory-level KL aggregation misallocates both computational and optimization resources. Empirical evidence demonstrates that (1) shallow turns dominate the loss budget due to their high KL and survivor frequency, leaving deeper decision points significantly under-trained, and (2) full-horizon rollouts expend compute on turns that convey low or noisy supervision, especially in the tails of trajectories.
Figure 1: A turn-aware perspective on agent OPD—standard OPD with fixed rollout depth and trajectory-level KL over-focuses on shallow tokens and tail turns, while TurnOPD budgets both rollout depth and KL based on turn-level statistics.
The solution proposed, TurnOPD, explicitly incorporates turn-awareness by adaptively regulating both the depth of agent rollouts and the normalization of the KL supervision budget, anchored in fine-grained analysis of signal structure across turns.
Signal Analysis and Diagnosis
A comprehensive diagnostic is performed to decompose the OPD supervision signal along the turn axis. Key observations established through per-turn reverse KL and teacher entropy (see Figure 2) include:
- The KL supervision signal is highly non-uniform, front-loaded on early turns and decaying with depth. This trend persists across embodied planning (ALFWorld) and web navigation/search (Multi-Hop Search) benchmarks.
- Outcome-separability is impaired at depth: In ALFWorld, the gap between failed and successful rollouts in per-turn KL actually inverts sign at late turns, indicating that shallower steps are over-emphasized and deep-turn decision divergence is compressed, often masked by trivial local continuations induced by degenerate student policy behavior.
Figure 2: Turn-resolved teacher uncertainty and reverse-KL in vanilla OPD; reverse-KL mass and teacher entropy are both highly turn-dependent and non-uniform.
A turn-level analysis of KL loss allocation Figure 3 reveals that the vast majority of gradient updates are concentrated on shallow tokens, with the deepest third of turns often receiving less than 5–13% of the raw KL loss across environments. This is a direct consequence of combining uniform token-level normalization with survivor bias and collapsed deep-turn signals.
Figure 3: KL loss mass is mostly allocated to shallow turns under standard trajectory-level reduction, marginalizing deeper decision points.
A contamination-compression formalism is developed to explain the failure of raw KL to reflect true policy misalignment at trajectory tails. Joint context-generation induces high forced-mass in token distributions, particularly for failed rollouts, so the observable KL becomes an unreliable proxy for the actionable student-teacher gap.
The TurnOPD Algorithm
TurnOPD addresses both external (rollout length) and internal (supervision allocation) mismatches by:
1. Adaptive Rollout-Depth Budgeting: Rollouts are truncated at a controller-determined horizon, balancing two arms:
- Efficiency-centric: Survivor-weighted KL centroid reflects where the remaining nontrivial correction signal is present.
- Coverage-bound: A lower-bound on the rollout depth ensures necessary coverage of successful completions (quantile of success-conditioned completion length).
The controller recurrently updates caps by combining both, yielding the minimal necessary horizon to maximize useful supervision per compute (see Figure 4 for adaptation dynamics).
2. Progressive Turn-Normalized Loss Budgeting:
- Distillation loss mass is adaptively reallocated, interpolating over training from standard token-count-based normalization toward a uniform turn-level budget. This linear blend counteracts shallow-bias, ensuring that deep decision points are progressively prioritized as shallow behaviors converge.
Empirical Evaluation and Numerical Results
Experiments span three high-complexity, long-horizon agent benchmarks: ALFWorld (embodied multitask planning), WebShop (grounded web navigation), and Multi-Hop Search (retrieval-augmented reasoning). Students are always distilled from task-specialized, GRPO-trained large teachers. Evaluation regimes include equal wall-clock “Least-Time” and same-step scheduling to measure both ultimate accuracy and efficiency.
TurnOPD consistently outperforms both vanilla OPD and strong curriculum-based baselines such as TCOD-F2B (Wang et al., 27 Apr 2026) across all domains:
| Task | Student | Method | Avg@4 (Least-Time) | Wall Time (h) | Speedup |
|---|---|---|---|---|---|
| ALFWorld (1.7B) | Qwen3-1.7B | OPD | 73.5 ± 1.8 | 4.42 | 1.0× |
| TCOD-F2B | 80.1 ± 1.4 | 1.87 | 2.4× | ||
| TurnOPD | 85.6 ± 1.0 | 1.93 | 2.3× | ||
| Multi-Hop Search | Qwen3.5-2B | OPD | 45.8 ± 0.5 | 4.45 | 1.0× |
| TCOD-F2B | 45.6 ± 1.1 | 3.80 | 1.2× | ||
| TurnOPD | 47.2 ± 1.0 | 2.94 | 1.5× | ||
| WebShop | Qwen3-1.7B | OPD | 77.0 ± 0.8 | 1.57 | 1.0× |
| TCOD-F2B | 80.5 ± 1.6 | 1.33 | 1.2× | ||
| TurnOPD | 82.8 ± 0.8 | 1.24 | 1.3× |
Key results: On ALFWorld-1.7B, TurnOPD yields a 2.3× reduction in wall-clock time and +12 point Avg@4 improvement compared to vanilla OPD. On Multi-Hop Search, it secures the best Least-Time accuracy. Across all environments, TurnOPD advances the global accuracy–efficiency frontier Figure 5.
Figure 5: Iso-training-time efficiency curves across tasks—TurnOPD consistently shifts the accuracy–time tradeoff frontier over baselines.
Ablations confirm the complementary effect of the two main interventions Figure 6: Adaptive depth alone greatly reduces compute but cannot improve shallow-bias; linear turn-balanced normalization alone sharply boosts accuracy but is still compute-intensive; only in combination does TurnOPD realize both acceleration and optimal accuracy. Furthermore, the controller’s behavior closely tracks the diagnostic metrics it was designed to optimize Figure 4.
Figure 4: Rollout-depth controller tracks dynamically the survivor-weighted KL centroid and success-coverage lower bound for efficient turn allocation.
Figure 6: Component ablation on ALFWorld—Adaptive depth (blue), blend norm (green), and full TurnOPD (red) compared to vanilla OPD (black) for both accuracy and wall time.
Implications and Future Directions
TurnOPD demonstrates that, for long-horizon language agents, neither rollout nor KL objective normalization should be statically allocated. Turn-aware budget controls are essential for maximizing supervision utility and computational efficiency. The formal diagnosis suggests that any sequence-level distillation can suffer from latent signal compression and allocation drift when the supervision unit (token) is not aligned to true decision structure (turn or interaction). Extensions might further consider non-linear turn weighting, explicit outcome-based weighting, or more granular diagnosis of forced-vs.-free signal mass in generative policy rollouts.
Practically, TurnOPD’s hardware-normalized efficiency is critical as compute budgets become the primary bottleneck for open-ended agentic training—especially relevant as benchmarks shift toward OOD generalization, non-stationary environments, and compositional planning. Theoretically, signal compression at depth and survivor bias suggest that future LLM-based agent training must confront and correct for these structural distortions in distillation signal.
Conclusion
TurnOPD constitutes a principled and practical refinement for on-policy distillation in multi-turn, long-horizon agent settings, leveraging turn-level analysis to implement adaptive rollout truncation and progressive KL loss balancing. Across several benchmarks, it achieves jointly optimal validation accuracy and wall-clock efficiency, robust to changes in agent architecture, environment, or reward structure. The results offer compelling evidence that turn-conditioned supervision and controller-based budgeting are necessary to scale LLM-based agents to longer, stateful, and more challenging interactive domains (2607.05804).
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
What is this paper about?
This paper is about training AI “agents” that work across many steps, like a chatbot that plans, clicks buttons on a website, or searches the web to answer a question. The authors look at a popular training method called on‑policy distillation (OPD), where a smaller “student” AI learns by copying a stronger “teacher” AI while acting in an environment. They show that regular OPD wastes time on the wrong parts of long tasks and misses chances to improve important later decisions. Then they introduce TurnOPD, a smarter way to train that pays attention to each “turn” (think: each move in a multi-step task), making training both faster and better.
What questions were the researchers asking?
They focused on two simple questions:
- Are we spending our training time on the turns that actually teach the student the most?
- When we compare the student to the teacher, are we spreading the learning fairly across early and late turns, or are we teaching the early turns too much and the late turns too little?
How did they do it?
They studied how OPD behaves in long, multi-turn tasks and then designed a turn-aware fix.
First, some plain-language background:
- Student and teacher: The student AI tries tasks on its own; at each turn, the teacher shows the “right” way to continue. The student adjusts to be more like the teacher.
- Turns: Imagine playing a long board game. Each move is a “turn.” Early moves affect everything that happens later.
- Measuring difference: OPD uses a number (you can think of it as a “difference meter”) that tells how different the student’s next action is from the teacher’s. Larger difference means more to learn at that spot.
What they discovered:
- Wasted effort at the tail: In long tasks, many final turns don’t add much good signal—the student and teacher look similar because the context forces predictable text (like repeating formats or boilerplate), even if the student still makes bad high-level choices. So grinding through those tail turns wastes time.
- Early turns hog the learning: Because there are more tokens (words) early on and they show bigger differences at first, the training loss puts most of its weight on the beginning of the task. Later, deeper decisions don’t get enough attention.
They also explain a key effect (in simple terms):
- “Contamination–compression”: As the conversation or action history grows, a lot of what the model writes becomes predictable filler or formatting. That makes the difference meter look small—even when the student still disagrees with the teacher on the important parts. Failed runs can get stuck in loops that look very predictable, which further hides real mistakes.
TurnOPD: Two simple ideas that fix this
TurnOPD adds two “budget controllers” that act like smart coaches:
- Adaptive rollout-depth budgeting
- Analogy: Instead of always playing a full game to the very end, the coach watches where the most useful learning is happening and sometimes stops early to save time.
- How it works: The system runs quick “probe” sessions to see how much useful teaching signal appears at each turn and how many runs even reach that turn. It then chooses how many turns to collect next time:
- If most useful signal is early, it stops earlier.
- If useful signal still exists later (and enough runs reach those turns), it goes deeper.
- It also keeps a safety rule to ensure it still covers most successful completions, so it doesn’t cut off too soon.
- Progressive turn-normalized loss budgeting
- Analogy: At first, the coach focuses practice on basics (early moves). As the player learns, the coach spreads attention more evenly across all moves, including late, tricky ones.
- How it works: Early in training, the loss follows token counts (more words = more weight), which is stable. Over time, it smoothly shifts to give each turn a more balanced share of attention, so deeper decisions get trained better.
What did they find?
They tested TurnOPD on three types of long, multi-turn tasks:
- ALFWorld: A text-based environment where an agent plans and acts in a virtual world.
- WebShop: An agent navigates an online shop to find products.
- Multi-Hop Search: An agent searches and reads across multiple pages to answer a question.
Main takeaways:
- Faster training for the same or better accuracy:
- On ALFWorld (small student), TurnOPD increased accuracy and cut 100 training steps from about 4.42 hours to about 1.93 hours.
- On WebShop, it improved accuracy and reduced time from about 1.57 hours to about 1.24 hours.
- On Multi-Hop Search, it reached higher accuracy sooner (given the same wall-clock time) and was faster per 100 steps (about 2.94 vs. 4.45 hours).
- Better focus on important turns:
- The adaptive depth saved time by skipping low-value tail turns.
- The progressive weighting pushed more learning into later turns, improving deeper decisions.
In short: TurnOPD moved the “accuracy vs. time” curve upward—getting more accuracy in less time than the usual OPD.
Why does this matter?
- Saves compute and money: Training long-horizon agents can be expensive. TurnOPD cuts waste by not over-training low-value tail turns and by distributing learning more fairly across the task.
- Improves decision quality: By giving enough attention to later, harder turns, the agent learns to finish tasks well, not just start them well.
- Broadly useful: The idea of budgeting by turn applies to many multi-step AI agents—planning, browsing, tool use, and beyond.
- Plays well with others: TurnOPD doesn’t replace the core training idea; it improves how we spend time and attention during training, so it can be added to many existing setups.
In everyday terms: TurnOPD teaches AI agents like a good coach—don’t overpractice the easy opening moves, don’t waste time when learning slows, and make sure the tough late moves get the focused practice they deserve.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a single, concrete list of what remains missing, uncertain, or unexplored in the paper, framed to guide follow-up research:
- Lack of convergence and bias analysis: How do adaptive rollout truncation and turn-normalized loss blending affect OPD’s convergence, gradient bias/variance, and stability relative to full-horizon, token-weighted OPD?
- Probe-induced estimation bias: Periodic full-depth probes estimate survivor-weighted KL and success quantiles, but how frequent must probes be to avoid biased or stale depth estimates, especially in non-stationary environments?
- Hyperparameter sensitivity: The method introduces several new knobs (probe frequency, EMA smoothing, coverage quantile p, centroid rounding, H_min/H_max, linear-blend schedule s/e). Their sensitivity and interactions are not systematically studied.
- Generalization beyond three benchmarks: Does TurnOPD transfer to other long-horizon domains (e.g., real-browser tasks like WebArena, complex GUI/OS agents, code and terminal tasks, AndroidWorld/OSWorld), and to noisier or partially observable settings?
- Teacher quality and mismatch: How robust is TurnOPD when the teacher is weak, miscalibrated, highly stochastic, or domain-mismatched, or when the student approaches/ surpasses the teacher (risk of harmful imitation or stalled learning)?
- Rare but crucial deep-turn supervision: Truncation and turn reweighting may under-expose infrequent yet critical late decisions. How to detect and protect rare deep-turn signals without sacrificing efficiency?
- Contamination-compression quantification: The paper theorizes forced/free components with but does not provide practical estimators for or methods to decontaminate KL signals; can we estimate or bound online to correct or reweight supervision?
- Alternative depth selection criteria: The centroid of survivor-weighted KL and a success-coverage floor are heuristic. Would bandit-style selection, marginal utility estimates, uncertainty-aware criteria, or value-aware proxies yield better horizons?
- Loss allocation alternatives: The linear interpolation between token- and turn-normalization is plausible but ad hoc. Are adaptive, outcome-aware, uncertainty-aware, or data-supported weighting schemes (e.g., per-turn confidence intervals, effective sample size, or outcome-separating KL) superior?
- Early-vs-late performance trade-offs: TurnOPD sometimes loses Same-Step accuracy (e.g., Multi-Hop Search). What conditions predict these trade-offs, and can schedules be adapted online to avoid early under-training of shallow turns?
- Interaction with exploration: Truncating rollouts reduces exposure to deep states. Does this hurt exploration or long-run policy improvement, and can targeted deep exploration (e.g., scheduled deep-rollout bursts) mitigate this?
- Mixed on-/off-policy integration: The approach is evaluated with OPD only. How does TurnOPD interact with mixed datasets (e.g., demonstrations, off-policy traces), or with KL-constrained RL fine-tuning (e.g., GRPO, PPO)?
- Robustness to environment stochasticity: Survivor counts, success quantiles, and per-turn KL can be noisy in stochastic environments. Are robust estimators (e.g., bootstrapped CIs, shrinkage, Bayesian updates) needed for stable control?
- Turn definition ambiguity: “Turn” is abstracted as a model–environment exchange. How does the approach extend to hierarchical actions, multi-step tool subcalls, streaming interactions, or environments where “turns” are ill-defined or asynchronous?
- Tokenization and action granularity: The approach relies on token-level KL and counts. How do varying tokenization schemes, long tool call arguments, or action-level representations (vs. tokens) impact the effectiveness of turn-normalized budgeting?
- Teacher query cost modeling: Probes and per-token KL queries drive compute. A principled analysis of teacher compute vs. wall-clock savings (and optimal probe cadence) is missing.
- Safety and regression control: As loss mass shifts to deeper turns, do shallow-turn behaviors regress? Are guardrails (e.g., lower bounds on shallow-turn weighting or regression tests) necessary?
- Failure modes for KL as a proxy: The work assumes reverse-KL is a usable supervision-value proxy despite contamination; alternatives (e.g., Jensen–Shannon, f-divergences, calibrated cross-entropy, or learned signal-to-noise estimators) are not evaluated with TurnOPD.
- Schedule adaptation and auto-tuning: The blend coefficient and rollout depth are scheduled or smoothed but not adapted by performance feedback (e.g., validation gains per unit cost). Can automated controllers learn optimal schedules online?
- Multi-task and transfer settings: The controller is described per task. How does it operate in multi-task training with diverse turn-depth distributions and task-specific horizons? Can it share information across tasks?
- Long-horizon credit assignment: Beyond reweighting KL, can outcome-aware mechanisms (e.g., success-conditioned reweighting, counterfactual relabeling, or long-horizon credit assignment) improve the alignment of deep-turn supervision with end-task success?
- Reliability of success-coverage floor early in training: When success is rare, estimates may be unstable or meaningless. What bootstrapping strategies ensure reasonable depth early on?
- Reproducibility and variance across seeds: Reported variability is over evaluation points, not independent training runs. How stable are gains across seeds and hardware configurations?
- Scaling to larger models and distributed setups: The paper does not characterize TurnOPD’s behavior with much larger students/teachers or in distributed training (e.g., controller coordination, probe scheduling across workers).
- Downstream impact at inference time: The method optimizes training efficiency, but does it change inference-time behavior (e.g., fewer tool calls, shorter plans), and how does that correlate with task success and safety?
- Open-source benchmarks and code: Implementation details (e.g., masking, probe rollout orchestration, environment wrappers) matter for adoption; providing standardized interfaces or releasing code would reduce ambiguity and facilitate replication.
Practical Applications
Immediate Applications
The following applications can be deployed now by integrating TurnOPD’s two controllers—adaptive rollout-depth budgeting and progressive turn-normalized loss budgeting—into existing LLM agent training and evaluation pipelines.
- Sector: Software/AI infrastructure
- Application: Turn-aware OPD plugin for training frameworks
- What it is: A drop-in module for OPD/RLHF stacks (e.g., PyTorch/DeepSpeed/Accelerate-based trainers) that:
- Runs periodic full-depth probes
- Computes survivor-weighted per-turn KL and coverage
- Sets a capped rollout depth Ĥ per training window
- Blends trajectory-level and turn-level KL aggregation over training
- Tools/products/workflows: Trainer callbacks, EMA-smoothed horizon scheduler, per-turn KL dashboards
- Assumptions/dependencies: Access to a stronger teacher; environments exposing turn boundaries and success signals; token-level KL logging; budget for periodic full-depth probes
- Sector: MLOps/Cost governance
- Application: Time-to-accuracy optimization and compute cost control
- What it is: Operational policies to cap rollouts at Ĥ and monitor iso-time accuracy curves, reducing wall-clock and energy
- Tools/products/workflows: “Least-time accuracy” dashboards, carbon/compute budget monitors, depth-cap enforcement in schedulers
- Assumptions/dependencies: Training telemetry (per-step wall-time, KL mass by turn), carbon accounting, buy-in to optimize for accuracy–time rather than only final accuracy
- Sector: E-commerce/Web automation
- Application: Faster training of web navigation agents (WebShop-like)
- What it is: Train shopping/research/checkout agents with adaptive horizons to avoid wasting supervision on late, low-signal turns
- Tools/products/workflows: Browser automation sandboxes, teacher-student distillation runs with turn-aware budgeting, success-coverage target (e.g., 80%)
- Assumptions/dependencies: Scriptable browser environment; teacher policy tuned for target site types; robustness to site variance
- Sector: CX/RPA (Customer Support and Robotic Process Automation)
- Application: Workflow bots for multi-step enterprise tasks
- What it is: Train ticket triage, form-filling, procurement, and KYC agents more economically by focusing supervision on turns that matter
- Tools/products/workflows: Replay of enterprise logs in simulators, turn-level loss blending to correct deep decision points
- Assumptions/dependencies: Data governance for log-based training; teacher availability; accurate success/failure labeling for coverage estimates
- Sector: Knowledge work/Enterprise search
- Application: Multi-hop research assistants (Multi-Hop Search-like)
- What it is: Use TurnOPD to reduce the cost of distilling small students that perform multi-document retrieval, planning, and synthesis
- Tools/products/workflows: RAG pipeline with turn-aware OPD; per-turn KL probes to confirm where supervision adds value
- Assumptions/dependencies: High-quality teacher with tool-use proficiency; reliable retrieval tools; task success labels
- Sector: Embodied simulation (academia/industry labs)
- Application: Efficient training of embodied planning agents (ALFWorld-like)
- What it is: Reduce simulation rollout compute by truncating low-value tails and rebalancing deep-turn learning
- Tools/products/workflows: Sim environments with turn segmentation; training scripts with Ĥ controller; per-turn loss audits
- Assumptions/dependencies: Simulator fidelity; teacher adaptation to sim domain; sufficient probe budget
- Sector: Evaluation/Benchmarking
- Application: Standardize iso-time and per-turn diagnostics for agent training
- What it is: Report accuracy–time frontiers, per-turn KL, and success/failure gap curves alongside final accuracy
- Tools/products/workflows: Evaluation harnesses that log turn-resolved metrics; reproducible reporting templates
- Assumptions/dependencies: Agreement on metrics; modest overhead for probe runs
- Sector: Human labeling/Active learning
- Application: Allocate human review to high-impact turns
- What it is: Use survivor-weighted KL mass m_t to route scarce human feedback to turns where teacher signal is weak or outcome-separating
- Tools/products/workflows: Annotation UIs showing per-turn disagreement; review queues prioritized by m_t
- Assumptions/dependencies: Access to per-turn KL; ability to segment conversations into turns; human labeler availability
- Sector: Safety/QA for agents
- Application: Loop and failure-mode detection using gap inversion signals
- What it is: Monitor success–failure KL gaps (G_t) to identify degenerate, looping contexts and craft mitigation tests/prompts
- Tools/products/workflows: Heatmaps for G_t over training; auto-generated probes at turns with negative gaps
- Assumptions/dependencies: Success labels; logging infrastructure; team capacity to create tests/guards
- Sector: On-device/Edge AI
- Application: Distill large, tool-using teachers into small on-device agents
- What it is: Use turn-aware budgeting to make small models competent on long-horizon tasks with less training compute
- Tools/products/workflows: Teacher service for probes; iterative student training; per-turn reweighting to boost deep decisions
- Assumptions/dependencies: Distillation rights; intermittent access to teacher inference; device-targeted constraints
- Sector: Cloud providers/Platform teams
- Application: Depth-aware job scheduling and SLAs
- What it is: Offer training presets that optimize for least-time accuracy and enforce adaptive rollout depth caps at scale
- Tools/products/workflows: Managed training templates; SLAs framed as Accuracy@Time; budget controllers as first-class APIs
- Assumptions/dependencies: Platform hooks into trainer loops; compatible telemetry; customer acceptance
Long-Term Applications
These opportunities require further research, scaling, or integration beyond standard OPD stacks and controlled environments.
- Sector: Real-world GUI/OS agents
- Application: Turn-aware distillation for desktop/mobile automation (OSWorld/AndroidWorld-like)
- What it is: Train agents to operate complex GUIs with adaptive depth and turn-balanced loss to learn late, critical decisions
- Tools/products/workflows: Robust GUI instrumentation, action validity checks, coverage-based depth floors per app
- Assumptions/dependencies: High-fidelity interaction logs; safety and access controls; generalization across apps
- Sector: Robotics/Autonomy
- Application: Turn-aware OPD in sim2real pipelines for mobile manipulation and navigation
- What it is: Use Ĥ to limit long sim rollouts and reweight deep-turn control decisions before real-world fine-tuning
- Tools/products/workflows: Mixed-modal KL (text+actions), probe episodes in simulation, curriculum over task horizons
- Assumptions/dependencies: Reliable teacher policies; sim2real transfer; safety compliance and validation
- Sector: Multi-agent systems
- Application: Coordination-aware budgeting (per-agent, per-turn) in multi-agent training
- What it is: Extend controllers to allocate supervision where agents’ interactions are most outcome-critical
- Tools/products/workflows: Multi-agent probes; coordination-aware KL decompositions; shared horizon schedules
- Assumptions/dependencies: Multi-agent simulators; stable multi-agent OPD objectives
- Sector: RLHF/Preference learning
- Application: Turn-aware human-feedback budgets
- What it is: Allocate human preferences or critiques to high-impact turns where reverse-KL provides weak or misleading signal (contamination compression)
- Tools/products/workflows: Hybrid teacher/human pipelines; turn-level preference datasets; progressive reweighting
- Assumptions/dependencies: Human-in-the-loop capacity; UI support; reliable turn segmentation
- Sector: Inference-time efficiency
- Application: Adaptive planning horizons at inference
- What it is: An inference analog of Ĥ to truncate tool-use/plan depth when marginal value per step diminishes (compute-for-quality control)
- Tools/products/workflows: Online proxies for supervision value (e.g., uncertainty, self-consistency); early stopping for tool chains
- Assumptions/dependencies: Calibrated uncertainty; safety constraints; product tolerance for variable-depth behavior
- Sector: Objective design/theory
- Application: Free-component-aware divergence objectives
- What it is: New losses that explicitly counter contamination compression by isolating and upweighting disagreement in the “free” distribution component
- Tools/products/workflows: Token partitioning heuristics (forced vs. free), entropy-regularized KL, structure-aware masking
- Assumptions/dependencies: Reliable proxies for forced mass λ(c); task-specific tokenization/structure
- Sector: Data-centric AI
- Application: Turn-aware dataset curation and curriculum
- What it is: Build corpora emphasizing turns with durable outcome-separating signal; schedule horizons as students improve
- Tools/products/workflows: Turn-level metadata in datasets; horizon curricula (TCOD+TurnOPD hybrids); progression rules
- Assumptions/dependencies: Annotated turn boundaries; success labels; accessible teacher runs for reference
- Sector: Privacy-sensitive domains (Healthcare/Finance/Legal)
- Application: Private, efficient distillation behind firewalls
- What it is: Train internal agents on long workflows with minimized teacher queries via adaptive horizons and turn-balanced loss
- Tools/products/workflows: Secure teacher endpoints; on-prem probes; audit logs of depth caps and turn weights
- Assumptions/dependencies: Compliance approvals; restricted data access; high-quality internal teachers
- Sector: Federated/Distributed training
- Application: Turn-budgeted distillation across clients
- What it is: Clients run probes locally; servers aggregate per-turn statistics to set global Ĥ, reducing bandwidth and cost
- Tools/products/workflows: Federated telemetry for turn stats; privacy-preserving aggregation; client-side depth caps
- Assumptions/dependencies: Heterogeneous environments; privacy constraints; robust aggregation schemes
- Sector: Market/Policy
- Application: Standards for reporting accuracy–time frontiers and energy per point of accuracy
- What it is: Procurement/evaluation policies that require iso-time curves and turn-aware diagnostics for agent training claims
- Tools/products/workflows: Benchmark governance (e.g., “Accuracy@Time” badges), grant criteria tied to efficiency
- Assumptions/dependencies: Community consensus; regulator and funder buy-in; reproducible measurement protocols
- Sector: Tooling/Products
- Application: TurnOps observability suites for agent training
- What it is: Commercial tools offering real-time per-turn KL, success–failure gap heatmaps, horizon controllers, and alerts
- Tools/products/workflows: SaaS dashboards; trainer SDKs; policy engines for depth and loss weights
- Assumptions/dependencies: Ecosystem adoption; integration with popular training stacks; data-sharing agreements
Cross-cutting assumptions and dependencies
- Requires a stronger, task-specialized teacher policy and permission to distill it.
- Environments must expose turn boundaries, success/failure labels (or reliable proxies), and enable periodic full-depth probes without leaking bias.
- Reverse-KL remains the supervision signal; while TurnOPD mitigates its pathologies (e.g., contamination compression), extreme forced-structure tasks may need objective refinements.
- Controller hyperparameters (probe cadence, EMA smoothing, coverage floor p, blend schedule) must be tuned to task and compute budgets.
- Benefits are largest for long-horizon, multi-turn tasks; short or single-turn tasks may see marginal gains.
Glossary
- 2WikiMultiHopQA: A multi-hop question answering dataset used to evaluate multi-step reasoning in information retrieval tasks. "including PopQA~\citep{mallen2023popqa}, NQ~\citep{kwiatkowski2019natural}, 2WikiMultiHopQA~\citep{ho2020twowiki}, HotpotQA~\citep{yang2018hotpotqa}"
- Adaptive rollout-depth budgeting: A strategy that dynamically chooses how many turns to collect in each training rollout based on signal quality. "adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length"
- ALFWorld: An embodied text-based planning benchmark for agent evaluation. "Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD."
- Avg@4: An accuracy metric computed as the average over the last four evaluation points. "Accuracy for each method is reported as the average accuracy over the last four evaluation points (each point is calculated using Avg@4) before the corresponding cutoff."
- Contamination-compression: A mechanism where context-forced tokens compress observable KL, obscuring true policy disagreement at deeper turns. "We formalize one contamination-compression mechanism for the mismatch phenomenon of OPD in long-horizon agent tasks, and demonstrate the existence of a potentially optimal rollout horizon."
- Coverage-side lower bound: A constraint ensuring rollouts are long enough to capture a target fraction of successful trajectories. "To prevent overly aggressive truncation, the coverage component sets a minimum rollout depth based on successful completions:"
- Exponential moving average: A smoothing technique that updates a running estimate by weighting recent observations more heavily. "and applies exponential moving average smoothing:"
- GRPO: A reinforcement learning approach used to train teacher models from which students are distilled. "Students are distilled from stronger, task-specialized GRPO-trained teachers"
- HotpotQA: A multi-hop question answering dataset requiring reasoning over multiple pieces of evidence. "including PopQA~\citep{mallen2023popqa}, NQ~\citep{kwiatkowski2019natural}, 2WikiMultiHopQA~\citep{ho2020twowiki}, HotpotQA~\citep{yang2018hotpotqa}"
- KL-constrained policy optimization: A view of training where policy updates are regularized by Kullback–Leibler divergence limits from a reference policy. "Later work viewed OPD as KL-constrained policy optimization, with the teacher--student log-ratio as a token-level reward"
- Least-Time: An evaluation protocol comparing methods at the minimal wall-clock time needed to reach a set number of steps. "We demonstrate that TurnOPD achieves the best Least-Time accuracy on tested benchmarks, advancing the accuracy--time frontier."
- Multi-Hop Search: A benchmark setting for multi-turn, tool-using language agents performing multi-step search and reasoning. "Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD."
- NQ: The Natural Questions dataset for open-domain question answering. "including PopQA~\citep{mallen2023popqa}, NQ~\citep{kwiatkowski2019natural}, 2WikiMultiHopQA~\citep{ho2020twowiki}, HotpotQA~\citep{yang2018hotpotqa}"
- Off-policy data: Data collected from a policy different from the one being optimized, often used to augment training. "GKD mixes on- and off-policy data and explores various divergence objectives"
- On-policy distillation (OPD): Training a student by matching a teacher on trajectories generated by the student’s current policy. "On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training."
- PopQA: A question answering dataset focusing on popular knowledge, used to evaluate multi-hop search agents. "including PopQA~\citep{mallen2023popqa}, NQ~\citep{kwiatkowski2019natural}, 2WikiMultiHopQA~\citep{ho2020twowiki}, HotpotQA~\citep{yang2018hotpotqa}"
- Probe rollouts: Full-length evaluation rollouts periodically collected to estimate turn-level statistics for control decisions. "using periodic full-length probe rollouts"
- Progressive turn-normalized loss budgeting: A loss-weighting scheme that gradually shifts from token-count weighting to more balanced per-turn supervision. "and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision."
- Quantile: A statistical measure indicating the value below which a certain percentage of observations fall. "the -quantile of the success-conditioned completion depth"
- Reverse KL: The Kullback–Leibler divergence computed as D_KL(student || teacher), used as a supervision objective. "We use reverse KL as the OPD objective:"
- Rollout horizon: The maximum or chosen length (in turns) of an interaction trajectory collected during training. "full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision"
- Rollout-depth controller: A mechanism that selects how deep to roll out trajectories during training to balance cost and signal. "The adaptive rollout-depth controller dynamically selects rollout length via periodic probes"
- Same-Step Avg@4: The Avg@4 metric evaluated after an equal number of training steps across methods. "linear KL blending improves Same-Step Avg@4 by reallocating supervision toward deeper turns."
- Survivor-weighted: Weighting signals by the fraction of trajectories that reach a given turn, to account for attrition with depth. "guided by survivor-weighted KL and coverage thresholds"
- Teacher entropy: The uncertainty of the teacher model’s token distribution at a given turn. "the left panel shows smoothed per-turn teacher entropy"
- Teacher--student log-ratio: The difference in log probabilities between teacher and student distributions, usable as a reward or training signal. "with the teacher--student log-ratio as a token-level reward"
- Trajectory-level normalization: Aggregating loss uniformly over tokens in a trajectory, which can overemphasize shallow turns. "Trajectory-level normalization gives uniform token weights, concentrating the KL signal on easy, shallow turns and starving deep, informative ones once the model learns basic interaction patterns."
- Turn-aware: Accounting explicitly for decision turns in analysis or training objectives for multi-turn agents. "A turn-aware perspective on agent OPD."
- TurnOPD: The proposed method that budgets rollout depth and loss allocation at the turn level for efficient distillation. "we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents."
- WebShop: A web-based navigation and shopping benchmark for evaluating language agents. "Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD."
