PivotOPD: Multi-Turn agent training for recovery and prevention of pivotal errors

• PivotOPD is a learning framework for multi-turn agents that identifies and mitigates pivotal mistakes to improve task performance.
• The framework uses preventive distillation at pivotal states and recovery distillation from post-mistake states, enhancing the agent's ability to recover from errors.
• PivotOPD improved performance across various benchmarks, with notable success rates on environments like ALFWorld and WebShop, demonstrating its effectiveness in practical applications.

PivotOPD is an on-policy distillation framework for training multi-turn agents to prevent and recover from pivotal mistakes. A pivotal mistake is an action that increases the remaining optimal task distance, formally satisfying $L(s_{t+1})>L(s_t)$, where $L(s_t)$ denotes the length of the shortest remaining trajectory that can complete the task from state $s_t$. PivotOPD combines group-based reinforcement learning with preventive distillation at the pivotal turn and recovery distillation from the post-mistake state. The teacher, pivot detection, and recovery procedures are used only during training; inference uses the resulting student policy without additional hints or teacher computation (He et al., 30 Sep 2026).

1. Motivation and problem formulation

In ordinary on-policy distillation (OPD), the student generates a trajectory and a teacher supplies token-level supervision on the responses that the student actually sampled. This is effective when the student visits states for which useful teacher behavior is represented in the sampled data. Multi-turn interaction creates a more difficult setting: an incorrect action changes the environment state, which changes all subsequent observations, contexts, and admissible actions. Later supervision is therefore conditioned on a state that may be substantially different from the state that would have followed a correct decision.

A student-generated mistake can consequently produce compounding errors. The teacher’s original continuation may no longer apply, recovery actions may have very low probability under the student, and every trajectory in a group may fail. In the latter case, group-relative reinforcement learning produces no useful distinction between actions. PivotOPD addresses this failure mode by treating prevention and recovery as separate learning problems.

The framework operates on multi-turn trajectories

$\tau=\{(o_t,y_t)\}_{t=0}^{T-1},$

where $o_t$ is an observation, $y_t$ is the student response, and $a_t=\operatorname{act}(y_t)$ is the committed environment action. At each turn, the action changes the latent environment state from $s_t$ to $s_{t+1}$. The student context at turn $t$ is

$L(s_t)$0

PivotOPD augments the reinforcement-learning objective with supervision targeted at the first action that makes the task harder or unsolvable and with teacher-generated continuations from the resulting state.

2. Pivotal mistakes and recoverability

A turn $L(s_t)$1 is pivotal when the committed action increases the shortest remaining task distance:

$L(s_t)$2

If the task becomes impossible, $L(s_t)$3. This definition includes actions that make completion impossible as well as actions that preserve solvability but require a longer completion trajectory. A trajectory may contain multiple pivotal turns, including a successful trajectory that eventually recovers.

The pivotal action is distinguished from the state in which it is selected and the state it creates:

• $L(s_t)$4: the state immediately before the mistake;
• $L(s_t)$5: the pivotal or unrecoverable action;
• $L(s_t)$6: the post-mistake state;
• $L(s_t)$7: the context from which recovery must begin.

The motivating analysis used ALFWorld’s symbolic oracle to compute $L(s_t)$8 for Qwen3-8B, Qwen3-30B-A3B, and Qwen3-235B-A22B on 140 held-out tasks. Fifty-nine percent of failed trajectories contained at least one pivotal mistake. For Qwen3-8B, 72 of 111 failed trajectories contained a pivotal turn. The first pivotal mistake typically occurred early, at a median of approximately turn 8–12 in a 30-turn budget, after which models wasted approximately 18–21 additional turns without recovering (He et al., 30 Sep 2026).

Counterfactual replay demonstrated that these failures are frequently recoverable. On the 72 Qwen3-8B failures containing a pivotal turn, unchanged trajectories achieved 8.2% success. Correcting the first pivotal action raised success to 59.0%. Leaving the mistake unchanged but forcing oracle actions for the next two turns yielded 58.3%, and forcing three post-mistake oracle actions yielded approximately 62.2%. Correcting a later non-pivotal turn produced only 17.2% success. These results motivate both preventive training at the pivotal turn and short-horizon recovery training after it.

Standard OPD reduced the overall ALFWorld failure rate from approximately 79% to 56%, but pivotal-turn-associated failures decreased only from 51% to 49%. Although the median probability of the mistaken action fell from 0.999 to below $L(s_t)$9, the probability of the oracle alternative at pivotal states remained below $s_t$0. With groups of eight rollouts, such recovery actions are rarely sampled.

3. Pivot detection and privileged self-teaching

PivotOPD uses a teacher model to inspect each student trajectory. The teacher receives the task, the complete trajectory, its outcome, and eligible turn indices. It selects up to $s_t$1 candidate turns and proposes a gold action $s_t$2 for each. A candidate is treated as pivotal when

$s_t$3

The teacher therefore performs both temporal localization and action diagnosis. It identifies where the student’s behavior diverged from a preferred action and supplies the action used for subsequent distillation.

The action is converted into a privileged hint $s_t$4, approximately of the form: “A reasonable approach at this step is: [action]. Reason from scratch in your own style…”. A frozen copy of the student, rather than necessarily an independently trained external model, is then conditioned on this hint:

$s_t$5

The trainable student receives only the ordinary context:

$s_t$6

This construction preserves the student’s own reasoning and response distribution while injecting teacher-provided action information during training. The hint is not included in the inference prompt.

On ALFWorld, teacher detection identified a pivotal turn within one turn of the oracle-labeled pivotal turn in 77.8% of failed trajectories on average. The corresponding rates were 71.1% for a Qwen3-30B-A3B teacher paired with a Qwen3-1.7B student and 84.4% for a Qwen3.5-122B-A10B teacher paired with a Qwen3-8B student. Random baselines achieved 33.6% and 28.1%, respectively (He et al., 30 Sep 2026).

False positives are possible because the teacher and student may select different but valid actions. PivotOPD mitigates this through action-validity checks and small preventive weights. Recovery supervision is also generated from states actually visited by the student, so the post-mistake training state remains operationally relevant.

4. Preventive and recovery distillation

PivotOPD uses different distillation directions for prevention and recovery.

Preventive distillation

At a detected pivotal turn, the preventive objective is a reverse KL:

$s_t$7

The expectation is under the student distribution. Consequently, preventive distillation reallocates probability among behaviors the student already samples, discouraging the committed pivotal action and favoring the gold-action-conditioned distribution.

For token $s_t$8 in a response $s_t$9, the distillation signal is

$\tau=\{(o_t,y_t)\}_{t=0}^{T-1},$0

At pivotal turns, the preventive advantage is

$\tau=\{(o_t,y_t)\}_{t=0}^{T-1},$1

with the principal ALFWorld, WebShop, and Search-based QA configurations using $\tau=\{(o_t,y_t)\}_{t=0}^{T-1},$2. Preventive distillation is therefore student-supported: it primarily modifies behaviors already represented in the student’s sampled response.

Recovery distillation

After a pivotal action, the teacher supplies recovery actions

$\tau=\{(o_t,y_t)\}_{t=0}^{T-1},$3

Recovery begins from the actual post-mistake context,

$\tau=\{(o_t,y_t)\}_{t=0}^{T-1},$4

At each recovery step, the hinted self-teacher samples a response conditioned on the recovery action hint:

$\tau=\{(o_t,y_t)\}_{t=0}^{T-1},$5

Responses are retained only when they commit a valid, schema-compliant action and do not mention or acknowledge the hint. Accepted recovery actions are executed in a replay copy of the environment, so later recovery states result from the preceding recovery actions rather than from the original erroneous suffix.

Recovery minimizes the forward KL:

$\tau=\{(o_t,y_t)\}_{t=0}^{T-1},$6

Unlike preventive distillation, the forward-KL expectation is under the hinted teacher. It can therefore train responses that the student almost never samples. The recovery advantage for token $\tau=\{(o_t,y_t)\}_{t=0}^{T-1},$7 is

$\tau=\{(o_t,y_t)\}_{t=0}^{T-1},$8

with $\tau=\{(o_t,y_t)\}_{t=0}^{T-1},$9 in the reported configuration.

The distinction between the two objectives is central. Reverse KL is used to suppress a pivotal action in a context the student already visits and samples from. Forward KL is used to transfer recovery behavior from teacher-generated responses when the student assigns negligible probability to the required action.

5. Joint optimization and training procedure

PivotOPD combines reinforcement learning, prevention, and recovery through a token-level PPO objective. Conceptually,

$o_t$0

The token-level advantage is

$o_t$1

With the token-level PPO ratio

$o_t$2

the clipped surrogate is

$o_t$3

where $o_t$4. Training also uses a low-variance KL penalty, token-mean loss aggregation, and gradient clipping at $o_t$5.

The training process has two stages. Stage I trains the Analyzer by supervised fine-tuning on solver-generated diagnostic labels containing a pivotal turn, failure mode, and optional recovery guidance. The accepted SFT sets contain approximately 959–1,719 trajectories per environment. Stage II collects groups of student rollouts, computes group-relative reinforcement-learning advantages, detects pivotal turns, applies preventive distillation to ordinary responses, generates replay-based recovery sequences, and performs a joint PPO update.

The main configurations use eight rollouts per task, learning rate $o_t$6, discount $o_t$7, maximum response length 512 tokens, and maximum recovery sequences of 64 per training step. The principal recovery budgets are $o_t$8 for ALFWorld and $o_t$9 for WebShop and Search-based QA. At test time, the Analyzer, teacher, hints, replay, and recovery generation are removed.

6. Evaluation, results, and limitations

PivotOPD was evaluated on ALFWorld, WebShop, Search-based QA, and SWE-Bench Verified. The principal students were Qwen3-1.7B and Qwen3-8B. The SWE-Bench experiment used Nemotron-3.5-SFT as the student and Nemotron-3-Super as the teacher.

For Qwen3-1.7B, PivotOPD improved over the strongest baseline by 5.5 percentage points on ALFWorld and 5.9 points on Search-based QA. On WebShop, it achieved the best score and strict success rate; relative to RLSD, the score gain was 1.2 points and the strict success-rate gain was 14.1 points. Reported PivotOPD averages were 73.7 on ALFWorld, 44.5 on Search-based QA, 84.4 on WebShop score, and 76.6 on WebShop success.

For Qwen3-8B, reported averages were 93.0 on ALFWorld, 47.4 on Search-based QA, 88.2 on WebShop score, and 81.9 on WebShop success. PivotOPD achieved the best average performance across the three primary benchmarks, with margins over the strongest baselines of at least approximately 1.8 points depending on the benchmark.

On SWE-Bench Verified, PivotOPD increased Nemotron-3.5’s resolve rate by 3.2 percentage points, compared with 0.2 points for standard OPD. This experiment used preventive distillation only because the audited point was the final committed action and provided no subsequent turns for recovery.

Ablations show that prevention and recovery are complementary. On ALFWorld with Qwen3-1.7B, average success was 72.5 for preventive-only training, 64.7 for recovery-only training, and 73.7 for full PivotOPD. Random pivotal turns achieved 62.6, hints without gold actions achieved 71.9, and reverse-KL recovery achieved 64.5. Direct replay of 72 oracle-identified Qwen3-8B pivotal mistakes produced 8.3% recovery for the base model, 20.3% for standard OPD, 45.8% for preventive-only PivotOPD, and 72.7% for full PivotOPD (He et al., 30 Sep 2026).

The method has several limitations. Teacher action and pivot detection errors can produce incorrect targets, and teacher detection agreed with the ALFWorld oracle within one turn in 77.8% of failed trajectories rather than perfectly. False-positive pivots remain possible when multiple actions are valid. Recovery after the first post-mistake turn requires replaying the environment, which may be difficult for stochastic services, live websites, or non-rewindable environments. Recovery budget and recovery weight interact strongly: increasing $y_t$0 increases computation and can destabilize training. On WebShop, increasing $y_t$1 from 1 to 3 reduced final validation score by 7.8 points.

PivotOPD also relies on task-specific pivot certificates, feasibility solvers, or teacher judgments during training. The formal definition based on $y_t$2 is not directly available in most environments, and the deployed method approximates it through teacher analysis. The evaluated tasks use bounded history windows, and the approach has not established performance on physical robots or substantially larger models.

Its defining methodological distinction from ordinary OPD is that it does not merely reweight student-sampled behavior. Preventive distillation acts at the state where the student made the pivotal mistake, while recovery distillation generates teacher-supported responses from the state created by that mistake. PivotOPD therefore treats multi-turn agent competence as the combination of two capabilities: avoiding pivotal mistakes and recovering from them when they occur.