PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agents

Abstract: On-policy distillation (OPD) is a promising approach for training language agents, providing dense teacher supervision on student-generated trajectories. However, in multi-turn interaction, an incorrect action changes the states the student encounters later, so errors compound across turns. In preliminary experiments across three Qwen3 models (8B to 235B), we find that more than half of the failed rollouts contain a pivotal mistake, an action that moves the agent farther from completing the task, and this mistake typically occurs early. These pivotal mistakes often remain recoverable: guiding the model for only a few turns after the pivotal turn can restore task success. We therefore propose PivotOPD, an on-policy distillation framework that jointly trains the student to prevent pivotal mistakes and to recover from the states they create. At each pivotal mistake, a teacher model provides a gold action and then names a recovery action at each of the next few turns. Preventive distillation uses the gold action with reverse KL to steer the student away from the pivotal mistake, while recovery distillation uses the recovery actions with forward KL to transfer recovery behaviors that the student rarely samples. Against 13 baselines on ALFWorld, WebShop, and Search-based QA, PivotOPD achieves the strongest average performance for both Qwen3-1.7B and Qwen3-8B students, improving over the strongest baseline on ALFWorld by +5.5% with the 1.7B student. The gains also transfer to another model family on the software engineering domain, where PivotOPD raises the resolve rate of a Nemotron-3.5 student on SWE-Bench Verified by +3.2%. Project page: https://research.nvidia.com/labs/lpr/pivotopd/

• The paper introduces PivotOPD, a method that significantly improves agents' ability to recover from pivotal mistakes in multi-turn environments (value-based) by incorporating both preventive across the task sequence.
• PivotOPD addresses the failure of standard OPD methods in multi-turn tasks, particularly in translating errors into functional correct actions.
• On benchmark benchmarks like ALFWorld and WebShop, PivotOPD consistently achieves higher performance, with improvements of up to 8.6% in success rates.

Problem formulation and empirical diagnosis

“PivotOPD: Learning to Recover from Pivotal Mistakes in Multi-Turn Agents” (2609.40285) studies a specific failure mode of on-policy distillation (OPD) for language agents. In a multi-turn environment, an action changes the subsequent state distribution. Consequently, a locally incorrect action can place the agent in states where the teacher’s preferred continuation is no longer directly applicable. The resulting training problem is not merely token-level imitation error; it is a state-distribution mismatch induced by the student’s own actions.

The paper defines a pivotal turn as a turn whose committed action increases the length of the shortest remaining solution trajectory, or renders the task unsolvable. In ALFWorld, this quantity can be computed with a symbolic environment oracle. The authors use this oracle diagnostically, but not during training. Their central empirical claim is that many failed trajectories are determined by an early pivotal mistake and that these mistakes are often still recoverable.

Across Qwen3-8B, Qwen3-30B-A3B, and Qwen3-235B-A22B, 59% of failed ALFWorld trajectories contain at least one pivotal mistake. The first pivotal turn typically occurs at median turn 8–12 in a 30-turn episode, after which the agents waste approximately 18–21 additional turns without recovering. The counterfactual replay analysis is especially informative: among 72 failed Qwen3-8B trajectories containing a pivotal mistake, forcing the oracle action at the pivotal turn increases success from 8.2% to 59.0%. Leaving the mistake intact but forcing the oracle action at the next two turns produces a comparable 58.3% success rate. Thus, recovery after the mistake is nearly as effective as prevention at the mistake itself, at least in this controlled environment.

Figure 1: Pivotal mistakes occur early, remain recoverable through short corrective sequences, and are not substantially repaired by standard OPD.

The comparison with standard OPD motivates the method. In a separate 120-step Qwen3-8B experiment, standard OPD reduces the overall held-out failure rate from 79% to 56%, but the fraction of failures associated with pivotal turns declines only from 51% to 49%. OPD sharply suppresses the probability of repeating the committed mistake—the median probability falls from 0.999 to below $10^{-5}$—yet the oracle action remains below $10^{-2}$ probability at every evaluated pivotal state. With a group size of eight, the student is therefore unlikely to sample the corrective action even once. The paper’s interpretation is that standard OPD learns to avoid some observed errors but does not supply a sufficiently strong learning signal for actions that are rare under the student policy and become relevant only after an error-induced state transition.

PivotOPD

PivotOPD augments group-based RL with targeted preventive and recovery distillation. Its design separates two distributions that standard OPD conflates: the student’s distribution over already sampled responses and a privileged self-teacher distribution that can generate responses toward a named action.

The training procedure has three stages. First, a teacher model reads each rollout together with its outcome and selects candidate turns where the student may have deviated from a desirable trajectory. For every selected turn, it names a gold action. A candidate is treated as pivotal when the student’s committed action differs from this gold action. The teacher then provides a recovery action for each of the next $K$ turns.

Second, preventive distillation is applied at the detected pivotal turn. The frozen student is conditioned on a hint naming the gold action and acts as a privileged self-teacher. The student’s recorded response is re-scored under this hinted distribution, and a reverse-KL-style signal shifts probability toward the gold action. Because the loss is evaluated on a response the student already generated, it primarily reweights the student’s existing support.

Third, recovery distillation generates new responses from the hinted self-teacher at the post-mistake state. The hint names the recovery action, but the response is trained without the hint. The procedure therefore transfers a recovery behavior into the unprivileged policy at a state caused by the student’s own mistake. For later recovery turns, the method executes the generated recovery action in a replayed copy of the environment, queries the teacher again, and continues from the resulting state.

Figure 2: PivotOPD detects pivotal turns, applies reverse-KL preventive distillation at those turns, and applies forward-KL recovery distillation on teacher-generated continuations from post-mistake states.

Both distillation mechanisms are integrated with PPO. Preventive distillation contributes a token-level advantage to the original rollout, while recovery distillation supplies a clipped token-level advantage on generated recovery sequences. Group-relative RL remains active on the original rollouts. At inference, the teacher, hints, and recovery-generation process are absent; PivotOPD changes only training.

The distinction between the two KL directions is theoretically central. Reverse KL on student-generated samples cannot reliably increase the probability of an action that the student almost never samples. If the student assigns probability $p(a^*)$ to a recovery action, the expected update from student-sampled signals scales with $p(a^*)$ and vanishes as $p(a^*)$ approaches zero. In contrast, recovery distillation samples under the hinted distribution $q$, where the recovery action has substantially greater mass. Its update on the recovery action is proportional to the difference between $q(a^*)$ and $p(a^*)$, subject to the clipping bound. The method thus obtains training examples precisely where on-policy RL and reverse-KL distillation are data-starved.

This argument depends on the privileged self-teacher placing meaningful probability on the named action and on the recovery response passing the method’s action-validity and hint-leakage filters. It is therefore not a claim that forward KL is universally preferable; the proposed division of labor is conditional on whether the target response is already present in the student-generated data.

Experimental design

The main experiments cover ALFWorld, WebShop, and Search-based QA. Students are Qwen3-1.7B and Qwen3-8B. External-teacher configurations use Qwen3-30B-A3B for the smaller student and Qwen3.5-122B-A10B for the larger student. All methods use the same training data, 160 training steps, eight rollouts per task, three random seeds, and validation-based checkpoint selection. The comparison includes 13 baselines spanning GRPO, OPD and self-distillation, turn-level OPD variants, skill-based methods, and pivotal-turn RL.

Pivot detection is imperfect but nontrivial. On ALFWorld, at least one detected pivotal turn falls within one turn of the oracle-labeled pivotal turn in 77.8% of failed trajectories on average, compared with 28.1–33.6% for matched random baselines. The agreement rates for the two external teachers are 71.1% and 84.4%. The remaining errors matter: teacher-selected actions can be suboptimal, equally valid alternatives can be incorrectly marked as pivots, and free-form actions in open action spaces can produce false mismatches.

Main results

PivotOPD achieves the strongest unweighted average on all three principal benchmarks for both student sizes. The main aggregate results are:

Student	Benchmark	PivotOPD	Strongest baseline	Improvement
Qwen3-1.7B	ALFWorld	73.7	68.2	+5.5
Qwen3-1.7B	Search-based QA	44.5	38.6	+5.9
Qwen3-1.7B	WebShop score	84.4	83.2	+1.2
Qwen3-1.7B	WebShop success	76.6	68.0	+8.6
Qwen3-8B	ALFWorld	93.0	90.8	+2.2
Qwen3-8B	Search-based QA	47.4	45.6	+1.8
Qwen3-8B	WebShop score	88.2	85.5	+2.7
Qwen3-8B	WebShop success	81.9	79.7	+2.2

The largest gains occur on tasks with low base success, particularly the smaller student on ALFWorld. For Qwen3-1.7B, the method reaches 93.7% on Clean, 72.6% on Heat, and 73.9% on Cool. These task types are precisely those where successful on-policy rollouts are scarce, limiting the usefulness of group-relative advantages. Recovery distillation provides a signal even when all group members fail, because the teacher can generate a corrective action from the state actually reached by the student.

WebShop separates partial progress from complete task success. With Qwen3-1.7B, PivotOPD improves over RLSD by only 1.2 percentage points in normalized score but by 14.1 percentage points in success rate. This gap supports the paper’s interpretation that recovery distillation converts partially completed trajectories into fully successful episodes rather than merely improving intermediate behavior.

Figure 3: In the self-distillation setting, PivotOPD remains the strongest method across ALFWorld, Search-based QA, and WebShop, exceeding the strongest baseline by 3.9% on average.

The self-distillation experiment removes the stronger external teacher by using the student as its own teacher for PivotOPD and the teacher-based baselines. PivotOPD remains best on all three benchmarks and exceeds the strongest baseline by at least 1.5% on each. Relative to the external-teacher configuration, however, its ALFWorld performance is 10.2 percentage points lower, while the degradation is smaller on Search-based QA and WebShop success. This result supports the claim that intervention location and KL direction are important independently of teacher scale, but it also demonstrates that teacher quality remains consequential.

The cross-family software-engineering experiment provides a limited transfer test. On SWE-Bench Verified, Nemotron-3.5-SFT is trained with Nemotron-3-Super. PivotOPD increases resolve rate by 3.2 percentage points, compared with 0.2 points for standard OPD. However, the SWE-Bench configuration uses $K=0$: pivot detection audits the final committed action, leaving no subsequent recovery turn. Consequently, this experiment evaluates preventive distillation rather than the full recovery mechanism. The paper explicitly leaves earlier-turn pivot detection and recovery in long, containerized software-engineering episodes unresolved.

Recovery behavior and ablations

The recovery analysis directly tests whether the method changes post-error behavior rather than only improving aggregate rewards. The authors replay 72 oracle-identified ALFWorld pivotal mistakes and allow each trained policy to continue autonomously. The final recovery rates are 8.3% for the base model, 20.3% for standard OPD, 45.8% for preventive-only training, and 72.7% for full PivotOPD. PivotOPD improves recovery on 60 of the 72 pivotal mistakes and worsens none. This is stronger evidence than an aggregate benchmark score because all policies face the same error-induced prefixes.

Figure 4: After an identical pivotal mistake, the base model continues toward failure while PivotOPD selects a sequence that restores task completion.

The ablations show that prevention and recovery are complementary. On ALFWorld with Qwen3-1.7B, full PivotOPD obtains 73.7% average success, compared with 72.5% for a strengthened preventive-only variant and 64.7% for recovery-only training. Replacing pivotal turns with randomly selected turns reduces performance to 62.6%, establishing that the temporal localization of supervision is necessary. Removing the named gold action from hints yields 71.9%, indicating that generic reflection is insufficient on task types requiring a specific action. Replacing forward-KL recovery with reverse-KL recovery yields only 64.5%, consistent with the theoretical claim that student-sampled reverse-KL updates cannot efficiently populate low-probability recovery modes.

The recovery budget is benchmark-dependent. $10^{-2}$0 is selected for WebShop and Search-based QA, whereas $10^{-2}$1 is selected for ALFWorld. Additional recovery turns are not monotonically beneficial: on WebShop, increasing the budget from one to three turns lowers the final validation score by 7.8%, and the difference across budgets reaches 19.5 percentage points. This sensitivity reflects both the depth of the required recovery chain and the interaction between $10^{-2}$2 and the recovery weight.

Figure 5: Recovery-budget ablations show that the optimal number of guided turns is benchmark-dependent and that excessive recovery supervision can reduce performance.

The computational cost is similarly sensitive to the budget. On ALFWorld with Qwen3-1.7B, preventive supervision adds 4.7% over GRPO, and one recovery turn raises total overhead to 12.4%. Two recovery turns raise overhead to 94.2%, primarily because later recovery states require environment replay. Three turns raise it to 112.3%. Thus, the method’s practical cost is modest for $10^{-2}$3 but can become substantial when multi-step recovery requires repeated environment execution.

The divergence analysis provides an empirical correlate of the theoretical mechanism. After a pivotal mistake, the preventive-only policy remains at least twice as far from its privileged self-teacher as the full method at every displayed recovery turn. Recovery distillation rapidly reduces this divergence, and the effect persists beyond the single explicitly trained recovery turn in the analyzed run. This suggests that learning a corrective action can alter subsequent trajectory behavior rather than merely fitting one isolated state.

Limitations and open questions

The method assumes access to a sufficiently reliable teacher capable of identifying pivotal turns and naming executable corrective actions. Teacher–oracle agreement reaches 77.8% within a one-turn tolerance on ALFWorld, but this leaves a substantial error rate. The method can also label reasonable alternative actions as pivotal when they differ textually from the teacher’s action. Its performance under weak, miscalibrated, or systematically biased teachers is not established.

Recovery beyond the first turn depends on exact environment replay. ALFWorld, WebShop, and Search-based QA support this procedure, but live websites, stochastic environments, and long-running software systems may not reproduce the same observation after replay. The paper’s SWE-Bench result avoids this issue by setting $10^{-2}$4, so it does not demonstrate recovery distillation in the most complex domain evaluated.

The diagnostic evidence is strongest in ALFWorld because only that environment provides a symbolic oracle for the remaining optimal trajectory. On WebShop, Search-based QA, and SWE-Bench, pivotality is teacher-defined rather than independently verified. The mismatch between textual action identity and semantic equivalence is especially important in open action spaces, where free-form search queries can be marked as mismatches despite being functionally adequate.

The recovery budget and recovery weight require benchmark-specific validation. The paper does not provide a general estimator for recovery depth, and its theoretical depth result assumes a chain of specific required actions with bounded plain-policy probability. That abstraction does not capture environments with multiple valid recovery paths, stochastic transitions, or delayed effects.

Finally, the main experiments use short history windows—at most five turns—and the analysis does not fully separate model inability to recover from information loss caused by truncated context. The observed wasted turns may therefore reflect both policy failure and partial observability induced by the training configuration.

Conclusion

PivotOPD identifies a concrete deficiency of standard OPD in multi-turn agents: suppressing a sampled mistake does not necessarily teach the policy what to do in the state created by that mistake. Its solution combines reverse-KL preventive distillation at teacher-identified pivotal turns with forward-KL recovery distillation on teacher-generated continuations from post-mistake states. Across three agentic benchmarks, two Qwen3 students, 13 baselines, and an additional Nemotron SWE-Bench experiment, the method obtains the strongest reported aggregate results and substantially improves controlled recovery after identical pivotal errors. The evidence supports targeted post-error supervision as a distinct training signal, while leaving teacher reliability, replay-free recovery, and recovery in long-horizon software agents as unresolved methodological questions.