AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

Abstract: Reinforcement learning (RL) with verifiable rewards constructs trajectory-level advantage estimates, yet it often fails to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks. Recent work introduces privileged self-distillation for credit assignment, providing denser supervision, but it remains unclear how such local signals should represent sequential credit. We propose AgentOPSD, a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence and recursively updates a Bayesian belief state in log-odds space. This yields a principled reweighting scheme that converts sparse outcome supervision into turn-level credit signals and identifies pivotal turns through the marginal belief revision between consecutive states. The method is fully compatible with standard policy optimization and requires neither an additional critic nor extra rollouts. We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B). AgentOPSD outperforms GRPO and strong self-distillation baselines, achieving 89.1% success on ALFWorld with Qwen2.5-7B. Ablation studies attribute the gains to turn-level aggregation and history-dependent recursive belief updates.

• The paper introduces a critic-free method that aggregates teacher–student evidence by environment-aligned turns, recursively updates success beliefs, and reshapes GRPO advantages without changing their direction.
• AgentOPSD achieves 89.1% ALFWorld success with Qwen2.5-7B and 84.4% with the 3B model, while improving Search-QA and WebShop performance across interactive benchmarks.
• The method reduces long-horizon performance degradation to 0.54 percentage points per additional turn versus 2.91 for GRPO, with ablations confirming the value of recursive belief updates, outcome alignment, and empirical priors.

AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

Problem formulation and motivation

“AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning” (2608.05987) addresses credit assignment in long-horizon reinforcement learning for language agents. In environments such as embodied simulation, web navigation, and search-based question answering, an agent produces a sequence of multi-token actions over multiple interaction turns, while the environment typically emits a verifiable reward only after trajectory termination. The central difficulty is that a terminal success or failure does not identify which intermediate decisions were causally important.

The paper situates this problem within group-relative policy optimization, particularly GRPO. For a group of sampled trajectories, GRPO computes an outcome-derived, trajectory-level advantage and broadcasts that scalar across every token in the trajectory. This design is attractive because it avoids a learned critic and additional rollout-based value estimation, but it imposes uniform credit on heterogeneous decisions. A trajectory may contain a single decisive action surrounded by routine operations, or it may contain useful intermediate reasoning despite ultimately failing. Uniform attribution cannot represent either structure.

Self-distillation provides a potentially denser signal. In privileged self-distillation, a student policy evaluates its own sampled actions under ordinary context, while a teacher branch evaluates the same actions using training-only information, such as a retrieved skill description. The teacher–student log-probability gap can reveal whether the privileged information makes particular tokens more likely. However, the paper argues that a local token- or step-level gap is not itself a valid sequential credit signal. It does not account for the evidence accumulated from previous turns, nor does it distinguish a genuinely pivotal decision from a redundant one that merely resembles desirable behavior.

AgentOPSD therefore proposes to transform self-distillation evidence into history-dependent, turn-level credit while preserving the critic-free structure of GRPO.

Core methodological contribution

The method has three principal stages: turn-level aggregation, recursive belief updating, and bounded advantage reshaping.

For turn $k$, the agent generates an action consisting of $L_k$ tokens. The privileged teacher and unconditioned student score the same student-generated action. The token-level contrast is computed as the difference between the student log-probability under ordinary context and the corresponding log-probability under a skill-conditioned context. These detached token-level contrasts are summed within the environment-aligned turn:

$e_k = \sum_{t=1}^{L_k} \delta_{k,t}.$

The resulting $e_k$ is interpreted as turn-level evidence. The aggregation is deliberately performed at the interaction boundary rather than across individual tokens, because the environment responds to a complete action and not to the intermediate tokenization of that action. This design distinguishes AgentOPSD from token-level self-distillation methods and from step-aware approaches that aggregate tokens but still treat each step independently.

The paper gives the evidence a Bayesian interpretation. Let $C$ denote eventual trajectory success. Ideally, the evidence contributed by an action would be the log Bayes factor

$$\log \frac{p(a_k \mid s_k,C)}{p(a_k \mid s_k,\neg C)},$$

equivalently the change in the log-odds of success after observing the action. These outcome-conditional action distributions are unavailable, so AgentOPSD approximates the evidence using the privileged self-teacher contrast. Under the paper’s assumptions, the skill-conditioned branch approximates success-associated behavior, while the unconditioned student distribution serves as a marginal or failure-dominated reference. The approximation is exact only under restrictive conditions, particularly when the success probability is sufficiently small; under weaker assumptions, the contrast is more appropriately interpreted as a pointwise mutual-information-like quantity. Importantly, the authors argue that its sign and ranking remain useful for identifying whether an action increases or decreases support for eventual success.

Recursive Bayesian belief state

The defining mechanism is the recursive update of a trajectory-success belief. The initial belief $B_0$ is set to the clipped success fraction of the GRPO trajectory group:

$$B_0 = \operatorname{clip}(R,\epsilon_0,1-\epsilon_0),$$

where $R$ is the group’s empirical success rate. This initialization grounds the belief in the verifier-derived estimate of task difficulty rather than imposing an arbitrary prior.

AgentOPSD accumulates evidence in log-odds space. With decay factor $\gamma$, the evidence accumulator evolves as

$L_k$0

and the belief is computed from

$L_k$1

The credit assigned to turn $L_k$2 is not the raw evidence $L_k$3, but the marginal belief revision

$L_k$4

This distinction is central. The same teacher–student gap can be highly informative early in a trajectory but redundant after preceding turns have already established strong support for success. Recursive revision captures this dependence on history. With no decay, the method resembles sequential accumulation of log-likelihood evidence; with $L_k$5, older evidence is progressively discounted.

The paper further shows that the first-order effect of a new turn is governed by the logistic sensitivity term $L_k$6. Evidence has its greatest influence when the current belief is uncertain and progressively less influence as the state approaches either zero or one. Thus, AgentOPSD does not simply reward large distillation gaps. It weights them according to both their direction and their effect on the current success belief.

For a trajectory with sequence-level advantage $L_k$7, the raw turn credit is outcome-aligned as

$L_k$8

This alignment ensures that belief revisions are interpreted relative to the terminal verifier outcome. A positive revision on a successful trajectory is consistent with the observed outcome, whereas the same positive revision on a failed trajectory indicates behavior that appeared promising according to the privileged signal but did not produce success. The method therefore uses the outcome to determine optimization direction while using recursive evidence to redistribute update magnitude.

Bounded advantage reshaping

AgentOPSD does not replace the GRPO advantage. Instead, it standardizes turn credits within each trajectory, clips the resulting multiplier to a bounded interval, and interpolates between the original GRPO update and the reshaped turn-level update. If $L_k$9 is the bounded multiplier, the turn-specific advantage has the form

$e_k = \sum_{t=1}^{L_k} \delta_{k,t}.$0

The bound guarantees that the modification remains controlled. Because the multiplier is strictly positive, AgentOPSD preserves the sign of the original GRPO advantage. Setting $e_k = \sum_{t=1}^{L_k} \delta_{k,t}.$1 exactly recovers GRPO, while larger $e_k = \sum_{t=1}^{L_k} \delta_{k,t}.$2 increases the influence of recursive credit. This construction is theoretically conservative: it introduces temporal structure without allowing the privileged teacher signal to reverse the verifier’s optimization direction.

The resulting advantages are assigned to all response tokens within the corresponding turn and optimized with the standard clipped GRPO objective. The method requires neither a learned critic nor additional environment rollouts. Its principal computational overhead is one teacher forward pass per trajectory, together with inexpensive elementwise belief-state operations.

Experimental design

The evaluation covers three interactive environments: ALFWorld, WebShop, and Search-QA. ALFWorld tests embodied household task completion across six task categories, including multi-stage pick-and-place, cleaning, heating, cooling, and object-location tasks. WebShop evaluates interactive product search and attribute-constrained purchasing. Search-QA evaluates search-mediated question answering over seven datasets, including both single-hop and multi-hop questions, with in-domain and held-out components.

Experiments use Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct. The principal comparisons include GRPO, skill-conditioned GRPO, OPSD, GRPO combined with OPSD, Skill-SD, RLSD, SDAR, and StepOPSD. Retrieved skills are available during training but removed at inference for AgentOPSD, allowing the experiments to test whether the skill-conditioned teacher’s information is internalized rather than simply supplied at evaluation.

The reported results support the paper’s central claim. On ALFWorld with Qwen2.5-7B, AgentOPSD reaches 89.1% average success, compared with 81.2% for GRPO, 82.0% for RLSD, 82.8% for Skill-SD, 88.4% for StepOPSD, and 88.4% for the strongest listed non-AgentOPSD comparison in the provided results. On the 3B model, AgentOPSD obtains 84.4%, exceeding GRPO’s 75.0%, RLSD’s 79.7%, and StepOPSD’s 73.4%.

The method also improves cross-environment aggregate performance. With Qwen2.5-3B, it obtains 46.7% on Search-QA and a WebShop score of 90.4% with 69.5% exact accuracy. With Qwen2.5-7B, it obtains 49.2% on Search-QA and a WebShop score of 90.2% with 79.7% exact accuracy. These results are not uniformly dominant on every individual subtask, but the aggregate comparisons indicate that the recursive turn-level construction is more reliable than applying self-distillation gaps as local auxiliary or multiplicative signals.

A particularly strong result concerns horizon robustness. On ALFWorld with Qwen2.5-7B, the authors regress per-subtask success against the mean number of turns in successful episodes. AgentOPSD loses only 0.54 percentage points per additional turn, compared with 2.91 points for GRPO and 3.59 points for RLSD. This result directly evaluates the method’s intended operating regime: performance degrades more slowly as trajectories become longer. The finding is consistent with the claim that uniform advantage broadcasting becomes increasingly problematic as the number of intermediate decisions grows.

Ablation evidence

The ablations isolate the contributions of the method’s main components. The full Qwen2.5-7B ALFWorld configuration reaches 89.1% success. Replacing turn-level accumulation with per-token accumulation reduces performance to 85.9%, supporting the argument that credit should be aligned with environment transitions rather than language-model token boundaries.

Replacing recursive belief revision with the raw local gap produces a larger decline, to 82.8%. This comparison is especially important because it separates the benefit of self-distillation from the benefit of sequential processing. The result indicates that a denser local signal is insufficient when it is not conditioned on the trajectory’s preceding evidence state.

Removing outcome-aligned direction and retaining only the magnitude of belief revision reduces success to 80.5%. The magnitude identifies turns that substantially alter the belief, but it cannot determine whether the revision is compatible with the terminal outcome. AgentOPSD consequently requires both localization and signed alignment.

Removing the empirical success-rate prior is most damaging among the reported ablations, reducing performance to 78.9%. This result suggests that the initial group-level estimate is not merely a numerical stabilization device. It determines the initial log-odds and therefore the sensitivity of early belief updates. The ablations support a decomposition of roles: recursive revision identifies history-dependent importance, signed alignment connects that importance to the verifier, and prior anchoring provides a task- and group-specific reference state.

Hyperparameter sensitivity is comparatively limited. The reshaping weight $e_k = \sum_{t=1}^{L_k} \delta_{k,t}.$3 has the clearest effect, with $e_k = \sum_{t=1}^{L_k} \delta_{k,t}.$4 performing best in the reported sweeps. Evidence decay values from 0.8 to 1.0 and upper policy-clipping values from 0.20 to 0.28 produce smaller variations. The diminished sensitivity on the shorter-horizon Search-QA tasks is theoretically consistent with the method: when little history accumulates, recursion has fewer opportunities to differentiate pivotal from redundant evidence.

Theoretical interpretation and limitations

The paper’s theoretical contribution is best understood as a structured reinterpretation of advantage reshaping rather than a proof that self-distillation gaps are calibrated probabilities. AgentOPSD’s belief state is explicitly described as relative support, not as a calibrated probability of success. The teacher–student contrast approximates an ideal Bayes factor only under assumptions about the relationship between retrieved skills, success-conditioned behavior, and the marginal student policy. These assumptions may fail when skills are incomplete, misleading, or correlated with superficial linguistic patterns rather than causal task progress.

The non-identifiability argument is nevertheless sound at the level of terminal rewards: two trajectories with identical outcomes can have radically different internal contribution structures, so no method using terminal reward alone can uniquely recover turn-level responsibility. AgentOPSD addresses this underdetermination by adding privileged behavioral evidence. It does not solve causal credit assignment in the strict counterfactual sense. Rather, it supplies a computationally efficient heuristic with a probabilistic interpretation and empirical validation.

Several limitations follow. First, the success-associated skill context is retrieved through keyword matching, making performance dependent on the quality and coverage of SkillBank. Second, the method requires privileged training information and an additional teacher computation, which may be costly at larger model scales. Third, the terminal verifier remains the final source of update direction; errors, reward hacking, or partial observability in the verifier can be propagated through the reshaped advantage. Fourth, the use of a single global decay factor may be inadequate for environments with heterogeneous temporal dependencies. Finally, the experiments focus on relatively controlled benchmarks and two Qwen model scales, leaving open the behavior of AgentOPSD under stochastic environments, continuous actions, nonbinary rewards, or substantially longer horizons.

Practical and theoretical implications

Practically, AgentOPSD offers a drop-in modification for critic-free agentic RL. It preserves the infrastructure of GRPO, does not require environment-side reward shaping, and avoids the rollout multiplication associated with Monte Carlo credit methods. This makes it attractive for tool-use agents whose actions are naturally segmented by API calls, browser operations, retrieval queries, or embodied environment steps. The method is also compatible with standard policy clipping and KL regularization, which reduces integration risk in existing RLVR pipelines.

The more consequential implication is conceptual. The paper argues that dense supervision should not be equated with fine-grained credit. A token-level signal becomes temporally meaningful only after it is aggregated according to the environment’s action semantics and interpreted relative to a history-dependent state. This distinction applies beyond self-distillation: any local process signal may require recursive normalization against accumulated evidence before it can serve as a useful advantage.

Future developments are likely to combine AgentOPSD’s critic-free recursion with stronger outcome models. Learned success predictors, process reward models, counterfactual continuation estimators, or state abstractions could replace or calibrate the heuristic teacher contrast. A natural extension would use multiple privileged views, allowing the belief update to integrate independent evidence sources rather than a single retrieved skill. Another direction is adaptive decay, in which $e_k = \sum_{t=1}^{L_k} \delta_{k,t}.$5 depends on estimated state uncertainty, expected remaining horizon, or the semantic dependency between turns. AgentOPSD could also be combined with anchor-state methods such as GiGPO, thereby integrating environment-derived repeated-state advantages with self-distillation evidence.

At larger scales, teacher computation could be amortized through cached logits, lightweight adapters, or a smaller privileged evaluator. More importantly, future work should test whether the observed horizon robustness transfers to realistic software engineering, GUI, and tool-use environments where actions have heterogeneous durations and delayed side effects. Such evaluation would clarify whether recursive belief revision is primarily exploiting benchmark structure or constitutes a general mechanism for long-horizon agent optimization.

Conclusion

AgentOPSD presents a principled method for redistributing sparse terminal supervision across the turns of long-horizon language-agent trajectories. Its essential contribution is to distinguish local self-distillation evidence from sequential credit: token gaps are aggregated at environment-aligned turn boundaries, recursively incorporated into a success belief in log-odds space, and converted into bounded, outcome-aligned advantage weights. Across ALFWorld, WebShop, and Search-QA, the method improves over GRPO and several self-distillation baselines, with particularly strong results on ALFWorld and a substantially lower degradation rate as task horizons increase. The approach does not provide fully causal credit assignment, and its Bayesian interpretation depends on nontrivial assumptions about privileged teacher behavior. Nevertheless, it offers an efficient and theoretically coherent mechanism for introducing temporal structure into critic-free agentic reinforcement learning.