AgentOPSD: Turn-Level Credit Assignment

• AgentOPSD is a critic-free, recursive reinforcement learning method that converts teacher–student log-probability gaps into turn-level evidence for long-horizon, multi-turn agents.
• The method updates a Bayesian-inspired success belief in log-odds space, uses outcome-aligned marginal belief revisions to reweight policy gradients, and requires neither a learned critic nor extra rollouts.
• On Qwen2.5-7B, AgentOPSD achieved 89.1% on ALFWorld, 49.2% on Search-QA, and 90.2% WebShop score, while reducing estimated success loss per additional turn to 0.54 points.

AgentOPSD is a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning. It converts privileged self-distillation signals into turn-level evidence, recursively updates a Bayesian-inspired belief state in log-odds space, and uses marginal belief revisions to reweight policy-gradient updates. The method is designed for long-horizon, multi-turn tasks in which terminal verifier rewards provide insufficient temporal discrimination. AgentOPSD requires neither an additional critic nor extra rollouts and is compatible with standard policy-optimization methods (Wang et al., 6 Aug 2026).

1. Problem setting and motivation

Agentic reinforcement learning differs from single-turn language-model optimization because an agent repeatedly observes an environment, generates an action, receives a new observation, and continues until termination. At turn $k$, the agent samples a complete action

$$a_k=(y_{k,1},\ldots,y_{k,L_k})\sim\pi_\theta(\cdot\mid s_k),$$

where $s_k$ is the visible interaction history and $y_{k,t}$ are the action tokens. The environment returns an observation $o_k$, and the next state is

$s_{k+1}=(s_k,a_k,o_k).$

A trajectory

$\tau=(s_1,a_1,o_1,\ldots,s_K,a_K,o_K)$

receives a terminal verifier reward $R(\tau)\in\{0,1\}$.

The central difficulty is temporal credit assignment. In group-relative policy optimization, including GRPO, a trajectory-level advantage is assigned uniformly to every token and every turn. For a group of $G$ trajectories with binary rewards $R^{(i)}$, the sequence advantage is

$$a_k=(y_{k,1},\ldots,y_{k,L_k})\sim\pi_\theta(\cdot\mid s_k),$$0

where

$$a_k=(y_{k,1},\ldots,y_{k,L_k})\sim\pi_\theta(\cdot\mid s_k),$$1

and $$a_k=(y_{k,1},\ldots,y_{k,L_k})\sim\pi_\theta(\cdot\mid s_k),$$2 is the group reward standard deviation. This scalar cannot distinguish pivotal decisions from routine actions. As the horizon grows, the same outcome signal is distributed over increasingly many actions, diluting the update associated with the decision that determined success or failure.

Privileged self-distillation provides denser supervision by allowing a training-only teacher to access information unavailable to the acting student, such as a retrieved skill or successful-behavior context. However, token-level teacher-student gaps are not necessarily environment-aligned: multiple tokens form one action, and the importance of an action depends on the preceding interaction history. AgentOPSD addresses this problem by aggregating token evidence into complete-turn evidence and measuring each turn by its marginal revision of a recursively maintained success belief.

2. Teacher–student evidence construction

At turn $$a_k=(y_{k,1},\ldots,y_{k,L_k})\sim\pi_\theta(\cdot\mid s_k),$$3, the ordinary student context for token $$a_k=(y_{k,1},\ldots,y_{k,L_k})\sim\pi_\theta(\cdot\mid s_k),$$4 is

$$a_k=(y_{k,1},\ldots,y_{k,L_k})\sim\pi_\theta(\cdot\mid s_k),$$5

where $$a_k=(y_{k,1},\ldots,y_{k,L_k})\sim\pi_\theta(\cdot\mid s_k),$$6 denotes the preceding tokens in the current response. The training-only teacher receives an additional privileged skill $$a_k=(y_{k,1},\ldots,y_{k,L_k})\sim\pi_\theta(\cdot\mid s_k),$$7:

$$a_k=(y_{k,1},\ldots,y_{k,L_k})\sim\pi_\theta(\cdot\mid s_k),$$8

The skill is retrieved from SkillBank using keyword matching. It is supplied during training and removed during inference.

The student and teacher share parameters $$a_k=(y_{k,1},\ldots,y_{k,L_k})\sim\pi_\theta(\cdot\mid s_k),$$9, but the contexts differ. For a generated token, AgentOPSD computes the detached log-probability gap

$s_k$0

A positive $s_k$1 means that privileged, skill-conditioned information assigns greater probability to the sampled token than the ordinary student context does. The complete-turn evidence is

$s_k$2

Thus, $s_k$3 measures how strongly the privileged context supports the complete action rather than an isolated token.

The interpretation is Bayesian-inspired. Let $s_k$4 denote eventual trajectory success. The ideal evidence for action $s_k$5 would be the Bayes factor

$s_k$6

AgentOPSD approximates this quantity through two assumptions:

• The skill-conditioned teacher approximates successful behavior:

$s_k$7

• When success is rare, the unconditioned student is approximately failure-dominated:

$s_k$8

Under these assumptions, $s_k$9 approximates a success-versus-failure likelihood ratio. More generally, it has a pointwise mutual-information-like interpretation:

$y_{k,t}$0

The evidence is not a calibrated probability or an exact Bayes factor in general. Its sign is the principal semantic quantity: positive evidence supports the sampled action under privileged information, while negative evidence indicates disagreement.

Teacher probabilities, evidence values, and subsequent belief computations are detached from the optimization graph. AgentOPSD therefore does not introduce a separate distillation loss; it uses the resulting turn weights to modify the ordinary policy objective.

3. Recursive Bayesian belief state

A local turn evidence value does not indicate whether the turn was pivotal in context. AgentOPSD maintains a recursive belief state that incorporates both the group-level success rate and the history of preceding turn evidence.

The initial belief is the clipped group success rate:

$y_{k,t}$1

Clipping prevents infinite log odds when every trajectory in a group succeeds or fails. Such all-equal groups have zero GRPO-relative advantage.

AgentOPSD maintains an exponentially decayed evidence accumulator:

$y_{k,t}$2

where $y_{k,t}$3 is the evidence-decay factor. The default is $y_{k,t}$4. The log-odds state is

$y_{k,t}$5

The corresponding success belief is

$y_{k,t}$6

The prior $y_{k,t}$7 remains fixed, while accumulated evidence is discounted according to $y_{k,t}$8. When $y_{k,t}$9, evidence is accumulated without decay. When $o_k$0, earlier successes and failures lose influence, preventing permanent saturation caused by early turns.

The quantity used for credit assignment is the marginal belief revision

$o_k$1

Because

$o_k$2

a first-order approximation yields

$o_k$3

For $o_k$4,

$o_k$5

This formulation has several consequences. Evidence has the greatest effect when the current belief is uncertain, because $o_k$6 is largest near $o_k$7. Evidence has little effect when the belief is already close to zero or one. With decay, a new turn must overcome residual evidence from earlier turns. Consequently, a large local teacher-student gap can have limited credit if the trajectory’s belief is already saturated.

The revisions telescope:

$o_k$8

The total belief change is therefore bounded and distributed across turns rather than represented by an unbounded accumulation of local scores.

4. Outcome-aligned turn credit

A successful trajectory should emphasize turns that increase the success belief. A failed trajectory should emphasize downward revisions because those revisions are consistent with the terminal outcome. AgentOPSD aligns belief revisions with the sequence-level advantage:

$o_k$9

For a successful trajectory, $s_{k+1}=(s_k,a_k,o_k).$0, so upward belief revisions are outcome-consistent. For a failed trajectory, $s_{k+1}=(s_k,a_k,o_k).$1, so downward belief revisions become outcome-consistent after sign alignment.

The method distinguishes the direction and magnitude of a revision. The magnitude $s_{k+1}=(s_k,a_k,o_k).$2 indicates how strongly a turn changed the belief, while the sign of $s_{k+1}=(s_k,a_k,o_k).$3 determines whether that change agrees with the final verifier outcome. Using only $s_{k+1}=(s_k,a_k,o_k).$4 would amplify large but outcome-inconsistent revisions.

Within each trajectory, the aligned credits are standardized:

$s_{k+1}=(s_k,a_k,o_k).$5

where $s_{k+1}=(s_k,a_k,o_k).$6 and $s_{k+1}=(s_k,a_k,o_k).$7 are the trajectory-level mean and standard deviation. A bounded multiplier is then defined by

$s_{k+1}=(s_k,a_k,o_k).$8

with $s_{k+1}=(s_k,a_k,o_k).$9 in the main experiments.

The reshaped turn advantage is

$\tau=(s_1,a_1,o_1,\ldots,s_K,a_K,o_K)$0

where $\tau=(s_1,a_1,o_1,\ldots,s_K,a_K,o_K)$1. Since $\tau=(s_1,a_1,o_1,\ldots,s_K,a_K,o_K)$2, $\tau=(s_1,a_1,o_1,\ldots,s_K,a_K,o_K)$3, and $\tau=(s_1,a_1,o_1,\ldots,s_K,a_K,o_K)$4, the multiplicative factor remains positive:

$\tau=(s_1,a_1,o_1,\ldots,s_K,a_K,o_K)$5

Therefore,

$\tau=(s_1,a_1,o_1,\ldots,s_K,a_K,o_K)$6

AgentOPSD can amplify or attenuate a GRPO update but cannot reverse the direction implied by the trajectory-level outcome. Setting $\tau=(s_1,a_1,o_1,\ldots,s_K,a_K,o_K)$7 exactly recovers GRPO. Every valid response token in turn $\tau=(s_1,a_1,o_1,\ldots,s_K,a_K,o_K)$8 receives the same reshaped turn advantage $\tau=(s_1,a_1,o_1,\ldots,s_K,a_K,o_K)$9, preserving token-level policy optimization while imposing environment-aligned credit at the turn level.

5. Integration with policy optimization

Let $R(\tau)\in\{0,1\}$0 be the rollout policy. For token $R(\tau)\in\{0,1\}$1 in trajectory $R(\tau)\in\{0,1\}$2, the policy ratio is

$R(\tau)\in\{0,1\}$3

Let $R(\tau)\in\{0,1\}$4 mask valid response tokens, and let $R(\tau)\in\{0,1\}$5 map token $R(\tau)\in\{0,1\}$6 to its interaction turn. The AgentOPSD clipped objective is

$R(\tau)\in\{0,1\}$7

The reported implementation uses an asymmetric policy clip:

$R(\tau)\in\{0,1\}$8

Other shared settings are a learning rate of $R(\tau)\in\{0,1\}$9, KL coefficient $G$0, dual-clip constant $G$1, gradient clipping $G$2, entropy coefficient $G$3, one PPO epoch per update, FSDP on one node, and training on eight H800 GPUs.

The method has no gradient path through the teacher probabilities, teacher-student gaps, belief recursion, standardization, clipping, or turn weights. Gradients flow only through the ordinary student policy objective. AgentOPSD therefore acts as a critic-free reweighting mechanism rather than as an independently trained value estimator.

A complete training iteration consists of the following operations:

1. Sample tasks and retrieve a training-only skill.
2. Generate $G$4 on-policy trajectories using the ordinary student context.
3. Interact with the environment until termination or the turn limit.
4. Obtain terminal verifier rewards.
5. Compute the group-relative sequence advantage.
6. Run the teacher over each generated response.
7. Compute token gaps and aggregate them into turn evidence.
8. Recursively update $G$5, $G$6, and $G$7.
9. Compute $G$8, outcome-aligned credit, normalization, and bounded turn weights. 10. Assign the turn advantage to all response tokens in that turn.
10. Optimize the clipped GRPO objective.

At inference, AgentOPSD is an ordinary agent-environment interaction process. The privileged skill is absent, the teacher is not run, the belief state is not maintained, and no additional rollouts are required. The additional training cost is one teacher forward pass per sampled trajectory plus scalar recursive computation. This avoids a learned critic and extra counterfactual continuations.

6. Evaluation, results, and limitations

AgentOPSD was evaluated with Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct on ALFWorld, WebShop, and Search-QA. The environments represent household interaction, interactive shopping, and search-based question answering.

ALFWorld includes six household task categories: Pick, Look, Clean, Heat, Cool, and Pick2. WebShop uses 128 fixed validation tasks and reports normalized completion score and exact completion accuracy. Search-QA uses NQ, TriviaQA, PopQA, HotpotQA, 2Wiki, MuSiQue, and Bamboogle; NQ and HotpotQA are in-domain, while the remaining datasets are held out. Retrieval uses E5.

The main reported aggregate results include:

Model	Method	ALFWorld	Search-QA	WebShop score / accuracy
Qwen2.5-3B	GRPO	75.0	36.4	79.8 / 63.3
Qwen2.5-3B	RLSD	79.7	43.4	85.0 / 68.0
Qwen2.5-3B	StepOPSD	73.4	43.7	82.4 / 66.4
Qwen2.5-3B	AgentOPSD	—	46.7	90.4 / 69.5
Qwen2.5-7B	GRPO	81.2	42.0	80.9 / 72.6
Qwen2.5-7B	RLSD	79.7	49.0	89.4 / 82.8
Qwen2.5-7B	StepOPSD	73.4	48.2	87.2 / 78.1
Qwen2.5-7B	AgentOPSD	89.1	49.2	90.2 / 79.7

The principal result is an 89.1% ALFWorld success rate for Qwen2.5-7B. AgentOPSD also exhibits greater robustness to longer horizons. The estimated success loss per additional turn is $G$9 points for GRPO, $R^{(i)}$0 points for RLSD, and $R^{(i)}$1 points for AgentOPSD.

Ablations on ALFWorld with Qwen2.5-7B isolate the method’s components:

Variant	Success
Full AgentOPSD	89.1%
Per-token accumulation instead of turn-level aggregation	85.9%
Raw local $R^{(i)}$2 instead of recursive $R^{(i)}$3	82.8%
$R^{(i)}$4 without outcome sign	80.5%
Remove empirical prior $R^{(i)}$5	78.9%

These results attribute performance gains to turn-level aggregation, recursive history dependence, outcome alignment, and the empirical prior. The best tested value of $R^{(i)}$6 was $R^{(i)}$7. Values of $R^{(i)}$8 in $R^{(i)}$9 changed results by only a few points without a monotonic trend, and the upper policy clip was relatively insensitive across $$a_k=(y_{k,1},\ldots,y_{k,L_k})\sim\pi_\theta(\cdot\mid s_k),$$00, $$a_k=(y_{k,1},\ldots,y_{k,L_k})\sim\pi_\theta(\cdot\mid s_k),$$01, and $$a_k=(y_{k,1},\ldots,y_{k,L_k})\sim\pi_\theta(\cdot\mid s_k),$$02.

AgentOPSD is Bayesian-inspired rather than a calibrated Bayesian estimator. Its interpretation depends on the privileged skill being meaningfully associated with successful behavior. An irrelevant, misleading, or systematically biased skill can produce incorrect turn credit. The method is retrospective and does not identify a true causal counterfactual contribution. Its evidence scale depends on teacher-student discrepancy, the group-rate prior can be noisy for small groups, and belief saturation can suppress later evidence, although decay mitigates this effect.

The teacher requires an additional forward pass, and the approach does not provide a complete solution to environment nonstationarity, hidden recurrent state, or long-horizon causal attribution. The method’s formal interpretation is conditional on fixed token-level contexts and does not establish global convergence for the coupled student, teacher, and on-policy context dynamics.

AgentOPSD has been demonstrated on tasks with discrete textual actions and terminal verification. Extending it to continuous actions, structured tool calls, partially observable environments, multi-agent interaction, or safety-critical control requires a common action representation, reliable privileged feedback, and additional constraints. In such settings, teacher-derived credit would need to be combined with environment validity, safety checks, and trajectory-level outcomes rather than treated as sufficient evidence of causal action value.