Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning

Abstract: Naive on-policy RL is often bottlenecked by the capabilities of the base model --- insufficient coverage can result in an inability to learn from certain rewards. At the same time, rewards in modern post-training systems often come from rich textual sources, e.g. judge models with a rubric of criteria. Can we utilize such structures to provide a rich directional signal that enables improving on otherwise difficult to optimize rewards? In this work, we examine a framework we refer to as targeted guidance that learns from per-turn textual feedback via an on-policy self-distillation objective. Our focus is not only on directly improving hard-to-optimize behaviors, but on how to do so throughout the training process without degrading baseline RL performance. Over a range of settings across an agentic multi-turn software engineering environment, we find success by combining targeted self-distillation with a per-criteria RL reward, and by limiting feedback to turns where the policy specifically fails. We further examine the design choices that lead to stable entropy and reduce oscillatory behavior throughout training.

• The paper introduces targeted self-distillation for stabilizing agentic reinforcement learning whereas on-policy reinforcement learning otherwise fails to learn sparse and rare-to-observe behaviours.
• It achieves significant performance gains on various stylistic criteria and is robust to conditions like training and learning to write poems in coding tasks.
• The additional computational requirement over the base method is primarily one expert forward pass per rollout batch, though memory and communication costs become significant for longer contexts and larger vocabularies.

The paper presents targeted guidance as a method for extending on-policy reinforcement learning beyond the behavioral support of a base LLM while preserving performance on an established primary task. Its central setting is agentic reinforcement learning with multi-turn tool use, where sparse terminal rewards and limited policy coverage can prevent policy-gradient methods from learning behaviors that are initially rare or absent. The proposed solution is to convert textual, per-criterion feedback into a localized self-distillation target and to apply that target only when the current policy fails the corresponding criterion.

Motivation and problem formulation

The paper addresses a specific failure mode of on-policy RL for language agents. Standard policy-gradient optimization increases the probability of actions already sampled in successful trajectories, but it provides little information about how to construct behaviors that the current policy almost never produces. This is especially consequential in multi-turn environments: a tool-call format, stylistic convention, or intermediate action may be required before the agent can obtain a useful terminal reward. If the relevant behavior lies outside the effective support of the policy, repeated scalar rewards do not necessarily create a path toward it.

Modern reward pipelines frequently contain more information than the scalar reward exposed to RL. A judge may evaluate several criteria and produce textual feedback explaining why a rollout failed. The paper treats this feedback as a directional signal rather than discarding it during reward aggregation. Each binary reward component has an associated feedback string, and the feedback is inserted immediately before the agent turn responsible for the failure.

The setting assumes a terminal reward decomposed into weighted criteria,

$R(\tau) = \sum_i \lambda_i R_i(\tau),$

where the primary criterion is typically task completion and auxiliary criteria encode properties such as tool usage, formatting, style, or output length. The authors assume that these criteria are sufficiently independent that an auxiliary behavior can be modified without necessarily damaging coding performance. This assumption is important: the method is designed primarily for targeted auxiliary changes, not for arbitrary transformation of the policy.

Targeted self-distillation

For a failed criterion at turn $t$, the method constructs a feedback-conditioned expert from the current policy itself. If $f_t$ denotes the textual feedback, the expert is

$$\mu(\cdot \mid s_t) = \pi_\theta(\cdot \mid s_t, f_t),$$

while at untargeted states it coincides with the ordinary policy. The live policy is then trained toward the feedback-conditioned distribution through a token-level KL objective. The teacher shares the student’s weights, so the method does not require a separately trained expert or autoregressive generation of replacement trajectories.

The final objective combines GRPO with the targeted distillation term:

$$\mathcal{L}(\theta) = \mathcal{L}_{\mathrm{GRPO}}(\theta) + \lambda \sum_{t \in F} \mathrm{KL} \left( \pi_\theta(\cdot \mid s_t) \;\middle\|\; \operatorname{sg}\bigl(\pi_\theta(\cdot \mid s_t,f_t)\bigr) \right),$$

where $F$ contains agent turns that fail at least one criterion, $\lambda$ is the distillation weight, and stop-gradient prevents the expert target from changing within the update.

Three design choices define the method’s stability profile. First, feedback is inserted locally before the offending turn rather than globally into the entire context. Second, the KL term is masked on all turns that do not contain a relevant failure. Third, the textual guidance is combined with an ordinary RL reward for the same criterion. The distillation term supplies a directional mechanism for reaching low-probability behaviors, whereas the RL term remains responsible for evaluating and refining those behaviors on-policy.

The paper motivates this construction using approximate policy-improvement arguments. A feedback-conditioned expert is useful when it has positive expected advantage under the current state distribution, while its divergence from the live policy should remain limited. Neither condition is guaranteed merely by supplying text. Consequently, the authors propose evaluating candidate feedback strings by comparing the performance of the feedback-conditioned model against the contextless model and retaining prompts that produce an empirical improvement.

The use of reverse KL is not presented as essential. Forward KL, Jensen–Shannon divergence, and a Rao–Blackwellized KL estimator reportedly behaved similarly in preliminary comparisons, although the paper does not provide a systematic study of these alternatives. The implementation uses full-vocabulary logits computed on the trainer side, avoiding a sample-based estimator and reducing inference–training mismatch. The additional computational requirement is therefore primarily one expert forward pass per rollout batch; no teacher rollout is required.

Tool-format correction

The first experiment isolates a concrete coverage problem: introducing a semantic-search tool format that was absent from the supervised fine-tuning distribution. The new format requires an additional parameter specifying the number of lines returned. Standard RL receives a reward for correct tool use, but the base policy does not initially produce the required syntax.

Targeted guidance injects feedback before turns in which the tool is called incorrectly. The feedback-conditioned policy exposes a distribution over the corrected format, and the distillation loss transfers this signal to the live policy. The reported result is qualitative but operationally important: the agent learns the new tool format without regressing on primary coding performance. The localization mechanism is especially relevant because the distillation signal concentrates on tokens where the feedback-conditioned and unconditioned policies disagree, rather than uniformly reshaping the entire response.

This experiment supports the paper’s claim that targeted guidance can address behaviors that are difficult to obtain through scalar on-policy rewards alone. It does not, however, establish that the method scales to arbitrary tool semantics or that the feedback-conditioned expert is reliable when the base model lacks the relevant conceptual knowledge. The demonstrated correction is a formatting behavior whose intended solution can be communicated directly in text.

Optimization of multiple stylistic criteria

The main comparison evaluates a Qwen3-Coder-30B policy in a multi-turn software-engineering harness. The training setup combines GRPO for coding pass-rate and length regularization with several methods for optimizing stylistic criteria. The policy is trained on a single-epoch problem set, with asynchronous inference and training, a learning rate of $3 \times 10^{-6}$, and batches consisting of 32 groups with 32 rollouts per group.

The stylistic criteria include requirements such as using Markdown, describing the problem, ordering content, requesting approval, bolding commands, running commands, and suggesting work. The results show a pronounced advantage for targeted guidance in the joint objective:

Method	Markdown	Solution description	Requirement order	Approval	Bolded commands	Run commands	Suggestions	Coding pass-rate
Baseline	0.00	0.70	0.04	0.00	0.43	0.00	0.03	0.33
GRPO	0.00	0.96	0.48	0.00	0.95	0.00	0.20	0.33
OPSD	0.77	0.88	0.29	0.25	0.90	0.01	0.44	0.30
Online SFT	0.01	0.78	0.29	0.02	0.69	0.00	0.28	0.26
LaTRO	0.00	0.57	0.03	0.00	0.21	0.00	0.07	0.14
Verifree	0.01	0.83	0.26	0.01	0.65	0.00	0.15	0.20
Targeted guidance	0.85	0.95	0.77	0.55	0.99	0.64	0.85	0.33

Targeted guidance achieves the highest reported value on every listed stylistic criterion while matching the $0.33$ coding pass-rate of GRPO and the baseline. The gains are particularly large for criteria that standard GRPO effectively fails to optimize: Markdown rises from $0.00$ under GRPO to $0.85$, command execution from $0.00$ to $0.64$, and suggestions from $0.20$ to $0.85$. The approval criterion improves from $0.00$ to $0.55$, while requirement ordering increases from $0.48$ to $0.77$.

The comparison also clarifies the role of combining RL with distillation. OPSD obtains strong performance on some criteria, including Markdown at $0.77$ and bolded commands at $0.90$, but its results are less consistent and coding pass-rate falls to $0.30$. Online SFT, LaTRO, and Verifree all reduce coding performance, with pass-rates of $0.26$, $0.14$, and $0.20$, respectively. The authors attribute these regressions partly to off-policy teacher-generated actions and, for Online SFT, to continual supervised pressure in an asynchronous training regime. These explanations remain hypotheses rather than controlled causal findings, since the methods differ in more than one implementation detail.

The strongest empirical claim is therefore not simply that textual feedback improves auxiliary behavior. It is that failure-gated, on-policy, token-level distillation combined with per-criterion RL can improve a broad suite of auxiliary objectives without the primary-task degradation observed in several alternative methods.

Multi-turn and interleaved feedback

The paper next evaluates criteria that apply at every agent turn rather than only to a final response. The chosen criterion requires each status report to be written as a poem, a behavior described as substantially outside the base model’s ordinary distribution. This setting tests whether guidance remains stable when feedback is repeatedly interleaved throughout a trajectory.

GRPO improves coding pass-rate but fails to optimize the poem criterion. Targeted guidance learns the required style while largely preserving coding performance. Two ablations identify the importance of the proposed stabilization components. Removing the per-criterion RL reward produces an OPSD variant that learns poems at a reasonable rate but becomes unstable, inconsistently maintains the behavior, and incurs a significant coding regression. Applying feedback on every turn, including turns where the criterion has already been satisfied, also causes a coding regression.

These ablations support a division of labor between the two objectives. Distillation initiates movement toward behavior outside the policy’s effective support, while the corresponding RL reward determines whether the behavior should be retained. Failure gating prevents the auxiliary objective from continuing to perturb already-correct behavior. The experiment also indicates that turn-level localization is not merely a computational convenience: it is a mechanism for controlling interference in multi-turn policies.

Entropy, collapse, and reasoning-token distillation

The paper identifies a sharp increase in policy entropy as an early indicator of collapse. In unstable runs, entropy growth is accompanied by increasing output length and followed by a decline in coding pass-rate. High distillation weights yield faster initial stylistic improvement but subsequently cause collapse in both stylistic and coding rewards. Lower weights improve style more slowly while remaining stable.

The proposed explanation is recursive self-distillation. Because the teacher and student share parameters, an increase in student entropy can also alter the feedback-conditioned teacher distribution. Subsequent distillation may reinforce this broadening, producing a positive feedback loop. The analysis implies that entropy growth itself is not necessarily pathological: some increase is expected when the policy begins assigning probability to low-likelihood behaviors. The failure condition is an abrupt spike indicating excessive or poorly localized exploratory pressure.

A static distillation weight is sufficient to maintain a stable regime in the reported experiments. The paper nevertheless leaves the choice of this coefficient as a practical tuning problem and suggests dynamic weighting as a possible efficiency improvement. No general stability criterion or adaptive schedule is derived.

A second instability concerns oscillations among competing stylistic criteria. When the distillation loss is applied only to final-message tokens, individual criteria improve and later regress as pressure shifts among incompatible or competing requirements. Including thinking tokens in the distillation objective substantially reduces this behavior. The authors report that thinking-token distillation changes the trajectory from oscillatory to comparatively monotonic improvement.

The qualitative explanation is that intermediate reasoning begins to encode multiple stylistic constraints. Without distillation on thinking tokens, the reasoning trace may contain no representation of the style requirements, leaving the final response to satisfy them through unstable local adjustments. With thinking-token distillation, the trace includes explicit reminders concerning summaries, explanations, formatting, and confirmation. In this interpretation, thinking-token KL acts as a mechanism for propagating feedback across the causal computation that precedes the final action.

This result is notable but should be interpreted narrowly. The paper does not establish that distilling into hidden reasoning is universally beneficial, nor does it separate the effect of additional token-level supervision from the effect of increased loss magnitude or sequence coverage. The evidence supports the claim that, in this agentic coding setup, excluding thinking tokens is associated with greater criterion oscillation.

Limitations and open questions

The method requires a designed feedback string for every reward criterion. Its effectiveness consequently depends on prompt engineering and on the policy’s ability to interpret the feedback in context. The judge must produce feedback that is both behaviorally specific and compatible with the policy’s capabilities; poorly targeted feedback can provide a misleading expert distribution and interfere with the primary objective. The paper does not provide a systematic evaluation of feedback quality, automatic feedback generation, or robustness to judge errors.

The empirical scope is limited to multi-turn software engineering with Qwen3-Coder-30B. Most auxiliary targets concern style and formatting rather than core reasoning or task competence. It therefore remains open whether the same stabilization principles apply when feedback concerns algorithm selection, long-horizon planning, factual reliability, or behaviors whose optimization directly conflicts with task success.

The comparisons are also incomplete relative to the broader OPSD literature. Several related methods are not ported into the experimental harness, and the paper does not fully isolate the contribution of each atomic choice: reverse versus forward KL, full-vocabulary versus sampled targets, per-turn versus global context insertion, failure gating, thinking-token coverage, and the additional per-criterion RL reward. The asynchronous training setup further complicates attribution because some baselines use teacher-generated actions that may become stale.

The asymptotic argument is conditional. If all criteria are consistently satisfied, failure gating removes the distillation term and the objective reduces to standard policy-gradient optimization. This supports convergence to the optimal policy only under the stated assumptions, including the existence of such a policy and adequate optimization. If feedback-conditioned and optimal policies continue to differ because of spurious context effects, the method may remain biased before the gating condition is reached. The paper does not prove that the gating mechanism prevents all such residual bias.

Finally, the computational advantage over teacher-generation methods depends on access to full-vocabulary logits and efficient trainer-side kernels. Although the authors estimate the main incremental cost as one additional forward pass, memory and communication costs may become material for longer contexts, larger vocabularies, or more frequent turn-level feedback.

Conclusion

"Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning" (2611.05266) develops a localized self-distillation procedure for converting textual reward feedback into an on-policy exploration signal. Its essential design is to condition a same-weights expert on per-turn feedback, apply full-vocabulary KL only at failed turns, and retain an ordinary RL reward for each guided criterion. Across software-engineering experiments, this combination learns behaviors that standard GRPO fails to discover, reaches substantially higher stylistic-criterion scores, and preserves the reported coding pass-rate at $0.33$. The results further identify distillation-weight control and thinking-token coverage as important stability variables. The principal unresolved issue is whether feedback design and failure-gated self-distillation remain reliable when the auxiliary objectives concern substantive capabilities rather than formatting and style.