Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents

Abstract: Reinforcement learning with verifiable rewards (RLVR) offers a verifier-bounded performance ceiling for training multi-turn tool-use agents, yet its trajectory-level credit assignment conflates heterogeneous per-turn outcomes into a single reward signal. On-policy distillation provides dense per-token supervision but is either teacher-bounded or prone to gradient concentration collapse. We introduce $\textbf{CrEST}$, a hierarchical credit assignment framework that retains RL's verifier-bounded ceiling while incorporating dense token-level signals from a privileged self-teacher. $\textbf{CrEST}$ resolves credit at two levels: turn-segmented verified advantages address inter-turn dilution, while entropy-gated self-teacher modulation refines intra-turn token contributions. Experiments on BFCL V3 and WildToolBench show that $\textbf{CrEST}$ consistently outperforms both RL and distillation baselines across two model scales, with the largest gains on long-trajectory and strict session-level metrics. Our work demonstrates that the teacher's role in policy optimization can be reduced from determining update directions to modulating update magnitudes, unlocking dense credit assignment without sacrificing the verifier-bounded ceiling.

• The paper introduces CrEST, a hierarchical credit-assignment method that combines independently verified turn-level advantages with entropy-gated self-teacher weighting while preserving verifier-determined update directions.
• CrEST reaches 52.00% average BFCL V3 accuracy for Qwen3-4B and 50.00% for Qwen3-8B, outperforming baselines in 13 of 14 evaluation cells and improving long-context performance most strongly.
• Ablations show that inter-turn segmentation and intra-turn modulation are complementary, while removing either the direction or entropy gate causes major losses, demonstrating the importance of bounded, uncertainty-aware updates.

Motivation and problem statement

Multi-turn multi-step tool-use agents expose a credit assignment failure that single-turn RL does not: within a session, individual turns succeed or fail independently, yet standard RLVR objectives broadcast a single trajectory-level reward across all tokens. The paper frames this as a two-level problem — inter-turn dilution, where heterogeneous per-turn outcomes are averaged into one advantage, and intra-turn uniformity, where pivot decisions and low-entropy formatting tokens receive identical gradient weight. The empirical stakes are high: on WildToolBench, no model exceeds 15% session accuracy, and performance degrades sharply with turn count.

The authors position CrEST against two existing paradigms. RLVR offers a verifier-bounded ceiling but sparse, trajectory-level signal; on-policy distillation (OPD) provides dense per-token supervision via reverse KL against a same-family teacher, but its ceiling is teacher-bounded and it requires tokenizer-matched teachers. On-policy self-distillation (OPSD) removes the external teacher by conditioning the student on privileged information (e.g., ground truth), but is brittle — collapsing within roughly 100 steps without per-token KL clipping due to gradient concentration on a few tokens — and remains teacher-bounded. The central question the paper poses is whether dense token-level signal can be retained without sacrificing the verifier-bounded ceiling.

Method

CrEST (Hierarchical Credit Assignment via Entropy-Gated Self-Teacher) uses a standard on-policy policy gradient with a structured per-token advantage:

$$\mathcal{A}_t = A_{[t]}^{\mathrm{turn}} \cdot \phi_t$$

Inter-turn level. Within a GRPO group of $G$ rollouts, each turn $k$ receives its own verified reward $R_k$, and group-relative advantages are computed independently per turn. A failed turn therefore receives negative advantage regardless of other turns' outcomes. This contrasts with MT-GRPO's cumulative propagation of future-outcome credit to earlier turns.

Intra-turn level. A privileged self-teacher — the same model conditioned on ground-truth context — yields a token-level divergence $\Delta_t$. This is converted into a bounded weight $$w_t = \mathrm{clip}(\exp(\mathrm{sign}(A_{[t]}^{\mathrm{turn}})\cdot\Delta_t), 1-\epsilon, 1+\epsilon)$$, where the sign-alignment construction ensures teacher agreement amplifies updates along the verifier-approved direction for both positive- and negative-advantage tokens.

Stabilization gates. Two composed gates control where and how much the teacher acts:

• Direction gate: suppresses the teacher entirely when $$\mathrm{sign}(A^{\mathrm{turn}}_t)\cdot\Delta_t \leq 0$$, ensuring the teacher never overrides verifier-determined direction.
• Entropy gate: modulates reweighting strength by student surprisal $u_t = -\log\pi_\theta(y_t)$, Z-normalized and passed through a sigmoid, so high-uncertainty content tokens receive amplified modulation while low-entropy format tokens are damped.

The composed factor satisfies three formal properties proven in the appendix: (P1) direction preservation ($\phi_t > 0$ always, so the sign of every token's advantage equals the turn-level sign); (P2) an absolute perturbation bound $$\|\sum_t(\phi_t-1)g_t\| \leq \lambda\epsilon\sum_t\|g_t\|$$ (≈8.4% per-token amplification with defaults); and (P3) sign-consistent amplification ($G$0 when active). Notably, the authors are careful to state these are local, token-wise guarantees — they do not imply aggregate gradient directions, stationary points, or a global verifier-bounded optimum, since reweighted token gradients can cancel differently. Only $G$1 is tuned; $G$2, $G$3, and $G$4 are fixed constants.

Experimental results

Experiments cover BFCL V3 multi-turn (Base, Missing Functions, Missing Parameters, Long Context splits) and WildToolBench, using Qwen3-4B-Instruct and Qwen3-8B with group size $G$5 on 8×H200 GPUs. Baselines span GRPO, MT-GRPO, EnvTuning (process rewards), OPD (with Qwen3-235B-A22B-Instruct / Qwen3-32B as teachers), and OPSD.

CrEST achieves 52.00% average accuracy on BFCL V3 (Qwen3-4B) and 50.00% (Qwen3-8B), versus base-model scores of 22.12% and 33.38%, and reaches best-or-tied results in 13 of 14 cells. On WildToolBench session accuracy — the strictest end-to-end metric — CrEST attains 7.03% (4B) and 9.38% (8B), still far below task accuracy (~48–52%), underscoring how difficult full-session consistency remains. Gains are largest precisely where credit dilution is worst: +7.0 over the strongest baseline on Long Context (4B) and +1.57 on WildToolBench session accuracy (8B).

Two secondary findings are notable. First, RL-based methods systematically outperform distillation-based ones at both scales; OPSD scores 38.75% on 4B BFCL Avg, below even plain GRPO (43.63%), providing direct evidence for the teacher-bounded ceiling claim. Second, within the RL family, finer-grained credit helps monotonically (MT-GRPO 49.25% > EnvTuning 47.25% > GRPO 43.63%), leaving headroom that CrEST fills.

Training dynamics and ablations

Mechanistically, CrEST converges faster than GRPO (0.60 rollout accuracy by step ~20 vs. ~0.57 at step 160) and surpasses the OPSD plateau (~0.49, declining thereafter) by step 20, continuing to ~0.70. Gradient concentration analysis shows OPSD places ~42% of total gradient magnitude on the top-1% of tokens and ~77% on the top-5%, while GRPO is diffuse (top-10% ≈ 31%). CrEST occupies an intermediate regime (top-10% ≈ 57%) — more differentiated than GRPO, far less collapsed than OPSD.

Ablations confirm both levels are necessary and complementary: inter-turn segmentation alone improves 43.63 → 47.88; intra-turn modulation alone reaches 48.75; combined, 52.00 (+8.37). Gate ablations show removing the direction gate costs −5.25 points (largest drop on Missing Parameters, consistent with the teacher overriding verifier signals on ambiguous parameter tokens); removing the entropy gate costs −5.75 (Long Context falls 60.0 → 49.0); removing both collapses to GRPO-level 43.50%. A λ sensitivity analysis identifies 0.3 as optimal — λ=0.5 causes over-concentration and stagnation below the inter-turn-only result.

Limitations and open questions

The evaluation is restricted to two tool-use benchmarks at 4B/8B scale, with a single training seed per method and repeated decodes measuring only decoding consistency rather than training variance. The turn-index alignment assumption — all rollouts in a group share the same fixed session and ordered turns — is specific to these benchmarks and does not extend to variable turn structures, terminal-only rewards, or strongly coupled turns lacking local verification. The entropy gate's use of surprisal as a proxy for token importance is acknowledged as a heuristic that may not capture informativeness in all generation contexts. The formal guarantees are explicitly local: the paper does not establish that aggregate optimization targets or stationary points remain verifier-determined, leaving open whether magnitude-only modulation preserves global optimality properties. Whether adaptive scheduling of λ across training stages, or combining CrEST with finer process-reward designs, yields compounding gains remains untested.

Conclusion

CrEST demonstrates that a privileged self-teacher can be useful without determining update directions: restricting it to uncertainty-gated magnitude modulation atop turn-segmented verified advantages resolves both inter-turn dilution and intra-turn token uniformity, yielding consistent gains over RL and distillation baselines across two model scales, with the largest improvements on long-trajectory and strict session-level metrics.