TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents

Abstract: Multi-turn agents solve complex tasks through extended sequences of tool interactions before producing a final answer, making credit assignment a fundamental challenge during post-training. Outcome rewards provide reliable supervision for short-horizon reasoning, but become sparse and high-variance as trajectories grow to tens or hundreds of tool calls. They can also be misleading: a failed rollout may contain many useful actions that move the agent closer to the goal, yet outcome-only training assigns them the same negative advantage as the eventual mistake. We propose TRACE (Turn-level Reward Assignment via Credit Estimation), a dense credit-assignment method for agentic reinforcement learning. TRACE represents rollouts as state transitions at tool-call boundaries, obtains gold-answer log-probabilities from a frozen reference model, transforms them into log-ratio state values, and derives per-action rewards as Temporal-Difference changes in those values. This requires no additional critic or process-label training, and its one-step log-ratio TD component telescopes across redundant tool calls. On long-horizon complex search, TRACE substantially improves base-model tool-use ability using pure RL, without a cold-start supervised fine-tuning stage, an agentic mid-training stage, or training on live-web data. On the closed-web BrowseComp-Plus benchmark, it raises Qwen3-4B from $7.2$ to $35.6$ and Qwen3-30B-A3B from $8.4$ to $42.6$. The learned search behavior also transfers to open-web benchmarks, and the learning curves show earlier improvement and faster convergence during RL training.

• The paper introduces TRACE, a critic-free method leveraging a frozen reference model to provide dense, turn-level reward signals.
• It employs temporal-difference increments at tool-call boundaries to propagate credit effectively, enhancing training efficiency.
• Empirical results show significant performance gains on benchmarks like BrowseComp-Plus compared to outcome-only RL approaches.

Turn-Level Reward Assignment via Credit Estimation for Long-Horizon Agents

Introduction and Motivation

TRACE ("Turn-level Reward Assignment via Credit Estimation") addresses a persistent challenge in long-horizon agentic reinforcement learning (RL): effective credit assignment. In complex multi-turn tasks—such as deep web search, extended tool use, or long-form multi-step reasoning—an RL agent must select a sequence of tool calls to gather evidence before producing a final answer. Traditional outcome-based RL assigns a terminal reward after the trajectory finishes, but this supervision is sparse and suffers from high variance with increasing trajectory length. Useful intermediate steps in a failed rollout and redundant or negative actions in a successful trajectory are both treated indiscriminately, reducing the informativeness of policy gradients and increasing sample complexity.

Prior attempts to densify the reward signal—such as process supervision, learned reward models, or intermediate label annotation—require significant engineering overhead or introduce reward drift and undesired biases. TRACE proposes a general, critic-free method that leverages a frozen reference model to infer local progress toward the answer without requiring process labels, strong judges, or policy critics.

Methodology: The TRACE Framework

TRACE operates at tool-call boundaries: after every action and its resulting observation, the agent's trajectory prefix is scored by evaluating the log-probability assigned to the correct (gold) answer by a frozen reference model. This log-probability sequence forms the basis for a log-ratio-based state value, $V(S_k)$, reflecting the relative closure of the agent's initial answer-likelihood gap as it gathers evidence. Credit for each tool transition is then computed as a temporal-difference (TD) increment in these log-ratio state values.

Figure 1: Credit assignment at tool-call boundaries allows dense, per-action TD advantages aligning with incremental progress toward answer prediction.

Unlike classical RL with learned critics, TRACE's per-turn reward is derived directly by temporal-difference propagation of the reference-model answer-likelihood gap, requiring no gradient updates to the value estimator. The algorithm optionally uses multi-step ($K$-step) lookahead with discounted backups to assign delayed credit for tool calls whose effect surfaces later in the trajectory. The final policy objective is a weighted combination of (1) standard outcome-level advantage for correctness (e.g., as in GRPO) and (2) the dense TD advantages for each tool call.

Figure 2: TRACE generates turn-level TD credits by scoring each trajectory prefix's gold-answer likelihood under a frozen model, transforming scores into log-ratio values and forming dense rewards.

Experimental Evaluation

TRACE is evaluated on the BrowseComp-Plus benchmark (closed-corpus, multi-step search) and three open-web deep-research benchmarks, using Qwen3-4B and Qwen3-30B-A3B as base models. The evaluation is strictly controlled, with all agentic RL baselines initialized from identical backbones and trained over the same environment, data, and protocol. No supervised warmup, mid-training, or human-in-the-loop data curation is employed.

TRACE achieves large, consistent gains:

• On BrowseComp-Plus, Qwen3-4B is improved from $7.2$ to $35.6$ and Qwen3-30B-A3B from $8.4$ to $42.6$.
• Learning efficiency is enhanced: TRACE provides earlier and steeper training improvement compared to outcome-only RL objectives, and the trained policy converges at a higher plateau.

  Figure 3: Learning dynamics—TRACE accelerates reward acquisition and evaluation accuracy, outperforming ablation baselines at both model scales.

The robustness of the dense credit signal is further validated by transfer to open-web benchmarks and by ablation studies. By keeping the RL recipe minimal and varying only the credit assignment signal, the authors demonstrate that the dense turn-level reward—not advanced data or backbone—is essential for superior long-horizon tool use.

Ablation Studies and Analysis

TRACE's effectiveness is dissected via ablations:

• Credit assignment formulation: The log-ratio closure of the answer-likelihood gap outperforms both raw log-probability deltas and linearized alternatives, confirming the theoretical motivation for relative progress normalization.
• Reward coefficient: Balancing outcome and turn-level advantages is critical; overly large weights on local TD progress can degrade final outcome alignment, while too small weights underexploit the dense signal.
• TD lookahead horizon ($K$): Moderate lookahead propagates delayed evidence usefully; excessive lookahead injects unrelated noise, while $K=0$ (no TD backup) approximates outcome-only RL.
• Reference model selection: Using the base checkpoint (or a slightly updated model) for answer-likelihood estimation is sufficient; TRACE's efficacy is not sensitive to the specific frozen reference checkpoint.

  Figure 4: Ablations on BrowseComp-Plus show the impact of the turn-level reward weight, TD look-ahead horizon, and reference model checkpoint.

Qualitative analysis of successful and failed trajectories reveals that TRACE's TD reward assigns credit (or blame) to precisely the turns responsible for decisive evidence acquisition or catastrophic trajectory collapse—capabilities entirely unavailable to trajectory-level reward baselines.

Related Work

TRACE advances methods for credit assignment in agentic RL [sutton2018reinforcement, arjona2019rudder] by eliminating the need for learned process critics, step-level labels, or human judges. It aligns with a growing literature of agentic RL with verifiable rewards [shao2024deepseekmath, guo2025deepseekr1], process supervision models [uesato2022process, setlur2024rewarding, wang2024mathshepherd], and agentic tool-use RL on browser and software benchmarks [yao2023react, schick2023toolformer, nakano2021webgpt, deng2023mind2web, feng2025gigpo], but is distinguished by its generality, stability, and hyperparameter robustness.

Implications and Future Directions

Theoretical consequences of TRACE are significant. It demonstrates that answer-likelihood under a frozen reference model is a stable and effective proxy for prefix progress in tasks with compact, verifiable outputs. The telescoping property of the TD log-ratio ensures that redundant or unhelpful tool calls are not spuriously rewarded, and the method sidesteps value drift by eschewing critics.

Practically, TRACE immediately enables more efficient and scalable RL post-training for LLM agents in complex environments where process annotation is infeasible. Its recipe—dense TD reward at tool-call boundaries, anchored to reference-model likelihood closure—improves both sample efficiency and final policy strength in multi-hop QA, web agents, and potentially other modular reasoning-and-acting agents.

Extending TRACE to tasks with long, structured, or subjective outputs will require rethinking the state-value estimator, possibly leveraging execution-based or structured-output proxies, hierarchical goals, or decomposable verifiable subgoals.

Conclusion

TRACE provides an effective, critic-free mechanism for dense credit assignment in long-horizon agentic RL, assigning meaningful reward to individual tool calls via reference-model-based answer-likelihood gaps and TD difference propagation. Without supervised warmup or process labels, it outperforms purely outcome-based RL baselines and competitive agentic search methods on both closed and open web research tasks. This methodological advance promises to improve the efficiency and interpretability of RL-driven language agents in increasingly complex reasoning domains, with further research required to generalize dense prefix reward to settings with long or open-ended outputs.