STAMP: Provenance-Guided Credit Assignment for Deep Search Agents

Abstract: Reinforcement learning for deep-search agents has largely focused on trajectory-level scoring -- outcome correctness, citation-aware rewards, and evidence coverage. Yet the actions that expose supporting documents receive no targeted credit, a gap we call the reward-credit mismatch. We propose STAMP, in which a reference-based verifier judges whether each cited document supports an entity or relation in a training-time evidence graph, and first-exposure attribution traces each supported citation back to the action that first surfaced it. This step credit is injected through sign-preserving advantage modulation, which redistributes advantage across steps without changing the trajectory-level reward or the relative ranking of trajectories within each group. On BrowseComp, BrowseComp-ZH, and xbench-DS, STAMP improves the GRPO baseline by +2.0/+5.5/+3.0 points under matched SFT initialization, training data, and search tools, and composes with both outcome-only and citation-rubric base rewards. Component ablations confirm that the provenance-based credit signal and the sign-preserving advantage modulation each contribute to the gains.

• The paper introduces a provenance-guided credit assignment mechanism that localizes RL rewards to evidence-producing steps, addressing the reward-credit gap.
• It employs reference-based evidence verification, first-exposure attribution, and sign-preserving advantage modulation to enhance deep search agent performance.
• Empirical results show accelerated convergence, improved relation verification rates, and additive gains across multiple deep search benchmarks.

Provenance-Guided Credit Assignment for Deep Search Agents: An Expert Review of "STAMP" (2607.11172)

Motivation and Problem Formulation

The STAMP framework is introduced to address a reward-credit gap in RL training of deep search agents: trajectory-level rewards (outcome correctness, citation-aware rubrics) are insufficiently localized to the specific actions that surface supporting evidence. Empirical analysis demonstrates pronounced credit sparsity—over 83% of steps in correct trajectories produce no evidence, while ~7% of steps in incorrect rollouts still expose entity-level or relation-level evidence. This mismatch occludes optimization signal that could reinforce evidence-producing behaviors.

Figure 1: Accuracy rapidly increases with higher evidence levels, from none to entity-only to relation-verified evidence.

Methodology: Provenance-Guided Step Credit via Advantage Modulation

STAMP implements a provenance-guided credit assignment pipeline comprising three technical core stages:

1. Reference-Based Evidence Verification: Each trajectory is analyzed via an LLM judge against a training-time evidence graph to determine which cited documents support entities or relation constraints.
2. First-Exposure Attribution: Step-level credit is attributed to the action that first surfaces each supporting document (queried or read), establishing temporal provenance without relying on local reward models.
3. Sign-Preserving Advantage Modulation: Step-level credit is injected into the RL signal by modulating the trajectory-level advantage computed by GRPO; modulation guarantees preservation of sign (i.e., positive/negative reinforcement direction) and bounded gradient scaling.

The provenance mechanism converts document-level support into fine-grained, step-level credit, and modulation is applied without altering trajectory-level reward or ranking. This design is orthogonal to trajectory reward formulation and introduces no separate local reward model.

Figure 2: Overview of STAMP: Citation provenance traces verified evidence to its earliest exposure step, and modulation selectively amplifies credited steps.

Experimental Evaluation and Numerical Results

STAMP is evaluated on Qwen3-30B-A3B-Thinking-2507 across three deep search benchmarks: BrowseComp, BrowseComp-ZH, and xbench-DS. All controlled baselines share SFT initialization, RL training data, and search tools, isolating the contribution of credit assignment. When applied to outcome-only GRPO, STAMP yields absolute gains of +2.0/+5.5/+3.0 points on the respective benchmarks, largest for settings lacking dense entity-level supervision. Ablations confirm that verification quality, first-exposure attribution, credit bounding, and negative-advantage attenuation jointly contribute to effect size.

Analysis of Training Dynamics and Attribution Sparsity

STAMP not only improves final accuracy but accelerates convergence and increases relation verification rates throughout RL training. Entity grounding saturates rapidly, while relation verification benefits most from selective reinforcement of evidence-producing steps—a critical property for complex, multi-hop queries. Credit attribution remains highly sparse (∼10–12% of steps credited), focusing learning pressure on informative actions.

Figure 3: STAMP increases outcome reward and relation verification rate while credit remains concentrated on a sparse subset of steps.

Ablation Studies and Design Analysis

Ablations on verification methods reveal that LLM-based judgment yields substantially higher credit quality compared to string matching. Regarding modulation variants, earliest-attribution, bounded step credit, and attenuation on negative-advantage trajectories are necessary for optimal performance; unbounded modulation or credit diffusion (crediting all steps) dilutes optimization signal and degrades results.

Theoretical and Practical Implications

The STAMP framework decouples trajectory-level scoring from step-level credit assignment, utilizing provenance inherent to agentic trajectories (citation logs) and verifiable evidence graphs. This enables deployment of RL agents that can efficiently learn to target evidence recovery—critical for long-horizon reasoning and open-web interaction domains. Because step-level credit is derived from reference-based judgements, it avoids assumptions about reward model granularity and can generalize across substantially different reward designs.

Practically, STAMP's credit assignment pipeline allows for reinforcement at the level of evidence discovery and verification, potentially enabling transfer to tool-augmented agents, iterative information-seeking, and hierarchical search protocols. Its bounded modulation guarantees that optimization pressure remains tractable even for long trajectories.

Future Directions

Extending STAMP to larger-scale models (70B+, MoE), diverse community checkpoints, and more complex action spaces (multi-tool, parallel search) represents a logical progression. Provenance-guided credit could further be integrated with learned attribution models or used to supervise process reward models for stepwise training in emerging agentic RL frameworks.

Conclusion

STAMP delivers a principled, modular solution to the reward-credit mismatch in RL for deep search agents by leveraging citation provenance for verifiable step credit and sign-preserving advantage modulation. Empirical results confirm additive, orthogonal gains relative to canonical reward designs. The provenance attribution mechanism constitutes a robust intermediate between sparse trajectory-level feedback and dense, local reward modeling, and the approach should generalize to broader agentic RL contexts as benchmarks and models scale.