Graph-Distance Contribution Reward (GDCR)

• The paper introduces GDCR as a method to address misallocated credit in multi-turn search by rewarding steps that retrieve or cite entities near the answer node.
• It leverages a training-time Entity-Relation graph to compute credit scores that decay exponentially with the graph distance, ensuring more precise step-level evaluation.
• Paired with Step Advantage Policy Optimization (SAPO), GDCR demonstrates measurable performance improvements across diverse benchmarks in agentic search tasks.

Searching arXiv for the specified paper to ground the article in the cited source. Graph-Distance Contribution Reward (GDCR) is a step-level process reward for agentic search that assigns credit according to how close newly retrieved and newly cited entities are to an answer node in a training-time Entity-Relation (ER) graph. It is introduced in “Beyond Trajectory Rewards: Step-level Credit Assignment for Agentic Search via Graph Modeling” (Liu et al., 28 May 2026) as a response to the limitations of trajectory-level outcome rewards in long-horizon information-seeking tasks, where different search steps contribute unequally to final success. The method is paired with Step Advantage Policy Optimization (SAPO), which converts GDCR into step-level advantages and combines them with trajectory-level outcome advantages during policy optimization (Liu et al., 28 May 2026).

1. Problem setting and motivation

The central motivation for GDCR is that standard reinforcement learning for agentic search typically uses trajectory-level outcome rewards: the entire search trajectory receives a reward based on whether the final answer is correct, and that reward is broadcast uniformly to all steps. The paper characterizes this as the operating mode of GRPO-style methods and argues that it is too coarse for multi-turn information-seeking processes, because some steps move the agent toward the answer whereas others are redundant, distractive, or logically far from the answer (Liu et al., 28 May 2026).

Within this formulation, the key deficiency is not merely sparse reward, but misallocated credit. A trajectory-level signal can indicate whether the overall search succeeded, yet it cannot identify which intermediate actions were useful. The paper states that such methods “overlook the fact that different steps in a multi-turn search process contribute differently to the final success” (Liu et al., 28 May 2026). GDCR is proposed to address this by assigning step-level credit without relying on costly tree sampling.

The paper also situates GDCR against existing step-level methods that estimate credit by expanding multiple continuations and backpropagating leaf rewards. Although such methods can be accurate in principle, they are described as too expensive for long multi-turn search trajectories with large branching factors. GDCR instead uses a structural prior: if a step retrieves or cites entities that are graph-closer to the answer node, that step should receive more credit (Liu et al., 28 May 2026).

2. Graph-theoretic formulation

The conceptual basis of GDCR is a graph view of knowledge and search. World knowledge is treated as a latent world graph,

$$\mathcal{G}^{\mathrm{world}} = (\mathcal{V}^{\mathrm{world}}, \mathcal{E}^{\mathrm{world}})$$

where nodes are entities and edges are semantic or relational dependencies. For each query $q$, the answer-relevant substructure is modeled as a latent task graph,

$$\mathcal{G}_q = (\mathcal{V}_q, \mathcal{E}_q), \quad \mathcal{G}_q \subseteq \mathcal{G}^{\mathrm{world}}$$

with answer node $v_q^\ast \in \mathcal{V}_q$ (Liu et al., 28 May 2026).

Under this interpretation, agentic search is exploration over the latent task graph. A good search step is one that makes graph progress toward the answer node. Retrieved or cited entities that lie on short paths to the answer should therefore receive more credit than entities that are semantically related but graph-distant (Liu et al., 28 May 2026).

Because the latent task graph is unobservable, the method constructs a training-time ER graph

$$\hat{\mathcal{G}}_q = (\hat{\mathcal{V}}_q, \hat{\mathcal{E}}_q)$$

for each query-answer pair, which serves as a proxy for $\mathcal{G}_q$. The answer entity is explicitly annotated as the answer node $v_q^\ast$ (Liu et al., 28 May 2026). This suggests that the operational success of GDCR depends on whether the proxy graph preserves the answer-directed connectivity structure that the latent task graph is assumed to contain.

3. Formal definition of GDCR

For any entity node $v \in \hat{\mathcal{V}}_q$, let $\hat{\mathcal{P}}(v, v_q^\ast)$ denote the set of reachable paths from $v$ to the answer node. The shortest-path graph distance is defined as

$q$0

where $q$1 is the number of edges in path $q$2 (Liu et al., 28 May 2026).

Each entity receives a contribution score that decays exponentially with distance:

$q$3

where $q$4 is a decay factor. The experiments use $q$5, so distance $q$6 gives score $q$7, distance $q$8 gives score $q$9, and distance $$\mathcal{G}_q = (\mathcal{V}_q, \mathcal{E}_q), \quad \mathcal{G}_q \subseteq \mathcal{G}^{\mathrm{world}}$$0 gives score $$\mathcal{G}_q = (\mathcal{V}_q, \mathcal{E}_q), \quad \mathcal{G}_q \subseteq \mathcal{G}^{\mathrm{world}}$$1 (Liu et al., 28 May 2026).

GDCR is defined over cumulative entity sets in a trajectory:

$$\mathcal{G}_q = (\mathcal{V}_q, \mathcal{E}_q), \quad \mathcal{G}_q \subseteq \mathcal{G}^{\mathrm{world}}$$2

where $$\mathcal{G}_q = (\mathcal{V}_q, \mathcal{E}_q), \quad \mathcal{G}_q \subseteq \mathcal{G}^{\mathrm{world}}$$3 contains entities explicitly cited in thoughts up to step $$\mathcal{G}_q = (\mathcal{V}_q, \mathcal{E}_q), \quad \mathcal{G}_q \subseteq \mathcal{G}^{\mathrm{world}}$$4, and $$\mathcal{G}_q = (\mathcal{V}_q, \mathcal{E}_q), \quad \mathcal{G}_q \subseteq \mathcal{G}^{\mathrm{world}}$$5 contains entities appearing in observations up to step $$\mathcal{G}_q = (\mathcal{V}_q, \mathcal{E}_q), \quad \mathcal{G}_q \subseteq \mathcal{G}^{\mathrm{world}}$$6, with initialization

$$\mathcal{G}_q = (\mathcal{V}_q, \mathcal{E}_q), \quad \mathcal{G}_q \subseteq \mathcal{G}^{\mathrm{world}}$$7

(Liu et al., 28 May 2026).

Newly cited entities at step $$\mathcal{G}_q = (\mathcal{V}_q, \mathcal{E}_q), \quad \mathcal{G}_q \subseteq \mathcal{G}^{\mathrm{world}}$$8 are

$$\mathcal{G}_q = (\mathcal{V}_q, \mathcal{E}_q), \quad \mathcal{G}_q \subseteq \mathcal{G}^{\mathrm{world}}$$9

and their reward is

$v_q^\ast \in \mathcal{V}_q$0

Newly retrieved entities at step $v_q^\ast \in \mathcal{V}_q$1 are

$v_q^\ast \in \mathcal{V}_q$2

and their reward is

$v_q^\ast \in \mathcal{V}_q$3

The final Graph-Distance Contribution Reward is

$v_q^\ast \in \mathcal{V}_q$4

so the step-level signal is the sum of credit from newly retrieved graph-linked entities and newly cited graph-linked entities (Liu et al., 28 May 2026).

This construction makes the notion of a positive step explicit. A step is informative if it retrieves new entities from the search environment that are close to the answer node in the ER graph, or explicitly cites new entities in the thought that were previously observed and are close to the answer node. The paper’s qualitative example contrasts “Christopher Nolan” as high value and close to the answer with “Science Fiction” as semantically related but graph-distant and less useful (Liu et al., 28 May 2026).

4. Construction of the training-time ER graph

The paper describes two synthesis pipelines for constructing the ER graph proxy used by GDCR. For the Chinese dataset, the pipeline is knowledge-graph-centric. It builds a large-scale knowledge graph from Baidu Baike content, uses a random walk strategy to derive a candidate training-time ER graph, and applies three quality checks: distant unreachability checks, intermediate unskippability checks, and connectivity checks. Then GPT-4o generates questions from the validated graph, followed by further obfuscation and filtering (Liu et al., 28 May 2026).

For the English dataset, the pipeline is search-agent-centric. It starts from the Asearcher synthesis pipeline, and after final questions are generated, the model is instructed to build a corresponding ER graph from the supporting statements. The answer must appear exactly as an entity in at least one triple (Liu et al., 28 May 2026).

Once the ER graph $v_q^\ast \in \mathcal{V}_q$5 is constructed, the distance from any entity node to the answer node is computed as the shortest path length in the graph:

$v_q^\ast \in \mathcal{V}_q$6

This shortest-path signal is the basis of GDCR (Liu et al., 28 May 2026).

A plausible implication is that the ER graph plays a dual role: it is both a supervision scaffold for reward construction and a structural approximation to the latent answer-relevant search topology. The paper’s later robustness analysis is consistent with this interpretation, since corruption of the graph degrades performance, especially when the answer node itself is perturbed (Liu et al., 28 May 2026).

5. Integration into Step Advantage Policy Optimization

GDCR is not used as a standalone reward; it is converted into a step-level advantage inside Step Advantage Policy Optimization. For each query $v_q^\ast \in \mathcal{V}_q$7, SAPO samples a group of trajectories

$v_q^\ast \in \mathcal{V}_q$8

Each trajectory receives a trajectory-level outcome reward based on final-answer correctness and format validity, and this is group-normalized into an outcome advantage $v_q^\ast \in \mathcal{V}_q$9 (Liu et al., 28 May 2026).

Given the GDCR sequence for a trajectory,

$$\hat{\mathcal{G}}_q = (\hat{\mathcal{V}}_q, \hat{\mathcal{E}}_q)$$0

SAPO normalizes it within the trajectory and clips it:

$$\hat{\mathcal{G}}_q = (\hat{\mathcal{V}}_q, \hat{\mathcal{E}}_q)$$1

(Liu et al., 28 May 2026).

The final advantage for all tokens in step $$\hat{\mathcal{G}}_q = (\hat{\mathcal{V}}_q, \hat{\mathcal{E}}_q)$$2 is then

$$\hat{\mathcal{G}}_q = (\hat{\mathcal{V}}_q, \hat{\mathcal{E}}_q)$$3

where $$\hat{\mathcal{G}}_q = (\hat{\mathcal{V}}_q, \hat{\mathcal{E}}_q)$$4 controls the step-level reward strength and $$\hat{\mathcal{G}}_q = (\hat{\mathcal{V}}_q, \hat{\mathcal{E}}_q)$$5 scales the contribution by outcome confidence (Liu et al., 28 May 2026). The paper explains this combination as preserving the final-answer objective via the trajectory reward while allowing the GDCR-derived signal to differentiate good and bad steps.

SAPO then uses a GRPO-style clipped objective,

$$\hat{\mathcal{G}}_q = (\hat{\mathcal{V}}_q, \hat{\mathcal{E}}_q)$$6

where $$\hat{\mathcal{G}}_q = (\hat{\mathcal{V}}_q, \hat{\mathcal{E}}_q)$$7, $$\hat{\mathcal{G}}_q = (\hat{\mathcal{V}}_q, \hat{\mathcal{E}}_q)$$8 is the token-level importance ratio, $$\hat{\mathcal{G}}_q = (\hat{\mathcal{V}}_q, \hat{\mathcal{E}}_q)$$9 is the advantage of the step containing token $\mathcal{G}_q$0, and $\mathcal{G}_q$1 are clipping bounds (Liu et al., 28 May 2026).

The implementation logic given in the paper is to maintain cumulative observation and cited entity sets, compute $\mathcal{G}_q$2 and $\mathcal{G}_q$3, evaluate shortest-path distances to the answer node, transform them with $\mathcal{G}_q$4, sum them into $\mathcal{G}_q$5, $\mathcal{G}_q$6, and $\mathcal{G}_q$7, normalize and clip within trajectory, combine with the outcome advantage using $\mathcal{G}_q$8, and optimize with the clipped policy-gradient objective. A central implementation property is that SAPO does not require extra search branches, critic models, or tree expansion (Liu et al., 28 May 2026).

6. Empirical behavior and benchmark results

The paper evaluates SAPO with GDCR on four agentic search benchmarks: BrowseComp, BrowseComp-ZH, xbench-DS, and GAIA, specifically the text-only validation subset with 103 samples (Liu et al., 28 May 2026). The reported results show consistent gains over SFT and GRPO.

For Qwen3-8B, the reported scores are as follows:

Benchmark	SFT	GRPO	SAPO
BrowseComp-ZH	17.9	19.7	22.2
BrowseComp	2.7	4.3	4.9
xbench-DS	18.0	20.0	22.0
GAIA	38.8	51.5	52.4

Under the same rollout budget, SAPO also outperforms ARPO on all four benchmarks, with ARPO reported as 15.9 / 4.1 / 16.0 / 47.3 and SAPO as 22.2 / 4.9 / 22.0 / 52.4 (Liu et al., 28 May 2026).

For Qwen3-30B-A3B-thinking, the reported scores are:

Benchmark	SFT	GRPO	SAPO
BrowseComp-ZH	27.3	33.2	45.7
BrowseComp	13.9	14.9	42.8
xbench-DS	53.0	67.0	75.0
GAIA	57.3	62.1	70.9

The paper provides several pieces of evidence specifically about GDCR rather than only the end-to-end SAPO system. First, correct trajectories reach smaller history-best distances to the answer node than incorrect ones. Second, the GDCR step score is positively correlated with final correctness, with

$\mathcal{G}_q$9

Third, SAPO reduces the average shortest-path distance over the course of a trajectory more than GRPO. Fourth, adding the newly-cited component increases the proportion of retrieved entities later cited. Fifth, ablations show that both GDCR parts matter: removing newly-retrieved reward hurts, removing newly-cited reward hurts, and full GDCR is best or tied best (Liu et al., 28 May 2026).

These results support the interpretation that GDCR is not merely a shaping heuristic attached to SAPO, but a reward construction that correlates with trajectory quality and changes search behavior in the intended answer-directed manner.

7. Scope, limitations, and failure modes

The paper explicitly limits GDCR to complex information-seeking tasks with definitive answers. It requires an answer node in the training-time ER graph and therefore does not directly apply to open-ended generation, subjective analysis, or tasks without a clear endpoint (Liu et al., 28 May 2026).

Performance depends on the quality of the search environment, the document index, and the graph-construction pipeline (Liu et al., 28 May 2026). This dependency is empirically examined through robustness experiments in which the training-time ER graphs are corrupted by node deletion, noisy node injection, and answer perturbation. The reported findings are that SAPO is stable under moderate corruption, stronger noise degrades performance, and answer perturbation is especially harmful (Liu et al., 28 May 2026). The last point is particularly significant because GDCR is explicitly anchored to the answer node; perturbing that anchor distorts the shortest-path signal used for contribution scoring.

The ablation studies further clarify the operational regime of the method. On Qwen3-8B, full SAPO is reported as best overall; removing the newly-retrieved reward lowers performance, and removing the newly-cited reward lowers performance more noticeably on some datasets (Liu et al., 28 May 2026). The step-level weight $v_q^\ast$0 is tested at $v_q^\ast$1, $v_q^\ast$2, $v_q^\ast$3, and $v_q^\ast$4, where $v_q^\ast$5 collapses to GRPO, $v_q^\ast$6 is best overall, and $v_q^\ast$7 is worse than the best setting (Liu et al., 28 May 2026). The distance decay factor $v_q^\ast$8 is tested at $v_q^\ast$9, $v \in \hat{\mathcal{V}}_q$0, and $v \in \hat{\mathcal{V}}_q$1, with $v \in \hat{\mathcal{V}}_q$2 degrading performance because it removes meaningful distance distinction, $v \in \hat{\mathcal{V}}_q$3 best overall, and $v \in \hat{\mathcal{V}}_q$4 producing mixed results that are not as good as $v \in \hat{\mathcal{V}}_q$5 (Liu et al., 28 May 2026).

A qualitative failure case in the paper shows GRPO failing because it does not sufficiently cite retrieved entities, causing the agent to drift to a wrong conclusion, whereas SAPO succeeds by progressively retrieving and citing answer-relevant entities (Liu et al., 28 May 2026). This illustrates a common misconception that retrieval alone is the crucial intermediate behavior. In the GDCR framework, newly retrieved and newly cited entities are both reward-bearing, and the citation component is treated as an essential part of step utility rather than a secondary logging artifact.

In summary, GDCR formalizes step-level credit assignment in agentic search by rewarding answer-directed graph progress in a training-time ER graph. Its defining equations,

$v \in \hat{\mathcal{V}}_q$6

and

$v \in \hat{\mathcal{V}}_q$7

embed a shortest-path prior into process supervision, while SAPO incorporates this signal into a trajectory-level optimization framework through

$v \in \hat{\mathcal{V}}_q$8

(Liu et al., 28 May 2026). The method is presented as a way to obtain step-level credit assignment without tree sampling overhead, with demonstrated gains on BrowseComp, BrowseComp-ZH, xbench-DS, and GAIA.