Path Reliance Degree in GraphRAG

• Path Reliance Degree (PRD) is a metric that quantifies the extent to which LLM decoders focus on shortest-path tokens within graph-based retrieval systems.
• It is computed by normalizing attention weights over retrieved subgraph tokens and comparing masses on shortest-path versus non-shortest-path tokens.
• PRD, when combined with semantic alignment scores, aids in identifying hallucination risks and supports improved reliability in knowledge-based question answering.

Path Reliance Degree (PRD) quantifies the attention distribution of LLMs within Graph-based Retrieval-Augmented Generation (GraphRAG) systems, measuring the extent to which the model's decoder over-emphasizes tokens corresponding to shortest-path triples between question and answer entities at the expense of the broader retrieved subgraph context. PRD is both lightweight and interpretable, serving as a mechanistic metric for identifying risk factors associated with hallucinated or unsupported generations in knowledge-based question answering systems (Li et al., 9 Dec 2025).

1. Motivation and Intuitive Basis

GraphRAG architectures enhance LLMs with structured knowledge by retrieving subgraphs—entity–relation–entity triples—linearized into textual input (e.g., “A rel B”). Despite the structural fidelity of graph retrieval, the transformer-based LLM decoder lacks explicit inductive bias for graph topology and frequently “shortcuts” reasoning by concentrating its attention on the minimal path(s) (shortest-path triples) directly connecting the question entity to the answer entity. This propensity leads to a localized “partial view” of the graph, potentially ignoring critical supporting facts elsewhere in the subgraph, which in turn contributes to hallucinations: outputs that are syntactically fluent but not grounded in the actual retrieved knowledge. PRD operationalizes the degree of such localized attention focus, flagging when the LLM's attention collapses onto salient shortest-path triples.

2. Formal Definition and Computation

Notation is fixed as follows:

• $G$: the linearized sequence of tokens representing the retrieved subgraph (heads, relations, tails).
• $S$: index set of tokens in $G$ corresponding to the shortest-path triples.
• $\bar S$: complementary set of token positions (other triples).
• $A = \{a_1, \ldots, a_{|A|}\}$: answer token positions (tokens generated after “ans:”).
• $L$: number of decoder layers.
• $H$: number of attention heads per layer.
• $a_{l,h,i,j}$: raw (pre-softmax) attention logits from decoder layer $l$, head $h$, answer token position $i \in A$, to source position $j$.

PRD is computed by first normalizing attention vectors with a softmax: $$\alpha_{l,h,i,j} = \frac{\exp(a_{l,h,i,j})}{\sum_{k=1}^{N_{\text{src}}}\exp(a_{l,h,i,k})}$$

Total in-path and out-of-path attention mass: $$\alpha_{l,h,i,S} = \sum_{j\in S}\alpha_{l,h,i,j} \qquad \alpha_{l,h,i,\bar S} = \sum_{j\notin S}\alpha_{l,h,i,j}$$

PRD averages, across all decoder layers and heads, the difference in attention mass paid to path and non-path tokens: $$\text{PRD} = \frac{1}{L H |A|} \sum_{l=1}^L\sum_{h=1}^H\sum_{i\in A} \left(\alpha_{l,h,i,S} - \alpha_{l,h,i,\bar S}\right)$$ Given that $\alpha_{l,h,i,S} + \alpha_{l,h,i,\bar S} = 1$, PRD is bounded in $[-1, 1]$. In practice, it is typically positive for GraphRAG decoders, indicating an attentional bias toward shortest-path tokens.

Pseudocode for PRD calculation is as follows:

function compute_PRD(AttentionScores, S, A): let L = number of layers let H = number of heads per layer sum_diff = 0.0 for l in 1…L: for h in 1…H: for each answer position i in A: v = AttentionScores[l][h][i][:] exp_v = [exp(vj) for vj in v] Z = sum(exp_v) alpha = [ e_vj / Z for e_vj in exp_v ] mass_S = sum(alpha[j] for j in S) mass_notS = 1.0 - mass_S sum_diff += (mass_S - mass_notS) PRD = sum_diff / (L * H * |A|) return PRD

No additional thresholds or hyperparameters are required; normalization via softmax is intrinsic to the formulation.

3. Empirical Observations and Case Studies

Empirical results demonstrate the sensitivity of PRD in distinguishing answer types and risk regimes. In the MetaQA-1hop knowledge-based QA setting:

• Truthful answers have median PRD ≈ 0.72, while answers labeled as hallucinations have slightly higher median PRD ≈ 0.74. A two-sample $t$-test over 5,000 examples yields $t(4998) = -3.31$, $p = 9.48 \times 10^{-4}$, effect size $d = -0.18$ (small effect).
• Quadrant analysis splits outputs by median PRD and Semantic Alignment Score (SAS), revealing regimes of risk:

Quadrant	PRD	SAS	Hallucination Rate
Q1: High PRD, High SAS	0.752	0.421	9.5%
Q2: Low PRD, High SAS	0.701	0.452	5.0%
Q3: Low PRD, Low SAS	0.707	0.340	22.2%
Q4: High PRD, Low SAS	0.754	0.344	10.9%

Lowest hallucination risk (5.0%) occurs in the Low PRD, High SAS regime (Q2). Notably, high PRD is not alone predictive of incorrectness; high semantic grounding (high SAS) can mitigate hallucination even when PRD is elevated.

4. Predictive Power, Limitations, and Interplay with SAS

Isolated PRD exhibits weak correlation with hallucination labels (Pearson $r \approx -0.047$) and achieves limited area under the ROC curve (AUC ≈ 0.55) and recall (≈ 0.48) on Llama-2-7B, indicating that attention over-reliance itself does not robustly predict ungroundedness. However, combining PRD with the Semantic Alignment Score (SAS)—which measures alignment of model representations with retrieved knowledge—increases the predictive AUC to ≈ 0.83 and macro-F1 to ≈ 0.75 in the Graph Grounding and Alignment (GGA) detector.

PRD values in single-hop QA cluster between 0.65 and 0.85. A PRD above ≈ 0.75 is indicative of significant focus on shortest paths; if accompanied by low SAS (< 0.4) this marks a regime (Q4) with increased hallucination risk (~11%).

PRD is attention-based, quantifying “where” a model's focus lies, but not “what” information it encodes or actually utilizes semantically. Distributed attention (low PRD) without grounding (low SAS) is associated with the highest hallucination risk (Q3: 22.2%), emphasizing that PRD must be interpreted jointly with SAS.

5. Methodological Considerations

Calculation of PRD proceeds in three steps:

• Shortest-path extraction: Prior to generation, use BFS or another graph solver to extract all gold shortest paths between input and target entities. All tokens mapping to these triples comprise set $S$.
• Attention extraction: Capture (hook) all decoder self-attention distributions for each answer token across all layers and heads, ensuring indices correspond to graph-linearized input.
• Aggregation: PRD is computed using the attention pseudocode, averaging over all layers and heads for robustness. The process is architecture-agnostic provided that access to self-attention matrices is available.

Best practice involves establishing a dataset-specific median PRD (e.g., ≈ 0.727 on MetaQA-1hop) to distinguish high from low reliance regimes. Researchers are encouraged to monitor the 2D distribution of (PRD, SAS) during prompt engineering and model iteration.

6. Applications and Integration in Hallucination Detection

PRD functions as a key feature in post-hoc hallucination detection pipelines, particularly when combined with SAS and lightweight surface cues (output length, unique word ratio, etc.). In the GGA detector, a simple XGBoost or logistic regression model leveraging these features delivers strong performance (AUC ≈ 0.83, F1 ≈ 0.75) without necessitating model fine-tuning.

PRD is applicable to any transformer model exposing self-attention weights, supporting analysis across distinct GraphRAG instantiations and retrieval-augmented generation frameworks. Extending PRD to settings involving more complex, multi-hop, or ambiguous gold path regimes may require redefining the set $S$ as a union of plausible reasoning backbones.

7. Practical Guidance and Recommendations

• Identification of shortest-path tokens: Employ BFS or equivalent algorithms to determine all minimal paths, and map all constituent sub-tokens in the serialized graph context.
• Attention matrix extraction: Ensure granular capture of attention scores for each answer token's decoder input, aligning textual and graph segment indices as required.
• Interpretation and thresholding: Use empirical or validation set ROC analysis to determine effective thresholds for high PRD regimes; in case studies, PRD above ≈ 0.75 flagged substantial path-overreliance.
• Monitoring: It is advisable to lower PRD (favoring distributed attention) while increasing SAS (semantic grounding) as a recurring model development objective.

In conclusion, Path Reliance Degree is a principled, lightweight interpretability metric for exposure of under-distributed reasoning in GraphRAG decoders. While its stand-alone predictive power for hallucination is modest, its role is central within composite detectors and for mechanistic interpretability of LLM-driven reasoning with graph-structured knowledge (Li et al., 9 Dec 2025).