Knowledge Attribution Score (KAS)

• Knowledge Attribution Score (KAS) is a continuous metric that measures the support or contradiction of textual claims by leveraging structured knowledge graphs.
• It integrates per-claim scoring, triplet similarity, and asymmetric normalization to deliver granular, reliable assessments for fact-checking and source attribution.
• KAS is applied in both claim verification and retrieval-augmented generation frameworks, guiding decision-making in high-stakes, automated information pipelines.

The Knowledge Attribution Score (KAS) is a continuous metric designed to quantify the degree to which textual content or database items are supported or implicated by underlying knowledge structures. It serves as both a reliability indicator in claim-level fact verification against structured knowledge graphs and a responsibility score for source tracing in retrieval-augmented generation (RAG) systems. In recent research, KAS arises in two principal frameworks: ClaimVer for claim-level text verification against trusted knowledge graphs (Dammu et al., 12 Mar 2024), and RAGOrigin for identifying the origins of corrupted outputs in poisoned RAG databases (Zhang et al., 17 Sep 2025). KAS outputs a scalar score, typically normalized or bounded, enabling granular downstream decision-making in high-stakes information pipelines.

1. Motivation and Conceptual Rationale

KAS emerged from a need for fine-grained validation and attribution under conditions of uncertain, incomplete, or adversarially manipulated knowledge. Traditional fact-checking methods provide discrete judgments (“true/false/unsupported”) at the sentence or document level and lack actionable, interpretable feedback. KAS addresses these limitations through:

• Granularity: Each claim or candidate text is scored for degree of support, contradiction, or influence, rather than only binary adjudication.
• Continuity: Enables ranking, thresholding, and learning from a real-valued scale instead of coarse categories.
• Asymmetry: Heavier penalization for misinformation versus commensurate rewarding of robust support, preventing overconfident aggregation when contradictory evidence is present.
• Attribution: Traces outcome responsibility to specific evidence, claims, or database items, supporting explainability and accountability in automated systems.
• Downstream Utility: Applicability to ranking candidates, gating for publication, and training feedback loops.

These features position KAS as a foundational reliability signal in modern knowledge-centric NLP applications.

2. Mathematical Formulation and Computation

ClaimVer Framework (Dammu et al., 12 Mar 2024)

KAS in ClaimVer is defined for an input text partitioned into $n$ claims:

1. Per-claim score:

$$\text{cs}(y_i)= \begin{cases} +2, & y_i=\text{Attributable}\ +1, & y_i=\text{Extrapolatory} \land |\text{rel\_triplets}_i|>0\ 0, & y_i=\text{Extrapolatory} \land |\text{rel\_triplets}_i|=0 \ 0, & y_i=\text{No attribution (omission)}\ -1, & y_i=\text{Contradictory} \end{cases}$$

2. Triplet Match Score (TMS):

$$\text{TMS}_i = \alpha\,SS(E(\text{claim}_i),E(\text{triplet}_i))+\beta\,EPR(E(\text{claim}_i),E(\text{triplet}_i))$$

• $SS$ is the cosine similarity between claim and triplet text embeddings.
• $EPR$ is the fraction of claim entities covered by triplet entities.
• Hyperparameters: $\alpha=\beta=0.5$ (default).

3. Aggregate and non-symmetric normalization:

$$\sigma_{\mathrm{mod}}(x; \gamma)=\frac{1}{1+\exp(-\gamma x)} \qquad \gamma=\begin{cases} 3 & x<0 \ 1 & x\geq0 \end{cases}$$

$$\mathrm{KAS} = \sigma_{\mathrm{mod}}\left(\sum_{i=1}^n [\text{TMS}_i \cdot \text{cs}(y_i)],\,\gamma\right)$$

RAGOrigin Framework (Zhang et al., 17 Sep 2025)

KAS (called the “responsibility score”) for candidate document $u_i$ in narrowed attribution scope $\mathcal U$:

1. Raw signals:
   • Retrieval ranking $r_i=\mathrm{sim}(E(q),E(u_i))$
   • Semantic relevance $s_i$ (proxy LLM log-prob of reconstructing question from $u_i$)
   • Influence-on-generation $f_i$ (proxy LLM log-prob of generating the incorrect answer from $u_i$)
2. Standardization:

$$\bar r_i = \frac{r_i-\mu_r}{\sigma_r},\quad \bar s_i = \frac{s_i-\mu_s}{\sigma_s},\quad \bar f_i = \frac{f_i-\mu_f}{\sigma_f}$$

1. Aggregate score:

$$\mathrm{KAS}(u_i) = \frac{w_r\,\bar r_i + w_s\,\bar s_i + w_f\,\bar f_i}{w_r + w_s + w_f}$$

• Default weights $w_r=w_s=w_f=1$.

2. Clustering: $k$-means on $\mathrm{KAS}(u_i)$ values, flagging the cluster with higher mean KAS as poisoned.

3. Computation Pipeline and Example Procedures

ClaimVer Pipeline

1. NER and KG linking of claim spans.
2. Coreference resolution for standalone claim formation.
3. Triplet retrieval via Woolnet (multi-node BFS up to 3 hops, max 4 paths).
4. Claim attribution: LLM outputs $(\text{span},\,y_i,\,\text{rel\_triplets}_i)$ + rationale.
5. Per-claim KAS calculation: Assign cs$(y_i)$, compute TMS$_i$, aggregate.
6. Final score: Sum weighted contributions, apply asymmetric sigmoid.

Example

For the claim “Apollo 11 landed on the Moon in 1969,” relevant KG triplets are retrieved, leading to:

• $y_1=$ Attributable, cs$(y_1)=+2$
• $SS=0.96$, $EPR=0.67$, TMS$=0.815$
• Weighted sum: $1.63$; KAS $\approx 0.835$

RAGOrigin Pipeline

1. Scope narrowing: Retrieve and chunk knowledge base by similarity to question; blocks tested for their ability to reproduce the misgenerated answer.
2. Score computation: For candidates in the final scope, calculate $r_i,s_i,f_i$.
3. Z-normalization: Standardize scores.
4. Score aggregation: Compute KAS for each candidate.
5. Clustering: Apply k-means; flag items as poisoned according to cluster mean.
6. Dynamic adaptation: No manual thresholds; block size and weights tested for stability.

4. Properties, Sensitivities, and Hyperparameter Effects

ClaimVer (Dammu et al., 12 Mar 2024)

• Range: $(0,1)$
• Interpretability: KAS near zero implies strong contradiction; near one, strong support.
• Score Dynamics: Attributable labels drive KAS up by $+2\cdot$TMS; contradictions penalize via $-1\cdot$TMS and asymmetric $\gamma=3$.
• Sensitivity: Weak triplet matches (low $SS$, $EPR$) yield low impact even for “Attributable” claims; contradictions with moderate match are aggressively penalized.
• Granularity: “Extrapolatory” claims receive partial credit; omission or neutral triplets do not contribute.
• Extensibility: Parameters $\alpha, \beta, \gamma$ can be fine-tuned as needed.

RAGOrigin (Zhang et al., 17 Sep 2025)

• Standardization: Z-scoring of all components is essential for consistent thresholding.
• Component contribution: Ablations indicate all three signals are necessary for high accuracy; absence of z-normalization or adaptive scope sharply degrades detection.
• Block size and model selection: Robust to parameter variations; default $K=5$, weights = 1, proxy model flexible.

5. Integration and Downstream Applications

ClaimVer

• Pipeline role: Final reliability metric for multi-claim text.
• Uses: Ranks candidate texts, thresholds for review/publishing, fine-tuning models to maximize factuality, and interactive UI feedback showing attribution scores.
• User trust: Provides both fine-grained explanations and a summary scalar, aiding transparency and cognitive load reduction.

RAGOrigin

• Provenance forensics: Identifies and isolates sources of generation corruption (knowledge poisoning).
• Automated operation: Works in black-box manner across multiple datasets, attack types, retriever models, and LLMs.
• No manual thresholding: Clustering adapts automatically to context and event.
• Efficiency: Average $2$ seconds, $<$\$0.0002 per query.

6. Empirical Findings and Observed Performance

ClaimVer (Dammu et al., 12 Mar 2024)

• Examples: High KAS correlates with claims marked mostly “Attributable”; low KAS dominated by “Contradictory.”
• Classification F1: Underlying claim classification reaches $\sim99\%$ F1.
• Quality correlation: Higher KAS aligns with factual user impressions and clear KG evidence.
• Asymmetry effect: $\gamma=3$ empirically selected; ablations suggest symmetric sigmoid over-penalizes contradictions.

RAGOrigin (Zhang et al., 17 Sep 2025)

• Detection Accuracy (DACC): $\geq 0.97$ across 5 QA datasets and 9 attacks; FPR $\leq 0.03$, FNR $\leq 0.01$.
• Removal efficacy: Flags enable removal of poisoned texts and drop attack success rates to zero, outperforming prior techniques (RAGForensics).
• Robustness: Stable across paraphrased questions, varying poisoned document numbers, different retrievers and LLMs, multi-attacker scenarios, adaptive attacks, and dynamic database changes.
• Component ablations: Use of all signals and adaptive scope is critical for maximal robustness.

7. Context and Plausible Implications

KAS bridges interpretable reliability assessment and source attribution in high-stakes information systems. In ClaimVer, KAS empowers nuanced trust modeling and granular evidence weighting in claim verification, allowing stakeholders to differentiate content based on degrees of knowledge graph support. In RAGOrigin, KAS enables forensic traceability of corrupted knowledge, mitigating the risk of semantic poisoning and unobtrusively adapting to attack variations and database volatility.

A plausible implication is that KAS will be increasingly central in both development of robust fact-checking pipelines and real-time provenance tracking in retrieval-augmented LLMs. Its formalization supports integration into broader feedback loops, user-facing reliability visualization, and autonomous review or gating of model outputs, providing a principled tool for both explainability and operational defense in automated knowledge systems.