ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents

Abstract: Tool-using LLM agents increasingly use the Model Context Protocol (MCP) to answer from heterogeneous evidence sources, including search, APIs, databases, clinical records, and formulary tools. Standard factuality metrics usually test whether an answer is supported by pooled evidence, missing a provenance-sensitive failure mode: a claim may be supported somewhere while being attributed to the wrong source. We call this cross-source conflation. We introduce ProvenanceGuard, a source-aware verifier for MCP-grounded answers. It consumes captured MCP traces with stable tool IDs, source IDs, and raw outputs; decomposes answers into atomic claims; routes claims to source-specific evidence; checks support with NLI and a token-alignment proxy; compares stated attribution with the routed source; and returns per-claim verdicts plus an answer-level allow/block decision. Blocked answers can be repaired with retrieval-augmented answer revision and re-verified. We evaluate on 281 medical-domain MCP-agent traces. A 266-trace adjudicated subset yields 2,325 LLM-assisted claim labels split by trace; 361 held-out labels are human-verified. On the 40-trace held-out split, ProvenanceGuard achieves block F1 0.802 and source accuracy 0.858 over 260 source-eligible claims, outperforming source-blind baselines that do not emit claim-to-source IDs. On a harder multi-source benchmark it reaches block F1 0.846, while source-plus-relation accuracy drops to 0.229, showing that exact source ownership remains difficult with semantically close sources. Repair-and-reverify resolves all blocked answers in the full trace set, often via conservative fallback. In 50 controlled clinical conflation probes, ProvenanceGuard detects all injected attribution swaps with no retained wrong attribution. These results show that source attribution is an independent axis for factuality verification in MCP-based agents.

• The paper introduces a novel source-aware verification approach that decomposes answers into atomic claims and assigns evidence via cosine similarity.
• It employs a calibrated RandomForest classifier combined with NLI and token alignment to achieve block F1 scores up to 0.802 while detecting conflations.
• Its practical impact spans critical domains like healthcare and finance by ensuring precise provenance attribution, reducing risks of misattributed evidence.

ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents

Motivation and Problem Formulation

The emergence of LLM agents employing the Model Context Protocol (MCP) and orchestrating evidence from heterogeneous sources (search engines, APIs, databases, structured EHRs) necessitates verification paradigms that go beyond context-supported factuality. Traditional verifiers operate under a source-blind abstraction, merely affirming factual support based on pooled evidence. This is insufficient in MCP-based workflows where an answer's acceptability depends not just on support, but on provenance fidelity: factual attributions must be anchored in the correct source. The paper formalizes cross-source conflation—“a claim is supported, but attributed to the wrong source”—as a critical, previously under-addressed failure mode in MCP-based agent pipelines.

System: The PROVENANCEGUARD Pipeline

ProvenanceGuard is introduced as the first general system for source-sensitive factuality verification in MCP-grounded LLM answers. Its design is modular, offline-friendly, and optimized for deterministic, fail-closed evaluations:

1. MCP Trace Ingestion: Each trace provides granular capture of tool calls, explicit source and tool IDs, and all tool outputs. No evidence pooling is performed; intermediate states remain provenance-differentiated.
2. Atomic Claim Decomposition: Answers are deterministically split into atomic, evidence-bearing claims, with exact preservation of literals and explicit attribution spans.
3. Source Routing: Each claim is embedded and routed via cosine similarity to its most likely MCP evidence source. Routing margin and rank diagnostics are preserved to evaluate assignment ambiguity and alternate candidates.
4. Support Verification: The routed claim-evidence pair is passed to a fine-tuned NLI model (MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli), yielding entailment/neutral/contradiction. A token-alignment attention proxy quantifies overlap and grounding at the claim-term level, supporting robust handling of literal values.
5. Calibrated Support Decision: Support is determined by a RandomForest classifier fusing routing, NLI, lexical, and token-derived features. Calibration operates to maximize block-F1 with a threshold of 0.65, yielding a conservative, high-recall, low-false-negative regime optimal for critical domains.
6. Attribution Check: Stated/implied attribution in the answer is lexically matched to the routed source’s family. Claims entailed by one source but attributed to another are flagged as conflations.
7. Answer Aggregation and Repair: Answers are blocked if any claim is unsupported, contradicted, or conflated. Blocked answers are submitted to a controlled RARR-style repair regime, which redrafts the answer using only the available source and re-verifies for correct provenance.

Empirical Evaluation

Benchmarks

• Primary Medical MCP Capture (Frozen prospective corpus): 281 agent traces, decomposed into 2,325 claims, with a held-out split of 40 traces/361 claims reviewed by human experts.
• Multi-Source Adjudicated Benchmark: Stress-slices involving chart-literature conflation, count-resource aggregation, and semantically-close distractors.
• Targeted Conflation Probes: 50 synthetic instances with explicit source-attribution swaps.

Results

Support Detection and Provenance Attribution:

• Block F1 = 0.802 (CI [0.664, 0.900]) and source accuracy = 0.858 on the main held-out split (260 source-eligible claims).
• Source-blind baselines (e.g., MiniCheck, RAGAS) attain similar block F1 (0.783, 0.758, respectively) but lack source attribution capability.

Multi-Source Benchmark:

• Block F1 = 0.846. However, source accuracy drops to 0.503 and source-plus-relation accuracy collapses to 0.229, reflecting the intrinsic challenge of exact attribution amidst plausible rival sources.

Targeted Attribution Probes:

• All 50 deliberately injected conflations were detected and repaired; no wrong attribution persisted.

Repair:

• On captured traces, the repair-and-reverify loop resolved all 173 blocked outputs, though most (144) reverted to conservative, non-informative fallback responses rather than fully rewritten answers.

Ablations and Calibration:

• The calibrated RandomForest head substantially increased verdict accuracy (from 0.363 for raw NLI-only to 0.812), demonstrating the necessity of feature fusion and operating point control.
• Direct verdict-head training improved raw accuracy, but final deployment prioritizes block recall and provenance.

Theoretical and Practical Implications

The decomposition of factual verification into support and explicit source attribution is established as an independent evaluation axis for MCP-grounded agents. Binary support (as in RAG or summarization) cannot substitute for provenance: correct fact-to-source pairing is essential when agent output is interpreted in settings where misattributed evidence incurs legal/regulatory or clinical risk.

Practically, ProvenanceGuard’s approach is tailored for data-sensitive, offline audit scenarios where high recall of attribution faults is prioritized. Its decision-theoretic regime (fail-closed) is effective in healthcare, financial, and enterprise environments, though at the cost of answer informativeness when evidence is ambiguous or incomplete.

Limitations and Future Directions

Key limitations include:

• The reference corpus is single-domain (medical), potentially limiting evaluation external validity.
• Human adjudication is restricted to the held-out subset; most labels are LLM-assisted.
• Evaluation is focused on explicit conflation, with adversarial/paraphrased scenarios under-explored.
• Claim-splitting precision lags at preserving complex literals; improved decomposition methods (possibly neural-sequence models) are needed.
• Longer context and more complex MCP scenarios demonstrated lower attribution performance, suggesting significant headroom for improvement via larger context models and more discriminative raw pretraining (e.g., full graph-structured source-flow models).

Future work should extend to:

• Larger and more heterogeneous MCP agent environments (e.g., legal, financial, industrial domains).
• Human-authored claim gold annotations and fine-grained adversarial probes.
• End-to-end neural architectures integrating source-routing, NLI, and attribution without heavy reliance on calibration.
• Benchmarking runtime and scaling efficacy as the number of sources and claims per trace increases.

Conclusion

ProvenanceGuard operationalizes source-aware factuality verification as a post-hoc alignment layer for MCP-based LLM agents, enabling systematic detection of unsupported, misattributed, and conflated claims. The system advances the field by elucidating attribution as a distinct, indispensable criterion alongside classical support in agent factuality evaluation. Its empirical performance validates feasibility, while ablation and error analyses reveal substantial remaining challenges—chiefly scalability, decomposition, and generalization to adversarial/multi-source scenarios. Ongoing research should co-evolve provenance verification and agent architectures to guarantee rigorous truth and trust in multi-tool LLM deployments.

Reference: "ProvenanceGuard: Source-Aware Factuality Verification for MCP-Based LLM Agents" (2606.18037)