TRIAGE: Trustworthy Retrieval Instrumentation And Graph Evaluation

Abstract: Knowledge graphs (KGs) that underpin Graph-based Retrieval-Augmented Generation (Graph-RAG) are increasingly built automatically by LLM-driven extraction rather than curated by experts. Proper evaluation would require instrumenting all pertinent stages: extraction, graph construction, and inference, coherently enough to localize failures, so that a failure at one stage is not discovered as a wrong answer at the end. We introduce TRIAGE, a stage-aware instrumentation framework for automated, document-grounded graph-RAG that asks not only whether the underlying graph can be trusted but at what cost it can be queried. TRIAGE attaches stage-specific, independently interpretable metrics to three stages: the KG Implementation (triple confidence, source coverage, and schema and canonicalization checks), the KG Validation by expert (graph-level structural quality, with correctness and completeness computed only as offline calibration when a reference is available), and the KG Usage (retrieval coverage, faithfulness, and retrieval cost); the deployed metrics need no gold annotations, the gold-requiring ones serving only as offline calibration. At usage time these metrics form a diagnostic chain of necessary conditions whose first broken link localizes the failure, and the diagnosis maps to the stage levers that can remedy it: extraction, graph and schema, or retrieval. TRIAGE is a theoretical framework with a proof of concept and a reproducible evaluation protocol.

• The paper presents TRIAGE, a stage-aware framework that evaluates automated graph-RAG pipelines with actionable, stage-specific trust metrics.
• It introduces independent metrics for extraction, validation, and usage phases, enabling precise fault localization without relying on gold standards.
• Empirical validation reveals significant performance gaps between structural retrieval adequacy and answer correctness, underscoring the need for cost-aware remediation.

TRIAGE: A Stage-Aware Framework for Trustworthy Evaluation of Automated Graph-RAG Pipelines

Problem Motivation and Context

The TRIAGE framework addresses the critical challenge in automated graph-based Retrieval-Augmented Generation (Graph-RAG) pipelines, where Knowledge Graphs (KGs) are increasingly constructed via LLM-driven extraction rather than traditional expert curation. This transition introduces opaque failure modes and a lack of stage-specific trust signals, raising new concerns regarding answer reliability, graph fidelity, and operational retrieval cost. Existing KG engineering methodologies, such as TKG [TKG-Amdouni2026], focus primarily on expert-driven elicitation processes and do not instrument automated pipelines or usage-phase applications. Conversely, recent graph-RAG systems (e.g., GraphRAG [GraphRAG-Edge2024], LightRAG [LightRAG-Guo2024], PathRAG [chen2026pathrag]) are optimized for efficiency and accuracy without providing diagnostics for extraction, graph integrity, or retrieval, leaving failed reasoning unlocalized and remediation uninformed. TRIAGE fills this gap by offering a formalized, multi-phase instrumentation suite for graph-RAG pipelines, enabling actionable, stage-aware failure localization and cost evaluation.

Framework Overview

TRIAGE operationalizes end-to-end instrumentation across three phases of the KG lifecycle:

• KG Implementation: Automated extraction, grounding, and canonicalization, with confidence and provenance metrics.
• KG Validation: Expert review and graph-level quality assessment, employing ground-truth-free and schema-dependent metrics.
• KG Usage: Retrieval, faithfulness, and computational cost metrics at inference, with pre-inference triage and failure localization.

Each phase is equipped with independently observable metrics—deployment primarily relies on gold-free, schema-free measures, while gold-dependent metrics serve as offline calibration. The metrics do not propagate between phases, enabling precise identification of faults and targeted remediation.

Computability Axes

TRIAGE categorizes metrics by:

1. Reference Requirement: None, ontology, or gold annotation.
2. Visibility: White-box (artifacts observable) versus black-box (only input/output visible).

This organization ensures that assessment degrades gracefully with diminished resources (ontology absence, hidden extraction), as opposed to failing completely.

Stage-Specific Metrics and Diagnostic Chain

KG Implementation Metrics

Extraction reliability is quantified via Triple Confidence Score ($C_{triple}$), Source Coverage Ratio, Extraction Consistency, Cross-Source Corroboration, Schema Compliance, Constraint Violation Rate, Canonicalization Success Rate, and Evidence Attribution Rate. Canonicalization operates in ontology-grounded or resolution-driven regimes, depending on schema availability. Persisted provenance artifacts enable auditability and downstream trust signal computation.

KG Validation Metrics

Ground-truth-free, graph-level metrics include Ontology Coverage, Dead Node Ratio, Semantic Redundancy Rate, Logical Consistency, and, in offline calibration, correctness and completeness (imported from TKG [TKG-Amdouni2026]). These establish global graph health and provide actionable signals for expert review.

KG Usage Metrics

At inference, structural adequacy is measured by Query Grounding Rate, Entity Retrieval Coverage (ERC), Retrieval Precision, Reasoning Readiness Score (RRS), Answer Grounding and Utilization Rates, Faithfulness, Retrieval Path Cost, and Reasoning Hop Depth. These metrics form a diagnostic chain of necessary conditions, where the first broken link localizes the failure (extraction, graph connectivity, retrieval). Pre-inference triage leverages these metrics to inform fallback strategies or issue low-confidence warnings prior to answer generation.

Numerical Results and Empirical Validation

Proof-of-concept evaluation on PathQuestion (synthetic multi-hop KB) demonstrates strong separation of evidence-present versus evidence-missing cases by structural metrics:

• ERC ($=1$): Context Coverage jumps to $98.4\%$ versus $44.3\%$ at $<0.5$ ERC, a significant $54$-point gap.
• RRS ($>0$): Coverage at $93.2\%$, versus $54.8\%$ for disconnected retrieval.

Conversely, answer correctness (Hits@1) is much less sensitive to retrieval adequacy, remaining high even when retrieval fails—highlighting "silent success" due to parametric memory. This discrepancy necessitates structural scoring distinct from answer evaluation.

The protocol prescribes stratified evaluation by coverage/connectivity quartiles, paired bootstrap confidence intervals, and per-phase Spearman correlations to demonstrate predictive validity, localization, and triage utility. Results falsify claims unless structural metrics monotonically track expected outcomes, with calibrated thresholds for operational deployment.

Contradictory and Strong Claims

• TRIAGE demonstrates that correct answers can coincide with structurally deficient retrieval, a direct contradiction to answer-level evaluation paradigms. The framework claims that answer correctness alone is insufficient for certifying retrieval integrity in LLM-powered graph-RAG.
• All key usage metrics (ERC, RRS) are shown to be computable without gold annotations or ontology schema, underpinning real-time operational deployment.
• The diagnostic chain enables actionable, stage-targeted remediation—failure localization is not achievable with existing end-to-end metrics or black-box judges.

Practical and Theoretical Implications

TRIAGE establishes a deployment-ready methodology for trustworthy, cost-aware evaluation of automated graph-RAG pipelines, addressing the unique challenges posed by LLM-driven KG construction. The framework enables continuous, monitored KG evolution, localizes failures with interpretable signals, and informs remediation at the correct phase, optimizing both reliability and resource allocation. This shifts KG engineering from monolithic score-based updates to principled, stage-aware interventions aligned with broader trustworthy AI objectives.

Future Directions

Potential lines of advancement include:

• Extending TRIAGE to multi-hop, multi-entity query decomposition for complex QA.
• Automating KG Update triggers from rolling metric aggregates, integrating diagnosis-driven lever selection.
• Active learning interfaces to formalize KG Validation and optimize annotation cost.
• Enhanced faithfulness metrics leveraging NLI-based techniques for relational verification.
• Operationalizing continuous health monitoring interfaces for KG-centric AI systems.

Conclusion

TRIAGE formalizes a stage-aware instrumentation methodology for automated, LLM-driven graph-RAG pipelines, organizing deployment-viable, reference-independent trust metrics across extraction, validation, and usage. Empirical evaluation evidences strong predictive association between structural retrieval adequacy and answer correctness, and exposes silent failures masked by parametric model knowledge. The framework's diagnostic chain localizes faults and informs precise, cost-effective remediation, marking a distinct advancement beyond prior task-level and lifecycle methodologies. TRIAGE’s approach aligns with ongoing efforts in trustworthy KG engineering and robust AI, setting the standard for principled evaluation and maintenance of retrieval-augmented generation systems.

---

References:

• "TRIAGE: Trustworthy Retrieval Instrumentation And Graph Evaluation" (2607.03447)
• "An End-to-End Trustworthy Knowledge Graph Engineering Methodology" [TKG-Amdouni2026]
• "From Local to Global: A Graph RAG Approach to Query-Focused Summarization" [GraphRAG-Edge2024]
• "Lightrag: Simple and fast retrieval-augmented generation" [LightRAG-Guo2024]
• "Pathrag: Pruning graph-based retrieval augmented generation with relational paths" [chen2026pathrag]