GSAR: Typed Grounding for Hallucination Detection and Recovery in Multi-Agent LLMs

Abstract: Autonomous multi-agent LLM systems are increasingly deployed to investigate operational incidents and produce structured diagnostic reports. Their trustworthiness hinges on whether each claim is grounded in observed evidence rather than model-internal inference. Existing groundedness evaluators (binary classifiers, LLM-as-judge scalars, self-correction loops) treat supporting evidence as interchangeable and emit a single signal that offers no principled control over downstream action. We present GSAR, a grounding-evaluation and replanning framework that (i) partitions claims into a four-way typology (grounded, ungrounded, contradicted, complementary), giving first-class standing to non-redundant alternative perspectives; (ii) assigns evidence-type-specific weights reflecting epistemic strength; (iii) computes an asymmetric contradiction-penalised weighted groundedness score; and (iv) couples that score to a three-tier decision function (proceed, regenerate, replan) driving a bounded-iteration outer loop under an explicit compute budget. We formalise the algorithm, prove six structural properties, and evaluate five design claims on FEVER with gold Wikipedia evidence under four independently-trained LLM judges (gpt-5.4, claude-sonnet-4-6, claude-opus-4-7, gemini-2.5-pro). Every ablation reproduces in the same direction on every judge: bootstrap 95% CIs on the rho=0 effect exclude 0 on all four; the no-complementary ablation under Opus 4.7 has CI [-96,-68] of 200; at n=1000 three independent judges converge to DeltaS(rho=0)=+0.058. A head-to-head against Vectara HHEM-2.1-Open is included. To our knowledge, GSAR is the first published groundedness framework coupling evidence-typed scoring with tiered recovery under an explicit compute budget.

• The paper introduces a four-way claim partition method that differentiates grounded, ungrounded, contradicted, and complementary claims in LLM systems.
• It employs type-weighted scoring and an asymmetric contradiction penalty to drive a tiered decision process for optimal recovery actions.
• Empirical evaluations on FEVER show reduced unnecessary replans and enhanced auditability, confirming GSAR's efficiency in autonomous diagnostics.

GSAR: Typed Grounding for Hallucination Detection and Recovery in Multi-Agent LLMs

Introduction and Problem Motivation

GSAR introduces a rigorous, deployable framework for hallucination detection and recovery tailored for multi-agent LLM systems operating in autonomous diagnostic and incident investigation settings. The critical challenge addressed is trust: accurate differentiation between tool-evidenced claims and model-internal inferences is nontrivial in structured diagnostic reports produced by multi-agent LLM platforms. Prior approaches—including binary classifiers (e.g., Vectara HHEM), scalar LLM-as-judge metrics (RAGAS, TruLens), and self-correction loops (Self-Refine, Reflexion, multi-agent debate)—fail to provide fine-grained, class-indexed groundedness signals that directly drive principled and cost-sensitive downstream actions.

GSAR fills this gap by integrating a four-way evidence-typed partition, epistemic weighting, contradiction penalization, and a three-way decision function within an explicit compute-budgeted outer loop.

GSAR Framework and Algorithmic Structure

GSAR’s framework hinges on several key design contributions:

1. Four-Way Claim Partition: Claims are explicitly partitioned as grounded, ungrounded, contradicted, and complementary, introducing a structured distinction between lack of evidence, explicit contradiction, and valid non-redundant alternative perspectives (K-class). This outperforms NLI-inspired three-way partitions, especially for multi-perspective agentic systems.
2. Evidence-Type Weighted Scoring: Each claim is annotated with evidence provenance, drawing from a defined taxonomy (e.g., tool, specific_data, signal_match, neg_evidence, complementary_finding, synthesis, inference, domain). Epistemic strengths are encoded via a configurable weight function $w : T \rightarrow [0,1]$, with tool-verified evidence maximally weighted and inferences or domain assertions penalized.
3. Asymmetric Contradiction Penalty: The contradiction component is penalized with a factor $\rho$, ensuring that contradictions cannot be elided from the denominator to achieve fraudulent score inflation.
4. Three-Tier Decision Function: The scalar groundedness score $S$ is mapped to $$\{\text{proceed}, \text{regenerate}, \text{replan}\}$$ actions, triggering cost-sensitive recovery: summary regeneration, full replanning, or immediate proceed.
5. LLM-as-Judge with Structured Outputs: A dedicated judging protocol emits the four-way partition, the scalar S, an abstain channel, and a rationale, enabling precise feedback loops and audit trails.

Algorithmically, the framework executes as a bounded-iteration loop: for each investigation, a plan is executed, evidence is aggregated and annotated, the judge partitions claims, and the computed $S$ value drives tiered recovery actions until the budget is exhausted or a “proceed” threshold is met.

Figure 1: GSAR-default decision distribution across the four 100\%-Locus runs, illustrating judge-dependent partitioning and action selection.

Scoring Function and Theoretical Properties

The GSAR score $S$ is defined as:

$$S = \frac{W(G) + W(K)}{W(G) + W(U) + \rho W(X) + W(K)}$$

where $W(P) = \sum_{c \in P} w(\text{type}(c))$ for partition $P$. The numerator includes grounded and complementary claims, while the denominator accumulates all epistemic mass, asymmetrically penalizing contradictions. The framework is formally shown to satisfy boundedness, monotonicity in grounded claims, contradiction penalization, complementary-claim value, contradiction non-suppression, and inference-observation asymmetry.

Complementary claims (K-class) contribute positively to $S$ but less than direct grounding, enabling retention of non-conflicting alternate perspectives without artificially inflating confidence.

The contradiction penalty $\rho$0 directly controls the incentive structure, suppressing modes that might suppress contradictions to inflate the groundedness score.

Empirical Evaluation and Ablation Analysis

Benchmarking Protocol

Evaluations are conducted on FEVER using gold Wikipedia evidence, employing Locus SDK, Cohere embeddings, and Oracle 26ai vector search for claim ingestion and search. Four independently-trained LLM judges (GPT-5.4, Claude-Sonnet-4-6, Claude-Opus-4-7, Gemini-2.5-Pro) are used for robustness. The framework is systematically ablated: uniform weights, no complementary class, contradiction penalty removed ($\rho$1), and collapse of three-tier to two-tier decision.

Key Results

• Four-way Partition Superiority: Removing the complementary class in high-activity settings (n=200, Opus 4.7) collapses the grounded-output rate by 82% ($\rho$2), confirming the necessity of the K class (C1).
• Contradiction Penalty Robustness: Suppressing $\rho$3 inflates $\rho$4 across all judges with no change in contradiction identification rate, exposing the crucial role of asymmetric penalization (C3).
• Tiered Decision Efficiency: Three-tier decision structure reduces unnecessary replans, especially prominent at scale (e.g., hundreds of replans saved per 1000 claims, C4).
• Evidence-Type Interpretability: Uniform weights reduce alignment with external faithfulness ratings, confirming the interpretability advantage of type-sensitive scoring (C2).
• Implementation Feasibility: Framework demonstrated on proprietary/production-grade stack as well as open infrastructure (C5).

Figure 2: Opus 4.7 n=200 ablation summary. No-complementary and uniform-weight variants sharply reduce both mean S and proceed rate, demonstrating partition and weight importance.

Figure 3: Contradiction penalty ablation. All judges show positive S inflation under $\rho$5, validating structural properties and necessary penalization.

Head-to-Head Baselines

Compared to Vectara HHEM and RAGAS, GSAR’s four-way partition enables more nuanced downstream action selection at the cost of lower M4 (contradiction catch-rate) on FEVER. This is not viewed as a limitation but as reflecting a structural shift: prioritizing faithful, auditable summary-level decisions over binary benchmarking.

Figure 4: Judge-selection trade-off in the M4$\rho$6M5 plane. GPT-family judges catch more contradictions; Opus 4.7 utilizes the complementary channel more. GSAR accommodates both.

Implications, Limitations, and Future Directions

GSAR’s framework provides a principled path for integrating epistemic structure into LLM-agent-based autonomous diagnostics. The explicit calibration of $\rho$7 and $\rho$8 enables rapid adaptation per deployment. The four-way claim partition supports the incorporation of legitimate, non-conflicting alternative hypotheses—a crucial feature in practical root-cause analysis. The asymmetric contradiction penalty and tiered recovery avoid perverse incentives and optimize compute in real-world pipelines.

Limitations include LLM-as-judge bias, dependency on atomic claims, dependence on human expert prior for evidence-type calibration, and vulnerability to adversarial tool outputs. Addressing these will require per-deployment audits, tool provenance checks, and periodic human-in-the-loop recalibration.

Future directions include using GSAR’s structured outputs as rich reward signals in RLHF, process reward models, and constitutional supervision regimes; benchmarking on domain-matched AIOps datasets; and embedding GSAR in full multi-agent orchestration stacks for robust production deployments.

Conclusion

GSAR introduces a robust, theoretically justified and empirically validated framework for hallucination detection and action selection in multi-agent LLM diagnostic systems. By integrating evidence-typed partitioning, weighted scoring, tiered recovery policies, and explicit contradiction penalization, GSAR sets a new standard for system-transparent, auditable, and controllable grounding evaluation. The framework’s modular design, practical configurability, and strong empirical evidence across models and infrastructures position it as the preferred backbone for faithful, cost-effective, and adaptive multi-agent LLM deployments.