FACET Framework for RAG Diagnostics

• FACET is a facet-level diagnostic framework for RAG that decomposes complex questions into atomic reasoning facets for independent evaluation.
• It constructs a Facet × Chunk evidence matrix by combining retrieval similarity and NLI-based faithfulness to compare controlled inference modes.
• The framework reveals that hallucinations in RAG stem more from evidence integration issues than retrieval failures, impacting design and monitoring strategies.

Searching arXiv for the FACET framework paper and closely related work on facet-level evaluation and RAG diagnostics. FACET is a facet-level diagnostic framework for retrieval-augmented generation (RAG) in question answering that decomposes each input question into atomic reasoning facets, constructs a Facet × Chunk evidence matrix over retrieved evidence, and compares controlled inference modes to diagnose how evidence is used or overridden during generation (Elchafei et al., 10 Apr 2026). It was introduced to address a persistent failure mode in RAG: hallucinated answers remain common even when relevant documents are available, while conventional answer-level or passage-level evaluation offers limited visibility into retrieval–generation misalignment, defined as cases where relevant evidence is retrieved but not correctly integrated during generation (Elchafei et al., 10 Apr 2026).

1. Conceptual scope and problem setting

FACET is designed for diagnostic analysis rather than for proposing a new retriever or generator. Its central claim is that hallucination in RAG is driven less by retrieval accuracy and more by how retrieved evidence is integrated during generation, and that this failure is obscured by answer-level evaluation (Elchafei et al., 10 Apr 2026). The framework therefore shifts the unit of analysis from whole answers or passages to atomic reasoning facets, each of which can be answered and validated independently.

For a question $q$, facet decomposition produces a set

$\mathcal{F}(q) = \{f_1, f_2, \dots, f_m\},$

where each facet $f_i$ is a standalone sub-question (Elchafei et al., 10 Apr 2026). This decomposition supports fine-grained tracing of whether evidence is absent, retrieved but misused, or successfully integrated. A plausible implication is that FACET treats faithfulness as a stepwise property of reasoning rather than as a property of the final answer alone.

The framework was evaluated on medical question answering and HotpotQA, using three open-source and closed-source LLMs—GPT-4o-mini, Gemini-2.0-Flash, and LLaMA-3-8B-Instruct—and exposes recurring facet-level failure modes including evidence absence, evidence misalignment, and prior-driven overrides (Elchafei et al., 10 Apr 2026).

2. Facet decomposition and evidence representation

FACET operationalizes question decomposition differently across datasets. In HotpotQA, facets are created by converting dataset-provided supporting facts into interrogative form via prompting, using type-specific prompts for Bridge and Comparison questions. In medical QA, GPT-4o-mini is prompted to yield atomic clinical sub-questions (Elchafei et al., 10 Apr 2026). This produced 24,039 facets for 3,000 HotpotQA questions and 11,007 facets for 1,500 medical questions, and facets are labeled with semantic types such as temporal, comparative, procedural, boolean, and numerical (Elchafei et al., 10 Apr 2026).

Retrieved source documents are segmented into chunks of 300 tokens with 50-token overlap. Retrieval uses BGE-base-en-v1.5 embeddings with cosine similarity, and Top-K is fixed at 5 chunks per facet query to balance recall and precision (Elchafei et al., 10 Apr 2026). If the retrieved chunk set for a facet is denoted by $C = \{c_1, \dots, c_K\}$, FACET computes two signals for each facet–chunk pair $(f_i, c_j)$.

The first signal is retrieval relevance,

$s(f_i, c_j),$

given by the retriever’s similarity function (Elchafei et al., 10 Apr 2026). The second is NLI-based faithfulness or grounding for a generated facet answer $a_i$ against chunk $c_j$,

$$\text{faith}(a_i, c_j) = P(\text{entail}) - P(\text{contradict}),$$

computed with RoBERTa-large-MNLI (Elchafei et al., 10 Apr 2026). Positive values indicate support and negative values indicate contradiction. The paper does not define a single scalar matrix entry combining these signals; instead, the Facet × Chunk matrix stores retrieval similarity and NLI-based faithfulness side-by-side, and downstream analyses consider them jointly (Elchafei et al., 10 Apr 2026).

In the illustrative pipeline, facet-level faithfulness is aggregated as the maximum over chunks. This max operator is used to capture the strongest available support (Elchafei et al., 10 Apr 2026). The framework does not introduce numeric thresholds for relevance or faithfulness; in particular, Strict RAG relies on a prompt-enforced NO_ANSWER policy rather than thresholding (Elchafei et al., 10 Apr 2026).

3. Uncertainty quantification and controlled inference modes

FACET quantifies retrieval uncertainty through a margin between the top two retrieved chunks for a facet: $$U_{\text{ret}(f_i)} = s(f_i, c_j^{*}) - s(f_i, c_j^{**}),$$ where $\mathcal{F}(q) = \{f_1, f_2, \dots, f_m\},$0 and $\mathcal{F}(q) = \{f_1, f_2, \dots, f_m\},$1 are the top-1 and top-2 retrieved chunks for facet $\mathcal{F}(q) = \{f_1, f_2, \dots, f_m\},$2 (Elchafei et al., 10 Apr 2026). Small margins indicate ambiguous retrieval. The framework does not introduce entropy over NLI distributions or calibration techniques; uncertainty analysis is anchored in retrieval margins and observed faithfulness-score distributions across taxonomy categories (Elchafei et al., 10 Apr 2026).

A distinctive component of FACET is its use of three controlled inference modes to disentangle retrieval insufficiency from generation-side evidence misalignment and prior-driven overrides (Elchafei et al., 10 Apr 2026). These modes are prompt-enforced interventions rather than architectural modifications.

Strict RAG enforces exclusive reliance on retrieved evidence and returns NO_ANSWER if unsupported: $\mathcal{F}(q) = \{f_1, f_2, \dots, f_m\},$3 with the explicit system constraint: “Use ONLY the evidence below… If the evidence does not clearly support an answer, respond exactly with NO_ANSWER.” (Elchafei et al., 10 Apr 2026)

Soft RAG allows integration of retrieved evidence and parametric knowledge: $\mathcal{F}(q) = \{f_1, f_2, \dots, f_m\},$4 (Elchafei et al., 10 Apr 2026)

LLM-only generation disables retrieval and uses only the facet question: $\mathcal{F}(q) = \{f_1, f_2, \dots, f_m\},$5 (Elchafei et al., 10 Apr 2026)

Because enforcement is purely prompt-based, FACET is not a decoding-time constraint framework. Its diagnostic power comes from comparing outcomes across these three modes. This suggests that the framework uses mode differentials as a causal probe of whether failures originate in retrieval, evidence integration, or parametric priors.

4. Evidence taxonomy and retrieval–generation misalignment

FACET assigns each facet a label from an evidence taxonomy by comparing outputs across the three inference modes and checking NLI faithfulness (Elchafei et al., 10 Apr 2026). The taxonomy comprises five categories.

Evidence Failure denotes cases where retrieval finds no relevant information for the facet and Strict RAG returns NO_ANSWER (Elchafei et al., 10 Apr 2026). Evidence Misalignment denotes cases where retrieved chunks are topically related but contradict or fail to support the generated facet answer, corresponding to negative or low faithfulness (Elchafei et al., 10 Apr 2026). Evidence Overridden denotes cases where relevant evidence is retrieved, but the LLM ignores it and relies on parametric prior knowledge; Strict RAG fails while Soft RAG or LLM-only succeeds (Elchafei et al., 10 Apr 2026). Evidence Helpful denotes cases where retrieved evidence corrects the LLM; LLM-only fails while Strict RAG succeeds (Elchafei et al., 10 Apr 2026). Evidence Robust denotes cases where all three modes produce correct, consistent answers (Elchafei et al., 10 Apr 2026).

Retrieval–generation misalignment is not expressed as a single numeric metric. Instead, it is diagnosed from the joint presence of high retrieval similarity and low or negative faithfulness, together with mode comparisons such as Strict RAG failure despite strong retrieval (Elchafei et al., 10 Apr 2026). This is a notable methodological distinction from prior work emphasizing answer-level accuracy or passage-level attribution, which the paper argues cannot diagnose how evidence is used across reasoning steps (Elchafei et al., 10 Apr 2026).

A recurrent misconception addressed by the framework is that stronger retrieval should reliably yield more faithful generation. FACET explicitly reports a weak and inconsistent relationship between retrieval quality and generation faithfulness: highly relevant passages are often ignored or overridden by parametric priors, whereas faithful answers sometimes appear despite imperfect retrieval (Elchafei et al., 10 Apr 2026). The article’s central interpretive consequence follows directly from this observation: retrieval success and grounded generation are neither equivalent nor monotonically coupled.

5. Evaluation setup and empirical findings

FACET was evaluated on a stratified subset of 3,000 HotpotQA questions across easy, medium, and hard difficulty, and on 1,500 questions from RAGBench–Medical, described as medical QA derived from clinical guidelines and authoritative texts (Elchafei et al., 10 Apr 2026). Retrieval used BGE-base-en-v1.5 with cosine similarity, chunk size 300 tokens with 50-token overlap, and Top-K = 5 per facet (Elchafei et al., 10 Apr 2026). Faithfulness scoring used RoBERTa-large-MNLI, and question-level evaluation used token-level F1 and BERTScore, while facet-level outputs were aggregated to final question-level answers via a prompt (Elchafei et al., 10 Apr 2026). Decoding was performed with temperature $\mathcal{F}(q) = \{f_1, f_2, \dots, f_m\},$6, facet-level max tokens $\mathcal{F}(q) = \{f_1, f_2, \dots, f_m\},$7, question-level aggregation $\mathcal{F}(q) = \{f_1, f_2, \dots, f_m\},$8 tokens, and LLaMA was run with 8-bit quantization (Elchafei et al., 10 Apr 2026).

The main quantitative result is that evidence override dominates retrieval failure on both datasets (Elchafei et al., 10 Apr 2026).

Dataset	Overridden	Failure
Medical overall	28.4%	7.0%
HotpotQA overall	42.3%	5.8%

On medical QA, the overall taxonomy distribution is Failure 7.0%, Overridden 28.4%, Misalignment 7.5%, Helpful 38.1%, and Robust 19.0% (Elchafei et al., 10 Apr 2026). Model-specific patterns differ: Gemini is described as balanced, with Failure 20.6%, Overridden 12.5%, and Robust 35.1%, while GPT and LLaMA show override-dominant patterns with 34.4% and 38.2% overridden, respectively, and near-zero failure (Elchafei et al., 10 Apr 2026).

On HotpotQA, the overall taxonomy distribution is Failure 5.8%, Overridden 42.3%, Misalignment 0.7%, Helpful 37.6%, and Robust 13.6% (Elchafei et al., 10 Apr 2026). Override increases in general-domain Wikipedia, which the paper states is consistent with stronger parametric priors (Elchafei et al., 10 Apr 2026).

Controlled mode comparison yields another important result. Soft RAG consistently outperforms Strict RAG at question level across datasets, particularly on complex reasoning and medium difficulty, and LLM-only often exceeds Strict RAG (Elchafei et al., 10 Apr 2026). The paper presents a representative HotpotQA example concerning the Grand Vizier question with gold answer “Serb”: Strict and Soft RAG outputs are wrong or NO_ANSWER across models, but GPT in LLM-only mode produces the correct answer with $\mathcal{F}(q) = \{f_1, f_2, \dots, f_m\},$9 (Elchafei et al., 10 Apr 2026). This indicates that strict evidence gating can suppress correct prior knowledge when retrieval or evidence integration is imperfect.

Facet semantics also matter. Comparative facets are reported as the most unstable, with higher misalignment and lower robustness (Elchafei et al., 10 Apr 2026). In medical QA, Boolean and Temporal facets have the highest failure rates, whereas in HotpotQA failure remains low across types (Elchafei et al., 10 Apr 2026). Overridden facets exhibit bimodal faithfulness distributions, which the paper interprets as confirming unpredictable integration or contradiction of evidence (Elchafei et al., 10 Apr 2026).

6. Workflow, reproducibility, and relationship to prior work

The end-to-end FACET pipeline comprises seven stages: facet extraction; retrieval of Top-K chunks and computation of similarities and retrieval margins; Facet × Chunk matrix construction; facet generation in the three controlled modes; NLI scoring; taxonomy assignment; and question-level answer synthesis with F1 and BERTScore evaluation (Elchafei et al., 10 Apr 2026). The paper specifies prompts and parameters in an appendix, but no code repository is announced (Elchafei et al., 10 Apr 2026). Retrieval configuration and model IDs are provided to aid replication.

The framework’s novelty is described in four parts: facet-level decomposition for atomic reasoning analysis; the Facet × Chunk matrix tracing retrieval relevance and NLI-based faithfulness for each reasoning step; controlled inference modes as diagnostic interventions to attribute hallucination sources; and an evidence taxonomy that exposes recurring failure patterns hidden by answer-level metrics (Elchafei et al., 10 Apr 2026). In this respect, FACET is best understood as an interpretive and auditing framework for RAG pipelines rather than as a new training or decoding method.

The paper also identifies several threats to validity. Reliability depends on the quality of facet decomposition and on NLI model fidelity, so noise or bias in either can affect taxonomy labels (Elchafei et al., 10 Apr 2026). The study is limited to two datasets and three LLMs, and results may vary for other tasks or languages (Elchafei et al., 10 Apr 2026). Evaluation is offline only; interactive behavior and user-facing calibration are not studied (Elchafei et al., 10 Apr 2026). These limitations imply that FACET’s strongest claims concern diagnostic visibility rather than universal generalization.

7. Practical implications for RAG design and monitoring

FACET motivates a monitoring regime centered on facet-level evidence use rather than answer-level correctness alone. The paper recommends enforcing facet-level grounding selectively, using Strict RAG at the facet level with NO_ANSWER fallback when support is unclear, while avoiding global strict gating that suppresses correct priors unnecessarily (Elchafei et al., 10 Apr 2026). It also recommends adaptive integration through Soft RAG with facet-aware decision rules, together with monitoring of when Soft RAG degrades performance (Elchafei et al., 10 Apr 2026).

For observability, the framework highlights three signals: retrieval margins $f_i$0 to detect ambiguous retrieval, NLI faithfulness $f_i$1 per facet, and bimodal faithfulness distributions in overridden facets (Elchafei et al., 10 Apr 2026). It further suggests prompt constraints emphasizing evidence precedence, reranking or filtering chunks that contradict facet answers, and aggregation designs that flag disagreements among chunks (Elchafei et al., 10 Apr 2026).

At deployment time, a facet-aware interface could surface whether a reasoning step was evidence helpful, overridden, or misaligned, enabling human review of unreliable steps (Elchafei et al., 10 Apr 2026). Continuous monitoring would then track facet-level override rate, failure rate, misalignment incidence, distributions of facet faithfulness by semantic type, and mode-differential performance across Soft RAG, Strict RAG, and LLM-only generation (Elchafei et al., 10 Apr 2026). A plausible implication is that FACET reframes RAG evaluation as a problem of evidence governance: not merely whether evidence was retrieved, but whether the model used it in the intended epistemic role.

In summary, FACET defines a facet-level diagnostic paradigm for RAG in which questions are decomposed into atomic reasoning units, evidence is traced through a two-signal Facet × Chunk matrix, and controlled inference modes expose whether failures arise from evidence absence, evidence misalignment, prior-driven overrides, or successful grounding (Elchafei et al., 10 Apr 2026). Its empirical findings challenge the assumption that retrieval quality alone determines hallucination behavior, and instead locate the dominant failure mode in generation-side evidence integration (Elchafei et al., 10 Apr 2026).