---
title: Selective Evidence-Grounded Finalizer
url: https://www.emergentmind.com/topics/selective-evidence-grounded-finalizer
type: topic
---

# Selective Evidence-Grounded Finalizer

Searching arXiv for the cited paper and closely related selective RAG verification work.
Selective Evidence-Grounded Finalizer denotes a retrieval-augmented generation (RAG) component that does not treat retrieval as proof, but instead verifies whether the retrieved evidence is actually sufficient to justify a candidate answer, and answers only when support is established. In "SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generation" [2605.03534], this role is instantiated as a three-way evidence sufficiency verifier over \((q,a,E)\), where \(q\) is a question, \(a\) is a candidate answer, and \(E=\{e_1,\dots,e_k\}\) is the retrieved evidence set. The central premise is that evidence sufficiency is a set-level property: topical passages may still fail to justify an answer because of missing hops, partial support, or unresolved conflicts, so selective answering requires verification rather than retrieval relevance alone [2605.03534].

## 1. Task semantics and selective answering

The paper formulates evidence sufficiency verification as prediction of

\[
y \in \{Supported, Refuted, Insufficient\}
\]

from \((q,a,E)\) [2605.03534]. The semantics are explicit. **Supported** means that the evidence entails the central factual content of \(a\). **Refuted** means that the evidence contradicts at least one central factual claim in \(a\). **Insufficient** means that the evidence cannot justify \(a\), including missing hops, partial support, topical-but-not-sufficient passages, and unresolved conflicts.

This three-way decision is converted into selective answering by the mapping:

- Supported \(\rightarrow\) Safe / Answer
- Refuted or Insufficient \(\rightarrow\) Unsafe / Abstain

Within this formulation, the verifier is not merely a classifier. It is a reject-option verifier that abstains unless the evidence is sufficiently strong [2605.03534]. The paper defines the accepted set under a thresholded selective score \(s\) as

\[
A_\tau = \{ i : s(q_i,a_i,E_i)\ge \tau \}
\]

with

\[
\mathrm{Coverage}(\tau)=\frac{|A_\tau|}{N}, \qquad \mathrm{Risk}(\tau)=\frac{\#\text{ unsafe answers in }A_\tau}{|A_\tau|}.
\]

These definitions make selective answering operational in terms of both how often the system answers and how often those answers are unsafe [2605.03534].

A plausible implication is that the notion of “finalizer” is narrower than end-to-end answer generation. The component is positioned after retrieval and candidate-answer production, and its role is to authorize or block the answer on evidence-grounded sufficiency criteria.

## 2. Shared pair-level verification and answer decomposition

SURE-RAG has three components: pair-level claim-evidence verification, answer-level sufficiency aggregation, and selective answering [2605.03534]. The answer \(a\) is decomposed into claims \(C(a)=\{c_1,\dots,c_m\}\). For each claim \(c_j\) and evidence passage \(e_i\), a shared DeBERTa-v3-base cross-encoder produces a local relation distribution,

\[
\mathbf{p}_{ij}=h_\theta(q,c_j,e_i)= [p_{ij}^{\mathrm{sup}, p_{ij}^{\mathrm{ref}, p_{ij}^{\mathrm{neu}] 
\]

corresponding to supported, refuted, and neutral [2605.03534].

The use of a shared pair verifier across SURE-RAG and the pooling baselines is methodologically important because any gain from SURE-RAG comes from the aggregation logic, not from a stronger encoder. The paper also notes that for long-form answers it extends FActScore-style atomic claim decomposition, while the main benchmark focuses on short-answer multi-hop verification [2605.03534].

This design separates local relational judgments from answer-level sufficiency. A single claim-passage pair can be locally supportive or refutational, yet the full answer may still be insufficient if another required claim is uncovered or if the evidence collection contains unresolved contradiction. This suggests that decomposition is not only a transparency device but also a structural precondition for set-level verification.

## 3. Set-level sufficiency aggregation and interpretable signals

The paper’s key idea is that sufficiency is a set-level property of the evidence collection, so pairwise scores must be aggregated into answer-level evidence structure [2605.03534]. SURE-RAG constructs an answer-level feature vector \(\mathbf{z}(q,a,E)\) from the set \(\{\mathbf{p}_{ij}\}\), then predicts

\[
P(y\mid q,a,E)=g_\phi(\mathbf{z}(q,a,E)),
\]

where \(g_\phi\) is a lightweight classifier over the aggregated features.

The aggregation produces five kinds of signals:

| Signal block | Features or role |
|---|---|
| Coverage | supported claim fraction; refuted claim fraction; insufficient claim fraction |
| Relation strength | max support; max refutation; mean neutral |
| Disagreement | dispersion-style signal derived from pair-level distributions |
| Conflict | support-refute contradiction across passages |
| Retrieval uncertainty | optional retrieval-side signal from retriever or retrieval metadata |

Coverage is derived by per-claim argmax over passage-level labels. Relation strength captures whether any passage provides a strong local signal. Disagreement measures internal inconsistency in the evidence collection. Conflict is intended to detect unresolved contradiction among retrieved passages. Retrieval uncertainty is optional and can be removed in no-oracle tests to check whether retrieval metadata itself is acting as a shortcut [2605.03534].

The paper’s rationale for these blocks is sharply contrasted with simpler pooling. Max pooling over-commits if one passage looks supportive but another hop is missing; mean pooling dilutes strong refutation or support; sufficiency requires understanding coverage, conflicts, and uncertainty across the full set [2605.03534]. This suggests that the “finalizer” is fundamentally an aggregation protocol rather than just a claim-passage scorer.

## 4. Decision rule, uncertainty penalty, and calibration

The pipeline outputs a three-way class distribution \(\boldsymbol{\pi}=g_\phi(\mathbf{z})\), and chooses

\[
\hat{y} = \arg\max_y \pi_y.
\]

For selective answering, SURE-RAG defines the selective score

\[
s(q,a,E)=P(Safe)-\beta u(q,a,E),
\]

where \(P(Safe)\) is \(P(Supported)\), \(u\) is an uncertainty penalty, and \(\beta\) is a risk weight [2605.03534]. The uncertainty penalty combines predictive entropy, evidence disagreement, conflict, coverage deficit, and optional retrieval uncertainty.

The final decision rule is:

1. Decompose \(a\) into claims \(C(a)\)
2. Compute all pairwise \(\mathbf{p}_{ij}\)
3. Aggregate into \(\mathbf{z}(q,a,E)\)
4. Predict \(\boldsymbol{\pi}=g_\phi(\mathbf{z})\)
5. Set \(\hat{y}=\arg\max_y \pi_y\)
6. Compute \(s=\pi_{Supported}-\beta u\)
7. Answer iff \(\hat{y}=Supported\) and \(s\ge \tau\)

This rule means that even if the classifier predicts Supported, the system can still abstain when uncertainty is too high [2605.03534]. Post-hoc calibration parameters, threshold \(\tau\), and risk weight \(\beta\) are all selected on the development split, with no test labels used. The calibrated variant uses the same mechanism but improves calibration metrics [2605.03534].

A plausible implication is that the selective score functions as an auditable gate: the answer is conditioned not only on the class label but also on a structured uncertainty estimate whose components are individually interpretable.

## 5. Benchmark construction and artifact-aware evaluation

The primary benchmark is HotpotQA-RAG v3, derived from HotpotQA’s multi-hop structure. It contains **4026 examples** and **900 question groups**, with group-disjoint train/dev/test splits by original question ID [2605.03534]. The benchmark includes five evidence conditions:

1. **Full** — all supporting facts plus distractors; Label: Supported  
2. **Partial** — one necessary supporting fact removed; Label: Insufficient  
3. **Hard-insufficient** — topically related/high overlap but missing required relation; Label: Insufficient  
4. **Irrelevant** — only distractors; Label: Insufficient  
5. **Naturally refuted** — answer is perturbed while original evidence remains, so evidence contradicts answer; Label: Refuted

A notable detail is that pair labels are derived from construction metadata, not copied from answer labels. The benchmark yields **20,130 claim-evidence pairs total**, with **3292 support**, **2202 refute**, and **14636 neutral** [2605.03534].

The evaluation protocol is explicitly artifact-aware. It includes shortcut baselines, counterfactual evidence swaps, no-oracle checks, GPT-4o semantic audits, risk-coverage metrics, and external transfer on HaluBench [2605.03534]. The shortcut baselines are majority, hypothesis-only, evidence-only, length-only, overlap-only, and concat TF-IDF. The no-oracle checks remove retrieval scores or replace them with BM25. Counterfactual evidence swaps compare \(P(Supported)\) on full versus degraded evidence for the same \((q,a)\). This is presented as a major part of the contribution: not just benchmark performance, but robustness against artifacts.

## 6. Empirical results and task boundary with hallucination detection

On HotpotQA-RAG v3, the reported Macro-F1 results are:

| Method | Macro-F1 |
|---|---|
| SURE-RAG | 0.8951 \(\pm 0.0069\) |
| SURE-RAG calibrated | 0.9075 \(\pm 0.0060\) |
| DeBERTa max-pool | 0.5948 |
| DeBERTa mean-pool | 0.6516 |
| DeBERTa top-\(k\) | 0.6444 |
| Concat cross-encoder | 0.8888 |
| SURE pooling-only | 0.8383 |

These results show that SURE-RAG decisively beats all pooling baselines, matches or slightly exceeds the strong concat cross-encoder, and remains auditable because it exposes decomposition and aggregation signals [2605.03534].

For calibration, the paper reports binary ECE on safe/unsafe of **0.0304** for raw SURE-RAG and **0.0198** for calibrated SURE-RAG [2605.03534]. For selective answering, the low-coverage regime is the clearest strength: at **Risk@30**, max-pool is **0.3183**, mean-pool is **0.2588**, SURE-RAG is **0.1642**, and calibrated SURE-RAG is **0.1670**, corresponding to a **37% reduction in unsafe answers** relative to mean-pooling at 30% coverage [2605.03534]. At higher coverage, the advantage shrinks: at **Risk@50**, SURE-RAG is **0.3600** and mean-pool is **0.3783**; at **Risk@70**, SURE-RAG is **0.5457** and mean-pool is **0.5350**. The stated conclusion is therefore nuanced: SURE-RAG clearly helps in low-coverage selective ranking, but it does not solve high-coverage risk control.

The no-oracle checks report **0.8370 Macro-F1** when retrieval score is dropped and **0.8347 Macro-F1** when BM25 retrieval score is used; both remain above DeBERTa mean-pooling, so retrieval metadata is not the sole driver [2605.03534]. Counterfactual swap sensitivity is near-perfect: full versus partial, hard-insufficient, and irrelevant each have success **1.0000**, while full versus refuting has success **0.9679** [2605.03534]. This supports the claim that the method is sensitive to evidence sufficiency rather than only surface features.

The paper also uses a cross-task comparison with GPT-4o to define the task boundary. On matched HotpotQA-RAG v3 examples, **GPT-4o: 0.7284 Macro-F1** and **SURE-RAG: 0.8951 Macro-F1** [2605.03534]. On HaluBench unsafe detection, the ranking reverses: **GPT-4o: 0.7389 unsafe-F1**, **SURE-RAG: 0.3343 unsafe-F1**, and **best pooling baseline: 0.4863** [2605.03534]. The paper treats this reversal as direct evidence that controlled evidence sufficiency verification and natural hallucination detection are distinct problems.

In the paper’s distinction, sufficiency verification asks whether the candidate answer is justified by the retrieved evidence, and it is a three-way evidence-grounded decision over supported, refuted, and insufficient. Hallucination detection is broader, usually binary, more open-ended, and affected by longer outputs and broader semantic issues [2605.03534]. A response may be factually plausible but still unsupported by retrieved evidence; likewise, refuted and insufficient are semantically distinct in sufficiency verification. This suggests that a selective evidence-grounded finalizer is best understood not as a general hallucination detector, but as a specialized verifier for evidence-conditioned answer authorization.

Source: https://www.emergentmind.com/topics/selective-evidence-grounded-finalizer