VERIRAG: Evidence-Driven Claim Verification

• VERIRAG is a retrieval-augmented verification framework that assesses biomedical claims using an 11-point, machine-readable checklist inspired by CONSORT, STROBE, and PRISMA guidelines.
• It computes a Hard-to-Vary (HV) score by aggregating document quality, novelty, and redundancy metrics to dynamically adjust claim-specific acceptance thresholds.
• Empirical results show VERIRAG outperforms traditional RAG systems by 10–14 F1 points, demonstrating enhanced reliability in clinical and biomedical evidence auditing.

VERIRAG is a retrieval-augmented verification and statistical audit framework designed to vet the methodological rigor and evidential quality of scientific claims—principally in clinical and biomedical domains—surfacing within Retrieval-Augmented Generation (RAG) systems. Unlike conventional RAG approaches, which are agnostic to the quality and provenance of retrieved documents, VERIRAG integrates a machine-auditable checklist, quantitative document-level evaluation, and a claim-sensitive dynamic threshold. Its contributions address the inability of existing systems to distinguish between high-credibility and low-credibility sources in processes such as clinical decision support or expert-facing biomedical QA (Mohole et al., 23 Jul 2025).

1. Motivation and Problem Setting

RAG systems are frequently deployed in clinical, scientific, and evidence-centric contexts to support claim verification, literature review, and question answering. However, they typically function as passive retrievers, surfacing articles or passages based on keyword or embedding proximity, without assessing the internal quality or validity of the underlying studies. This methodological blindness results in problematic equivalence: retracted articles, anecdotal findings, and rigorous meta-analyses are treated interchangeably. A critical limitation is their incapacity to (a) audit source reliability, (b) account for methodological flaws, and (c) set claim-specific standards for evidentiary support.

VERIRAG was developed to address these deficiencies by providing a systematic, transparent, and auditable approach to claim vetting within the RAG paradigm, thereby aligning retrieval and evidence evaluation with the scientific method (Mohole et al., 23 Jul 2025).

2. The Veritable 11-Point Checklist

At the core of VERIRAG is the "Veritable" checklist: an 11-point, machine-readable audit inspired by CONSORT, STROBE, and PRISMA guidelines. Each retrieved document is evaluated for the following dimensions:

Checklist #	Audit Aspect	Principal Test/Measurement
C1	Data Integrity	Internal $N$ consistency across sections
C2	Missing Data Patterns	Imputation/bias handling evidence
C3	Sample Representativeness	Target-population alignment
C4	Outcome Variability	Dispersion reporting (CI, std, IQR)
C5	Estimation Validity	Test appropriateness for data
C6	Statistical Power	Power analysis, $\alpha$/$\beta$, sample calc
C7	Outlier Influence	Sensitivity/outlier analysis
C8	Confounding Control	Multivariable adjustment for confounders
C9	Source Consistency	Consistency of claim with prior art
C10	Effect Homogeneity	Meta-analysis heterogeneity ($I^2$, $Q$)
C11	Subgroup Consistency	Pre-specified vs. post hoc subgroup claims

Each check yields a Pass ($1$), Fail ($0$), or Uncertain ($0.5$), accompanied by a model-generated justification. Not every check applies to every paper; applicability is determined through a metadata-driven JSONPath filter. The result is an audit vector $v_{i,k}$ and a binary mask $m_{i,k}$ per document $\alpha$0 (Mohole et al., 23 Jul 2025).

3. HV Score: Aggregating Evidence by Quality and Diversity

VERIRAG’s quantitative backbone is the Hard-to-Vary (HV) score, which aggregates supportive, refuting, and neutral sources proportional to their intrinsic quality and novelty. The computation proceeds as follows:

1. Document Quality: For each $\alpha$1, quality is assessed as

$\alpha$2

where $\alpha$3.

1. Redundancy Penalty: To mitigate over-counting effectively duplicate evidence, redundancy is quantified per chunk $\alpha$4 within $\alpha$5 using cosine similarity in tf-idf space against preceding chunks. Let $\alpha$6 be this maximum similarity; novelty is $\alpha$7 (averaged over all $\alpha$8).
2. Weighting: Effective evidence contribution is $\alpha$9.
3. Aggregate Tallies: Summed weighted contributions for supporting ($\beta$0), refuting ($\beta$1), and neutral ($\beta$2) stances are computed.
4. HV Score:

$\beta$3

$\beta$4

Regularization parameters $\beta$5 and $\beta$6 are tuned on held-out validation data. This approach penalizes the total weight of neutral evidence and prevents singularities with low evidence counts (Mohole et al., 23 Jul 2025).

4. Dynamic Acceptance Threshold

Unlike static classifiers, VERIRAG employs a claim-specific, dynamically raised threshold $\beta$7 to implement the maxim “extraordinary claims require extraordinary evidence.” The threshold determination involves:

• Extraction of claim specificity ($\beta$8), testability ($\beta$9), and required standard ($I^2$0) via LLM prompt;
• Calibration by prior $I^2$1 for standard $I^2$2 and a ridge regression $I^2$3:

$I^2$4

• Scaling with incoming evidence count ($I^2$5) relative to an initial base count ($I^2$6):

$I^2$7

with $I^2$8 tuned to clamp $I^2$9 into $Q$0. The verdict for claim $Q$1 is “Valid” if $Q$2 and “Invalid” otherwise (Mohole et al., 23 Jul 2025).

5. Experimental Protocol and Empirical Performance

Evaluation was conducted on 100 human-curated biomedical claims from 200 source documents, comprising four temporal datasets—retracted-only (TY0), conflicting (TY1), comprehensive (TY3), and settled science (TY5). Each RAG system rendered 400 verdicts. Baselines included COT-RAG, Self-RAG, FLARE, and CIBER; the core metric was macro F1 on binary decision (Valid/Invalid).

Dataset	Best Baseline F1	VERIRAG F1	Absolute Gain
TY0	0.4017	0.5325	+0.1308
TY1	0.4243	0.5686	+0.1443
TY3	0.4902	0.5932	+0.1030
TY5	0.5315	0.6542	+0.1227

Ablation demonstrates that the HV Score and Dynamic Threshold both contribute critical increases to performance (removal drops F1 below 0.37 and 0.22, respectively). Agreement per audit item between LLM auditors and VERIRAG averages 88% (Cohen's $Q$3), indicating robust alignment (Mohole et al., 23 Jul 2025).

6. Operational Pseudocode and System Workflow

VERIRAG’s deployment follows a deterministic sequence:

1. Retrieve candidate documents for claim $Q$4 using embedding-based search.
2. For every $Q$5:
   • Parse methodology into JSON;
   • Segment into chunks;
   • Batch LLM prompt for stance ($Q$6), audit vector ($Q$7), and applicability mask ($Q$8).
3. For each $Q$9, calculate $1$0, redundancy $1$1, novelty $1$2, and effective contribution $1$3.
4. Aggregate $1$4, $1$5, $1$6; compute HV score per equations above.
5. Extract claim features and compute $1$7.
6. Compare HV against $1$8; output verdict.

This operational flow ensures every decision is rooted in quantitative scoring and verifiable audit trails (Mohole et al., 23 Jul 2025).

7. Significance, Limitations, and Prospective Impact

VERIRAG achieves a 10–14 absolute F1 point advantage over prompt-only and heuristic RAG baselines. A notable contribution is the integration of audit checklists and statistical controls directly into RAG pipelines, providing transparent evidence audits and dynamically adjusting to changing evidence landscapes. The system can flag unreliable claims when all retrieved evidence is methodologically weak—even in high-volume evidence settings—countering majority-vote or retrieval count bias.

Limitations include reliance on LLM-driven audit judgments and evidence parsing. Parameter sensitivity around redundancy penalties and threshold scaling may necessitate dataset-specific calibration. Future directions, as suggested, involve adaptation to modalities beyond text, refinement of audit criteria as reporting standards evolve, and possible extension to domains such as legal or regulatory verification that exhibit similar methodological heterogeneity (Mohole et al., 23 Jul 2025).

---

VERIRAG’s approach represents a significant advancement in trustworthy, evidence-centered retrieval-augmented claim verification, establishing a formal pipeline from machine-auditable evidence quality assessment to adaptive, claim-sensitive acceptance criteria. This enables RAG systems to surface not only relevant but also methodologically robust evidence in support of expert judgment.