SafePassage: Grounded IE Pipeline

• SafePassage is a model-agnostic pipeline for high-fidelity information extraction that ensures extracted data is grounded in source documents.
• It employs a three-stage process—LLM extraction, string-based alignment using Smith–Waterman, and evidential scoring—for robust verification.
• Empirical results show up to 85% hallucination reduction, with high agreement between pipeline judgments and human evaluations.

Searching arXiv for the SafePassage paper and closely related grounding/verification work. SafePassage is a model-agnostic pipeline for high-fidelity information extraction (IE) with black box LLMs. It defines a “safe passage” as context generated by the LLM that is both grounded in the document and consistent with the extracted information. The framework was introduced to address a central reliability problem in LLM-based IE: unlike traditional extraction pipelines, the information “extracted” is not guaranteed to be grounded in the source document. SafePassage operationalizes grounding through a three-step procedure—LLM extraction, string-based alignment, and evidence scoring—and uses the resulting judgments both as online guardrails and as an evaluation instrument for extraction quality. Reported results show hallucination reduction by up to 85% on IE tasks, with high agreement between pipeline judgments and human judgments (Barrow et al., 30 Sep 2025).

1. Problem setting and core definition

The SafePassage framework is situated in the broader shift from conventional IE systems to black box LLM-based extractors. In this setting, configuration is relatively easy—schemas can be specified in prompts and extracted values can be rendered in structured form—but trust is difficult because extracted fields may be unsupported, partially paraphrased, or fabricated. SafePassage addresses this by making evidential support a first-class output rather than an after-the-fact audit (Barrow et al., 30 Sep 2025).

Its central object is the safe passage itself: a snippet that both exists in the document and supports the extracted structured value. This definition has two separable requirements. First, the evidence string must actually be present in the source document. Second, the evidence must entail, or otherwise adequately support, the normalized extraction. The distinction matters because an LLM can produce plausible context that is not literally present, and it can also copy a real span that merely co-occurs with the answer without justifying the structured field.

The framework therefore treats each extraction as a pair $(y, c)$, where $y$ is the structured output and $c$ is a required context field. Unsafe extractions are then flagged or suppressed if the context fails either grounding or support checks. This makes SafePassage not only a hallucination filter, but also a concrete formalization of document-grounded IE for black box extractors.

2. Three-stage pipeline architecture

SafePassage consists of three sequential components: an LLM extractor, a string-based global aligner, and a scoring model (Barrow et al., 30 Sep 2025).

Stage	Output	Main mechanism
LLM extractor	Structured entity, raw string, context	Prompted extraction with required evidence field
Global aligner	Best-aligned document span $\hat{c}$	Smith–Waterman local sequence alignment at character level
Scoring model	Safe/unsafe judgment	NLI entailment or LLM grading

In the first stage, the LLM is prompted to return a structured entity together with the raw document string and a context field. The context is intended to be a minimal substring from the document that provides proof for the entity. The paper’s schema examples emphasize that grounding is not optional: the context field is always required. For normalized fields such as dates, the extractor may output a normalized representation such as ISO format, or None if normalization fails. This separates representation from evidence and makes downstream verification explicit.

In the second stage, the LLM-provided context is aligned back to the document. The aligner is described as string-based and global in purpose, but its core mechanism is local sequence alignment via the Smith–Waterman algorithm at the character level. This choice is motivated by real document noise, including OCR artifacts, typos, minor paraphrases, and small insertions or elisions. Exact substring matching would reject many otherwise valid grounded spans.

In the third stage, the system verifies whether the aligned context actually supports the extraction. Two scoring modes are described. One is a Natural Language Inference formulation in which the aligned span is the premise and a stringified entity is the hypothesis. The other is an LLM grader prompted to judge the pair using strict labels such as supports versus insufficient. The output of this stage determines whether the passage is safe.

3. Alignment and evidential verification

The alignment stage formalizes grounding as approximate document membership. Let $c$ denote the LLM-supplied context and $\hat{c}$ the best-aligned span in the document. The alignment score is defined as

$s_{\text{align}}(c,\hat{c}) = \frac{M}{L},$

where $M$ is the number of character matches and $L$ is the alignment length (Barrow et al., 30 Sep 2025).

Only $(y,c)$ pairs with alignment score above threshold are retained; the experiments use $y$0. The paper gives a representative example in which the context “hearing june 19, 2013” aligns to the document span “hearing l’audience june 19, 2013” with score $y$1. The example illustrates why character-level local alignment is preferable to exact matching in document collections affected by OCR or multilingual noise.

Support verification is then separated from string matching. In the NLI variant, the aligned context becomes the premise and the stringified entity becomes the hypothesis, with score

$y$2

In the LLM-grader variant, the verifier makes a binary judgment:

$y$3

This decomposition is one of the framework’s main conceptual contributions. Alignment answers whether the evidence exists in the document; scoring answers whether that evidence supports the normalized extraction. The paper explicitly notes that the scoring step is intended to prevent cases in which evidence merely co-occurs with the target field rather than justifying it.

4. Empirical performance and hallucination reduction

The main empirical claim is that SafePassage reduces hallucinations by up to 85% on IE tasks while maintaining minimal risk of flagging non-hallucinations (Barrow et al., 30 Sep 2025). In the reported experiments, the best-performing NLI scorer achieved recall of 85.7% for unsafe passages and precision of 92.8%, indicating that most hallucinated extractions were flagged while false alarms remained limited.

The reported scorer comparison is more specific. HHEM 2.1 obtains precision 0.800, recall 0.235, and F1 0.364. A DeBERTAv3 NLI baseline obtains precision 0.319, recall 0.210, and F1 0.253. The LLM grader reaches precision 0.878, recall 0.782, and F1 0.827. The best model, denoted $y$4, reaches precision 0.928, recall 0.857, and F1 0.891 after fine-tuning first on LLM-generated pseudolabels and then on 500 human-labeled examples.

High agreement with human judgments is another central result. SafePassage scores are reported to closely match subject matter expert labels both for hallucination detection and for model-level extraction quality. This is what enables dual use: the pipeline can serve as a production safeguard for extracted outputs and as an evaluation measure for ranking extractor models. The paper also reports that larger LLMs generally achieve higher SafePassage and human scores, with examples such as GPT-4.1 outperforming Mini and Nano variants, and Gemini 2.5 Pro outperforming Flash and Flash-Lite; it further notes exceptions, including o4-mini outperforming all and Llama 4 Scout struggling with structured output instructions.

These findings suggest that SafePassage is not limited to post hoc filtering. It also functions as a document-grounded measurement interface for black box models whose internal uncertainty estimates are unavailable.

5. Scoring models, annotation regimes, and cost structure

A notable feature of SafePassage is the comparison between flexible but expensive LLM grading and lightweight task-specific inference models. The LLM scorer generalizes to new entity types without retraining, but it incurs higher computational cost and latency. By contrast, the transformer encoder scorer requires task-specific fine-tuning when new entity types or formats are introduced, but once tuned it can outperform the LLM grader (Barrow et al., 30 Sep 2025).

The paper describes this outcome as surprising: a transformer encoder fine-tuned on a small number of task-specific examples can outperform an LLM scoring model at flagging unsafe passages. The reported annotation burden is low. About 500 human labels—collectable in as little as 1–2 hours—are sufficient for the small NLI model to match or exceed the LLM grader. Pretraining on silver labels generated by the LLM further improves performance across label counts.

The cost differential is also concrete. The transformer encoder scorer is reported as approximately 2000 times less expensive than the LLM scorer, with throughput around 250 predictions per second on a common GPU. For fixed-schema settings such as legal IE, the paper recommends the lightweight NLI scorer because it offers both accuracy and speed. For exploratory settings or rapidly changing schemas, the LLM scorer remains useful because it avoids repeated retraining.

One important consequence is that “model-agnostic” in SafePassage refers primarily to the extraction stage and the overall architecture, not to a single universal verifier that dominates across all deployment settings. Verification can be specialized when the schema is stable.

6. Uses, misconceptions, and significance

SafePassage can be deployed as an online guardrail or as an offline evaluation framework. In an online configuration, unsupported extractions are flagged or suppressed before downstream use. In an offline configuration, SafePassage provides a way to compare black box extractors according to groundedness rather than only task-level extraction accuracy (Barrow et al., 30 Sep 2025).

Several common misunderstandings are addressed by the architecture itself. One is that requiring the LLM to output a context field is sufficient for grounding. SafePassage rejects that view by inserting an explicit alignment stage, because model-generated context may not actually exist in the document. A second misunderstanding is that document membership alone guarantees faithfulness. The scoring stage rejects this as well: a real span can still be evidentially insufficient for the structured value. A third is that the strongest available LLM should also be the best verifier. The reported results contradict this in fixed-schema settings, where a small fine-tuned NLI model surpasses the LLM grader on precision, recall, and F1.

The broader significance of SafePassage lies in its operational definition of grounded extraction. Rather than treating factuality as a vague property of model outputs, it decomposes reliability into extract-then-align-then-verify. This decomposition is technically modest but methodologically consequential. It provides a mechanism for auditing black box IE systems without requiring internals, token-level provenance, or bespoke retraining of the extractor itself. In that sense, SafePassage occupies a practical middle ground between unconstrained prompting and fully custom traditional IE pipelines: it preserves the configurability of LLM-based extraction while restoring an explicit evidential contract between document, extraction, and support (Barrow et al., 30 Sep 2025).