SEval-Ex: Factuality Evaluation Framework

• SEval-Ex is a statement-level framework that evaluates factual consistency by decomposing source and summary texts into atomic statements with evidence-backed LLM verification.
• It extracts concise statements to classify true positives, false positives, and false negatives, enabling precise computation of precision, recall, and F1 scores.
• The StSum_Text variant enhances local context preservation and achieves a Spearman correlation of 0.580 on SummEval, outperforming several prior evaluation methods.

Searching arXiv for SEval-Ex and closely related summarization-evaluation work to ground the article. arXiv search query: (Herserant et al., 4 May 2025) SEval-Ex summarization evaluation factual consistency SEval-Ex is a statement-level, explainable evaluation framework for assessing the factual consistency of a generated summary with respect to its source document. It decomposes both the source and the summary into atomic statements and then aligns and classifies them to produce transparent, evidence-backed scores. The framework was introduced to address a persistent trade-off in summarization evaluation: summary-level, “black-box” evaluators can correlate well with human judgments but offer limited interpretability, whereas interpretable metrics often struggle with paraphrase, semantic equivalence, and factuality. SEval-Ex focuses specifically on factual consistency rather than sentence importance, relevance, or coherence, and on the SummEval benchmark it attains a Spearman correlation of 0.580 with human consistency judgments, surpassing GPT-4-based evaluators cited in the paper’s narrative at 0.521 while maintaining statement-level evidence (Herserant et al., 4 May 2025).

1. Problem formulation and design objective

SEval-Ex is motivated by two difficulties in summarization evaluation. First, metrics operating at the surface level, such as lexical overlap metrics, miss semantic equivalence and can conflate fluency with correctness. Second, strong LLM-based evaluators can achieve good correlation with human ratings but remain expensive, opaque, and sensitive to prompt design and positional biases. SEval-Ex addresses this by reducing the evaluation problem to verification over atomic statements: self-contained units of information expressed in natural language that can be checked for truth against the source without relying on broader discourse (Herserant et al., 4 May 2025).

The framework evaluates factual consistency, also described as “faithfulness,” by asking whether each summary statement is supported by the source. This decomposition yields statement-level alignment evidence, a transparent error structure in terms of true positives, false positives, and false negatives, and summary-level precision, recall, and F1 scores that are intended to be intuitive and explainable. Because the design is centered on factual verification, it does not attempt to model sentence importance, relevance, or discourse coherence; this specialization is essential to its interpretation and to the scope of its reported performance (Herserant et al., 4 May 2025).

A plausible implication is that SEval-Ex should be understood less as a universal summarization metric than as a specialized factuality evaluator. This suggests that it is most informative when paired with complementary measures for fluency, coherence, or relevance rather than used as a single scalar surrogate for all aspects of summary quality.

2. Atomic statements, alignment, and formal scoring

The central representational unit in SEval-Ex is the atomic statement. The source document $D$ and summary $S$ are decomposed into extracted statement sets, written in the paper as

$$S_{\text{src}} = \{s_i\}_{i=1}^m,\qquad S_{\text{sum}} = \{t_j\}_{j=1}^n.$$

The paper also gives equivalent notation

$$E(D) = \{d_1, d_2, \ldots, d_n\},\qquad E(S) = \{s_1, s_2, \ldots, s_m\}.$$

A verification function then labels relationships between source and summary statements:

$$V: E(D) \times E(S) \to Y,\qquad Y = \{TP, FP, FN\}.$$

In practice, this labeling is driven by an LLM-based notion of semantic equivalence, denoted $\equiv$, rather than by a fixed structural representation such as subject–predicate–object triples (Herserant et al., 4 May 2025).

Under this formulation, a summary statement $t_j$ is a true positive if there exists a source statement $s_i$ such that $s_i \equiv t_j$. A summary statement is a false positive if no source statement is semantically equivalent to it. A source statement is a false negative if there is no semantically equivalent summary statement. Equivalence means expressing the same fact despite paraphrase or lexical differences, as judged by the LLM. The framework therefore formalizes consistency as statement support rather than lexical overlap or sentence-level entailment alone (Herserant et al., 4 May 2025).

The summary-level score is computed from the statement counts:

$$\text{Precision} = \frac{TP}{TP + FP},\qquad \text{Recall} = \frac{TP}{TP + FN},$$

$S$0

The paper expresses the final scoring function concisely as

$S$1

This formulation makes precision the driver of faithfulness, since it measures the proportion of summary statements supported by the source, while recall captures the proportion of source facts reflected in the summary. F1 serves as the paper’s summary-level consistency score (Herserant et al., 4 May 2025).

The paper explicitly contrasts this design with a classical maximum-weight bipartite matching formalization,

$S$2

subject to one-to-one constraints on $S$3, but states that SEval-Ex deliberately does not compute an explicit similarity matrix, does not define numeric $S$4 values, and does not solve a Hungarian assignment. Instead, equivalence is decided categorically by an LLM using the source text in context (Herserant et al., 4 May 2025).

3. Pipeline architecture and operational variants

SEval-Ex uses a two-stage LLM-centric pipeline followed by a parser. The first stage is statement extraction via a function $S$5 that decomposes both source document and summary into concise, self-contained factual propositions. The second stage is statement matching and verdict reasoning via a function $S$6, which labels summary statements as supported or unsupported and identifies source statements that remain unmatched. A final parsing step reads the structured LLM outputs, constructs the confusion matrix, and computes precision, recall, and F1 (Herserant et al., 4 May 2025).

The extraction stage has two reported variants. In the Base variant, a single LLM call is applied to the entire text. This is computationally efficient but is described as prone to losing local context in long documents. In the 3-Chunk variant, the text is segmented into three-sentence chunks and extraction is performed on each chunk separately; the paper reports that this improves local context preservation and statement extraction accuracy. The framework does not prescribe canonicalization of entities or automatic coreference resolution beyond the LLM’s own capabilities, and any additional filtering is prompt-based rather than algorithmic (Herserant et al., 4 May 2025).

The crucial improvement in the verification stage is the “StSum_Text” variant, in which summary statements are matched against the source text directly, rather than against isolated extracted source statements. This design is presented as preserving context, reducing semantic drift, and avoiding information loss. The LLM is prompted to verify whether a summary statement is entailed by the source and, if so, to locate supporting evidence spans in the source (Herserant et al., 4 May 2025).

The matching stage is therefore not an exhaustive pairwise comparison over all source–summary statement pairs. Instead, it integrates verification into the prompt itself and asks the LLM to produce categorical decisions and, optionally, highlighted evidence sentences. The paper reports no explicit thresholds, no entailment probabilities, and no additional calibration steps beyond prompt optimization and chunking strategies. This places the framework conceptually between statement extraction systems and LLM-as-judge evaluators: it retains the granularity of decompositional methods while avoiding a full combinatorial matching procedure (Herserant et al., 4 May 2025).

The paper provides pseudocode for the StSum_Text variant. After extracting source and summary statements, each summary statement is checked against the source document by an LLM verifier. Supported statements form the TP set; unsupported statements form the FP set; and false negatives are then defined as source statements for which no entailed summary statement is semantically equivalent. This ordering is important because FN is not produced independently by a separate classifier but inferred from coverage relative to the supported summary statements (Herserant et al., 4 May 2025).

4. Explainability, evidence generation, and hallucination analysis

The distinguishing feature of SEval-Ex is its evidence trail. For each summary statement, the framework can label it as entailed and point to supporting source sentences, label it as unsupported and thereby surface likely hallucinations, or identify source statements that remain unmatched and thus represent coverage gaps. The framework’s explainability is therefore not post hoc in the sense of a separate rationale generator; it is built into the statement-level verification process itself (Herserant et al., 4 May 2025).

The paper gives illustrative examples. A source sentence stating that “The FDA approved the vaccine in December 2020” and a summary statement saying “The vaccine received FDA approval in Dec. 2020” are treated as semantically equivalent and labeled TP. A source claiming that a company reported a net loss in Q3 and a summary claiming it posted a profit in Q3 yields an FP verdict, because the source states the opposite. If the source contains a fact such as “The merger was finalized on May 15” and the summary omits it entirely, that source statement is counted as FN (Herserant et al., 4 May 2025).

The robustness study focuses on synthetic hallucinations inserted into SummEval summaries. The paper defines three hallucination types: Entity Replacement, Incorrect Events, and Fictitious Details. The evaluation uses a balanced dataset of 1,600 samples from SummEval, with each summary modified by one hallucination type using controlled prompting. The reported mean correctness-score drops are 0.435 for Entity Replacement, from 0.964 to 0.529; 0.650 for Incorrect Events, from 0.970 to 0.328; and 0.223 for Fictitious Details, from 0.973 to 0.750. All differences are reported as statistically significant with $S$7 (Herserant et al., 4 May 2025).

These results show that SEval-Ex penalizes unsupported statements, with larger drops for entity and event perturbations than for plausible embellishments. The paper attributes this to the fact that a single altered predicate can invalidate an entire atomic statement. This suggests that the granularity of the atomic-statement representation makes the framework particularly sensitive to local factual corruption, even when the resulting summary remains superficially plausible.

5. Empirical evaluation on SummEval

The reported main benchmark is SummEval, described in the paper as comprising 1,600 summaries from 16 systems on CNN/DailyMail with human annotations for coherence, consistency, fluency, and relevance. SEval-Ex targets the consistency dimension through factual statement verification and does not prioritize the other dimensions (Herserant et al., 4 May 2025).

The paper reports Spearman’s rank correlation as the evaluation metric, using

$S$8

where $S$9 is the rank difference for item $$S_{\text{src}} = \{s_i\}_{i=1}^m,\qquad S_{\text{sum}} = \{t_j\}_{j=1}^n.$$0 and $$S_{\text{src}} = \{s_i\}_{i=1}^m,\qquad S_{\text{sum}} = \{t_j\}_{j=1}^n.$$1 is the number of items. For factual consistency, the reported correlations are 0.231 for the Base pipeline, 0.306 for the 3-Chunk variant, and 0.580 for StSum_Text. The paper interprets these gains as supporting two hypotheses: preserving local context improves statement extraction, and directly comparing summary statements to the source text improves verification reliability (Herserant et al., 4 May 2025).

The benchmark comparisons reported in the paper place traditional n-gram metrics such as ROUGE-1/2/L at approximately 0.11–0.19 on consistency, embedding metrics such as BERTScore and MOVERScore at approximately 0.11–0.16, QuestEval at 0.306, UniEval at 0.446, GPTScore at 0.449, and G-Eval with a reported baseline around 0.521 in the narrative, with a table entry showing 0.507 for its best consistency setting. SEval-Ex attains 0.580 on consistency. On other dimensions, it reports moderate correlations: 0.351 for fluency, 0.264 for coherence, and 0.300 for relevance (Herserant et al., 4 May 2025).

Variant or metric	Consistency correlation
Base	0.231
3-Chunk	0.306
StSum_Text	0.580

Compared approach	Consistency correlation
QuestEval	0.306
UniEval	0.446
GPTScore (GPT-3)	0.449
G-Eval baseline in narrative	0.521
SEval-Ex	0.580

The progression from Base to 3-Chunk to StSum_Text is especially informative because it isolates two distinct sources of improvement within the same framework. First, chunking improves extraction quality by preserving local context. Second, replacing statement-to-statement comparison with direct statement-to-source verification produces the largest gain. This suggests that in decompositional factuality evaluation, contextual preservation during verification may matter more than refining the extraction representation alone.

6. Implementation, limitations, and relation to prior approaches

SEval-Ex uses Qwen2.5:72B with 4-bit quantization, selected after comparative experiments. The framework relies on prompting rather than training or fine-tuning, and is therefore described as a reference-free evaluator. The code and prompt templates are to be released at the project repository. The paper does not report hardware or runtime, but states that computational cost is dominated by LLM inference; the 3-Chunk variant increases the number of calls, whereas StSum_Text increases context length per call (Herserant et al., 4 May 2025).

The framework has several explicit limitations. It is specialized for factual consistency and does not model sentence importance, so relevance is only partially captured. Coherence and fluency are not primary targets, and their lower reported correlations are consistent with that specialization. Performance depends on prompt design, and transfer to new domains may require prompt re-tuning. Long documents remain challenging even with chunking, because large source contexts strain LLM context windows. The verifier itself is an LLM and can inherit biases or hallucinate judgments, although the framework attempts to mitigate this by requiring evidence. Finally, all atomic statements are weighted equally, so trivial supported facts and central facts contribute equally unless additional post-processing is introduced (Herserant et al., 4 May 2025).

The paper situates SEval-Ex relative to several families of prior methods. Summary-level scorers such as ROUGE, BLEU, BERTScore, and MoverScore are described as efficient but unreliable for factual consistency. NLI-based metrics such as FactCC and SummaC compute entailment between sentences but face long-document scalability and pairing-complexity issues. QA-based approaches such as QAGS and QuestEval improve sensitivity to facts but raise their own explainability and cost issues. LLM-as-judge methods such as GPTScore and G-Eval have strong average correlations but remain black-box and expensive. SEval-Ex’s novelty is defined in the paper as an end-to-end, statement-level pipeline that outputs TP, FP, and FN together with evidence while retaining strong consistency correlation (Herserant et al., 4 May 2025).

A common misconception would be to interpret SEval-Ex as a general-purpose summary evaluator because it outputs a single F1 score. The paper’s own framing argues against this interpretation: the score reflects factual statement verification, not salience, stylistic quality, or discourse organization. A second misconception would be to treat it as a classical graph-matching method over extracted facts. The paper explicitly states that it does not compute an explicit similarity matrix and does not solve a Hungarian assignment; the decisive step is in-context LLM verification against source text (Herserant et al., 4 May 2025).

SEval-Ex thus occupies a specific methodological niche in summarization evaluation. It reframes consistency assessment as alignment over atomic statements, couples that decomposition with source-grounded LLM verdict reasoning, and returns an evidence-rich confusion structure rather than only a summary-level scalar. Its reported results on SummEval and under controlled hallucination perturbations indicate that explainability and competitive empirical performance need not be treated as mutually exclusive objectives in factuality evaluation (Herserant et al., 4 May 2025).