---
title: Extractive Evaluation Techniques
url: https://www.emergentmind.com/topics/extractive-evaluation-techniques
type: topic
---

# Extractive Evaluation Techniques

Extractive evaluation techniques are a class of methodologies and metrics developed to assess the quality and properties of extractive summarization systems. These techniques address limitations of standard lexical-overlap metrics by incorporating semantic coverage, redundancy control, information facet coverage, and faithfulness, ensuring a more holistic and fine-grained evaluation. Recent work has produced principled metrics that address challenges such as ranking quality, adaptability to multiple references, redundancy sensitivity, and error-specific faithfulness assessment.

## 1. Semantically and Gain-based Evaluation: Redundancy-Aware Sem-nCG

The redundancy-aware Sem-nCG metric generalizes normalized cumulative gain (nCG) from information retrieval to semantic evaluation of extractive summarization. Given a document $D$ (sentences $S_1,\dots,S_n$) and a set of reference summaries $R_1,\dots,R_m$, each sentence’s semantic relevance to the references is computed (e.g., with cosine similarity of STSb embeddings), forming an “ideal” sentence ranking for the document.

For a summary of $k$ extracted sentences $X$, cumulative gain $\mathrm{CG}@k$ is the sum of relevance scores of its top-$k$ sentences; $\mathrm{ICG}@k$ is the total semantic gain from the ideal ranking. The normalized gain is:
\[
\text{Sem-nCG}@k = \frac{\mathrm{CG}@k}{\mathrm{ICG}@k}.
\]
To penalize redundancy, a diversity-controlled penalty $\text{Score}_\mathrm{red}(X)$ is applied:
\[
\text{Score}_\mathrm{red}(X) = \frac{1}{|X|}\sum_{i=1}^{|X|} \max_{j\neq i} \text{Sim}(x_i,x_j)
\]
where $\text{Sim}$ can be any sentence similarity, such as ROUGE, BERTScore, or cosine similarity.

A tunable parameter $\lambda\in[0,1]$ trades off importance and diversity:
\[
\text{Final Score}(X) = \lambda \cdot \text{Sem-nCG}@k + (1-\lambda)(1-\text{Score}_\mathrm{red}(X))
\]
With $\lambda=0.5$, the metric demonstrates improved correlation with human judgments of Coherence (+14% Kendall’s $\tau$) and Relevance (+5% $\tau$) over both ROUGE and the original Sem-nCG. The metric supports multi-reference evaluation by averaging relevance across reference summaries, using ensemble similarity or rank-aggregation strategies to derive a unified ideal ranking [2308.02270].

## 2. Facet-Aware Coverage and Support Metrics

The facet-aware evaluation framework decomposes a reference summary into information “facets”—one per sentence—and determines which subsets of document sentences (support groups) collectively cover the semantics of each facet. With gold or automatically identified facet-support mappings, evaluation proceeds by checking the extracted summary’s coverage:

- **Facet-Aware Recall (FAR):** Fraction of facets covered by any support group fully included in the extract.
\[
\text{FAR}(\mathcal{E}) = \frac{1}{R}\sum_{i=1}^{R} \max_{j=1}^{N_i} I(\mathcal{S}_j^i, \mathcal{E})
\]
- **Support-Aware Recall (SAR):** Fraction of all support sentences present in the system extract.

Facet-aware evaluation enables detailed analyses (e.g., distinguishing summary redundancy from true information coverage). FAR demonstrates better correlation with human rankings than ROUGE-1 F1 (ρ ≈ 0.457 versus 0.44) and supports comparative and fine-grained diagnostic analysis [1908.10383].

## 3. Faithfulness Detection: Error-Typology and Automated Metrics

Extractive summaries can exhibit multiple forms of unfaithfulness, despite directly copying from the source. Five fine-grained error types are labeled:

1. **Incorrect Coreference:** Anaphora refer incorrectly to antecedents from the original.
2. **Incomplete Coreference:** Anaphora in the summary lack clear antecedents.
3. **Incorrect Discourse:** Discourse linkers imply false relations due to extraction boundaries.
4. **Incomplete Discourse:** Discourse units or connectives are incomplete, confusing interpretation.
5. **Other Misleading Information:** Content selection biases or misleads beyond strict entailment.

The ExtEval metric detects these errors through four sub-detectors. It runs SpanBERT-based coreference resolution, explicit discourse marker identification, and sentiment difference calculation (RoBERTa). ExtEval is computed as the sum of binary indicators for each error (plus sentiment gap):
\[
\mathrm{ExtEval}(S,D) = f_1 + f_2 + f_3 + f_4
\]
Where each $f_i$ proxies a specific error subtype. On system-level ranking, ExtEval achieves Pearson $r \approx 0.96$ and Spearman $\rho \approx 0.88$ with human-judged faithfulness—substantially outperforming ROUGE, FactCC, and QuestEval [2209.03549].

## 4. Extractive Fragment Coverage and Density

To analyze *how* extractive a summary is, two canonical techniques are defined:

- **Extractive Fragment Coverage (Coverage):** The fraction of summary tokens that appear as contiguous fragments in the article.
\[
\text{Coverage}(A,S) = \frac{1}{|S|}\sum_{f\in\mathcal{F}(A,S)} |f|
\]
This quantifies literal copying; Coverage ≈ 1 for fully extractive summaries.

- **Extractive Fragment Density (Density):** The average squared length of each fragment per summary token.
\[
\text{Density}(A,S) = \frac{1}{|S|}\sum_{f\in\mathcal{F}(A,S)} |f|^2
\]
Higher Density indicates longer contiguous copied spans. These metrics distinguish between summaries with similar Coverage but differing composition (e.g., many single-word vs. a few long fragment matches). Newsroom analyses show wide variance in both coverage and density across publications and relative to other datasets like CNN/DailyMail [1804.11283].

| Metric             | Definition                        | Interprets           |
|--------------------|-----------------------------------|----------------------|
| Coverage           | Fraction of tokens in fragments    | Extractiveness       |
| Density            | Mean squared fragment length/token | Extractive grouping  |

## 5. Experimental Protocols, Correlation, and Analysis

Evaluation of extractive metrics follows structured protocols:

- Correlating single/multi-reference metric scores to expert human ratings on dimensions such as Consistency, Relevance, Coherence, and Fluency.
- For redundancy-aware Sem-nCG, λ=0.5 yields robust trade-offs; multi-reference strategies (Ensembleₛᵢₘ, Ensembleᵣₑₗ) stabilize ranking compared to per-ref metric averaging [2308.02270].
- Facet-aware experiments highlight that systems may achieve high support retrieval (SAR) while exhibiting localized redundancy, underscoring the need for nuanced evaluation [1908.10383].
- Faithfulness metrics reveal that even ROUGE-oracle summaries are frequently unfaithful; ExtEval enables targeted reranking or automatic post-editing to repair coreference/discourse errors [2209.03549].

## 6. Strengths, Limitations, and Future Directions

Extractive evaluation techniques now address multiple desiderata: semantic coverage, redundancy control, ranking quality, and faithfulness. Innovations such as redundancy-aware Sem-nCG and facet-aware recall yield better alignment with human preferences and uncover properties invisible to standard overlap-based metrics. All such methods have structural limitations:

- Dependency on embedding/similarity model calibration; robustness on highly abstractive or paraphrased content is limited.
- λ or threshold hyperparameters typically require domain-specific tuning.
- Most approaches currently evaluate only extractive summaries or datasets with clear extractive structure.
- Annotation and mapping of facets/support groups for evaluation are resource-intensive; automatic methods improve precision but often sacrifice recall.

Potential future directions include learned, instance-adaptive weighting between importance and diversity, extending redundancy penalties and faithfulness detection to generative (abstractive) summaries, and incorporating factuality verification or reference-free evaluation modules [2308.02270, 1908.10383, 2209.03549].

---

In summary, contemporary extractive evaluation combines semantic gain, redundancy control, facet coverage, and error-typology-based faithfulness scoring, supplementing and superseding lexical overlap metrics for systematic, fine-grained, and human-aligned assessment of extractive summarization systems.

Source: https://www.emergentmind.com/topics/extractive-evaluation-techniques