---
title: 'GUM-SAGE: Graded Entity Salience Model'
url: https://www.emergentmind.com/topics/gum-sage-model
type: topic
---

# GUM-SAGE: Graded Entity Salience Model

GUM-SAGE is a dataset and methodology for graded entity salience prediction in English documents. Graded entity salience quantifies entity importance on a discrete 0–5 scale, reflecting the relative prominence of entities within a text based on their inclusion in multiple reference summaries. The GUM-SAGE framework combines human and LLM-generated one-sentence summaries, robust entity–summary alignment, and lightweight model architectures to establish new benchmarks in graded salience correlation, substantially surpassing both position-based heuristics and few-shot prompting of large language models [2504.10792].

## 1. Dataset Construction

GUM-SAGE is constructed atop the Universal Dependencies English GUM corpus, encompassing 213 documents from 12 genres: academic, biography, conversation, fiction, interview, news, reddit, speech, textbook, vlog, voyage, and wikihow. The corpus contains approximately 203,781 tokens, 57,360 mentions, and 32,300 distinct entities (both named and non-named), averaging 148 entities per document.

Each document is paired with five aligned summaries:

1. **One “gold” expert-written summary**, with 24 test documents receiving a second gold summary.
2. **Four “silver” summaries (training)** generated by prompting off-the-shelf LLMs (GPT-4o, Claude 3.5 Sonnet, Llama 3.2 3B Instruct, Qwen 2.5 7B Instruct) for single-sentence, ≤380 character summaries.
3. **Four additional human summaries (development/test)** crowdsourced from graduate students under strict guidelines for consistency and factuality.

This results in exactly five summaries per document across all splits. Entity salience scores are computed as:

\[
S(e)\;=\;\sum_{j=1}^{5}\mathbf{1}\bigl(e\in \text{summary}_j\bigr)
\]

where $S(e)\in\{0,1,2,3,4,5\}$. An entity is non-salient if $S(e)=0$, with $S(e)=1$–5 interpreted as a graded measure of salience.

The corpus exhibits genre-related variation: on average, 13.8% of entities per document have nonzero salience, rising to 32.9% in wikihow and dropping to 6.3% in academic texts, highlighting strong genre dependence in entity inclusion within summaries.

## 2. Graded Entity Salience Modeling Approaches

GUM-SAGE evaluates two primary task-specific modeling approaches:

### Stanza Coreference Model

Documents concatenated with each summary are processed using the Stanza Coreference system (XLM-RoBERTa-large, trained on CorefUD). For a given summary, any mention sharing a coreference cluster with a document mention is considered “present.” Each entity's summary-level presence is then summed to yield its $0\text{–}5$ salience score. No additional neural layers or task-specific fine-tuning are used.

### Ensemble Logistic-Regression Model

This method constructs a feature vector $\mathbf{x}(e,j)$ for each entity-summary pair, composed of:

- **Three binary alignment-module flags**:
    - String match (exact or high-overlap)
    - Stanza coreference
    - GPT-4o prompt-based match
- **Linguistic features** from GUM:
    - One-hot entity type (Person, Organization, Abstract, etc.)
    - One-hot genre (12 genres)
    - Normalized document position of first mention $[0,1]$

A logistic regression classifier predicts entity-summary alignment probability:

\[
p_{j}(e) = \sigma (\mathbf{w}^\top \mathbf{x}(e,j) + b),\quad \sigma(z)=\frac{1}{1+e^{-z}}
\]

The model, trained with binary cross-entropy loss on development-aligned data, thresholds outputs at 0.5 to determine presence, then sums results as before to compute entity salience.

Entities are not represented by contextual token embeddings at any stage; only alignment-based features inform salience estimation.

## 3. Training Regimen and Evaluation Protocol

For the Stanza coreference approach, the coreference system is applied as provided, with no domain-specific adaptation. The ensemble method uses off-the-shelf scikit-learn logistic regression (L2 regularization, $C=1.0$), trained solely on manually aligned dev data; no neural architectures or additional training loops are introduced.

Evaluation employs three main criteria:

- **Spearman’s $\rho$**: correlation between predicted and gold $S(e)$.
- **Calibration (RMSE)**: root mean square error between predicted and actual scores.
- **Top-$k$ Retrieval (precision, recall, F1)**:
    - Top1: Entities with true $S(e)=5$
    - Top3: Entities with $S(e)\in\{3,4,5\}$

Baselines include a position-only heuristic (first 10% of sentences $\rightarrow$ score 5, etc.), and direct zero- and few-shot prompting of LLMs (e.g., GPT-4o, Llama 3.2-Instruct, Mistral 7B-Instruct), with LLMs tasked to select salient entities and provide a 1–5 salience score.

## 4. Empirical Results

Key outcomes on the test set are summarized below:

| Model/Baseline        | Spearman’s $\rho$ | RMSE  | Top1 F1 | Top3 F1 |
|-----------------------|-------------------|-------|---------|---------|
| Position baseline     | 0.153             | 2.554 | —       | —       |
| GPT-4o 3-shot         | 0.254             | 1.111 | 0.405   | 0.361   |
| Llama 3.2-Instruct    | 0.223             | 1.296 | —       | —       |
| Mistral 7B-Instruct   | 0.254             | 1.206 | —       | —       |
| **Stanza coref**      | 0.384             | 1.031 | 0.321   | 0.448   |
| **Ensemble (ours)**   | **0.540**         | 1.067 | 0.367   | 0.527   |

The ensemble’s $\rho$ improvement over GPT-4o 3-shot is statistically significant (Wilcoxon $p<0.001$). The ensemble also achieves highest recall for top-1 salient entities (0.755) but at some cost to precision. Top-3 F1 performance is strongest for the ensemble (0.527), confirming that combining multiple alignment signals enhances graded salience retrieval.

## 5. Analysis and Ablations

A series of ablations isolates alignment module contributions:

- String match only: micro-F1 = 0.56
- GPT-4o alignment only: micro-F1 = 0.75
- Stanza coref only: micro-F1 = 0.77
- All three plus linguistic features (ensemble): micro-F1 = 0.98 (positive-class F1 = 0.90)

This demonstrates that integrating both precision-oriented (string match, GPT-4o) and recall-oriented (Stanza coref) alignment signals with document-level features yields substantial improvements in alignment, which translate to graded salience prediction gains.

Error analysis surfaces several trends:

1. Both LLM and alignment-based models over-predict mid-range scores (3/4) for true low-salience entities, reducing calibration at the extremes.
2. Entity-type breakdown shows that concrete types (animal, plant, organization) exhibit high LLM precision/recall, while abstract/event/time entities remain challenging. Person/place types display high recall but suffer from over-prediction, lowering precision.
3. False positives are concentrated among early-mentioned entities (“over-salience” due to positional bias), whereas false negatives are more evenly distributed.

Coarse features such as entity type, genre, and entity position stabilize predictions, particularly for entities with limited summary evidence.

## 6. Context and Implications

GUM-SAGE’s proxy-based, summary-alignment-driven approach offers increased stability compared to subjective gradient scoring and increased expressivity versus traditional summarization-based binary salience. The methodology highlights the limitations of zero-/few-shot LLM prompting for fine-grained entity ranking and formalizes multi-summary alignment as a robust, explainable method for graded salience prediction.

A plausible implication is that summary-derived, graded salience signals may generalize to user-facing applications in summarization, browsing, and document interpretation where calibrated entity prominence is necessary. The GUM-SAGE dataset and codebase are publicly released to enable further research on multi-document, multi-reference, and genre-sensitive salience modeling.

Source: https://www.emergentmind.com/topics/gum-sage-model