Higher-Order ROUGE Variants

• Higher-order ROUGE variants are advanced metrics that extend traditional n-gram matching by integrating semantic equivalence and topic modeling to better reflect summary quality.
• They combine lexical overlap with techniques like Personalized PageRank and synonym expansion to achieve a higher correlation with human evaluations in summarization.
• Recent innovations include differentiable n-gram objectives and topic-centric measures that address reference variability and improve metric robustness across diverse domains.

Higher-order ROUGE variants extend the original ROUGE framework by incorporating n-gram matches beyond unigrams, integrating semantic equivalence, topic coverage, and robustness to reference variability. These developments address the limitations of traditional ROUGE, which relies on exact surface overlaps, making it less effective when evaluating summaries that employ paraphrasing, varied terminology, or abstractive techniques. The term "higher-order ROUGE" encompasses both metrics that operate over longer n-grams (such as ROUGE-2 and ROUGE-3), as well as recent variants that embed semantic or topic-level matching, synonym expansion, and differentiable formulations to improve alignment with human judgment and evaluation consistency.

1. Higher-order n-gram ROUGE and Empirical Performance

The original ROUGE metrics include ROUGE-N, which evaluates the overlap of n-grams (typically unigrams, bigrams, trigrams) between system and reference summaries. Empirical analysis reveals that, in scientific summarization, higher-order variants such as ROUGE-2 and ROUGE-3 exhibit substantially stronger correlation with manual Pyramid scores than lower-order variants. For example, ROUGE-1-F achieves a Pearson correlation of approximately 0.454, while ROUGE-3-F reaches 0.878 and ROUGE-2 variants score between 0.816–0.824 (Cohan et al., 2016). However, reliance on exact n-gram matching still limits the ability to account for paraphrasing or domain-specific vocabulary, particularly in cases of high information compression or terminology variation.

ROUGE Variant	Pearson r with Pyramid
ROUGE-1-F	0.454
ROUGE-3-F	0.878
ROUGE-2-F	0.816–0.824

This suggests that, while greater n is beneficial for specific technical domains, further enhancements are needed for broader generalization and semantic fidelity.

2. Semantic and Lexico-Semantic ROUGE Extensions

Recent work has introduced semantic and lexico-semantic approaches to ROUGE scoring, notably GRouge (ShafieiBavani et al., 2017). This variant computes scores by combining lexical overlap and semantic similarity using Personalized PageRank (PPR) vectors on the WordNet graph. Each word or n-gram is mapped to a sense (via alignment/disambiguation) and PPR vectors are generated, encoding the importance of each sense as random walks on the graph. Semantic similarity between n-grams is measured as Weighted Overlap of sorted PPR vectors, yielding a score in [0,1]. The final GRouge-N combines lexical match counts and semantic similarity in a linear blend:

$$Sim_{LS}(\text{gram}_n, P) = \beta \cdot \text{Count\_match}(\text{gram}_n, P) + (1 - \beta) \cdot Sim_{sem}(\text{gram}_n, P)$$

$$GRouge\text{-}N = \frac{\sum_{S \in \text{ModelSummaries}} \sum_{\text{gram}_n \in S} Sim_{LS} (\text{gram}_n, P)}{\sum_{S \in \text{ModelSummaries}} \sum_{\text{gram}_n \in S} Count(\text{gram}_n)}$$

This structure allows GRouge-2 and GRouge-su4 to recognize semantic equivalence in paraphrased bigrams or skip-bigrams. Empirical tests show significantly improved correlation with human judgments, especially for abstractive summaries where lexical variability is prevalent.

3. ROUGE 2.0: Topic and Synonym-aware Variants

ROUGE 2.0 proposes several variants that address semantic equivalence and content coverage by integrating topic modeling and synonym expansion (Ganesan, 2018). These include:

• ROUGE-N+Synonyms: Counts n-gram matches where tokens are either identical or synonymous per external resources (e.g., WordNet).
• ROUGE-Topic: Measures overlap of key content words (topics), typically nouns/verbs selected via POS-tagging.
• ROUGE-Topic+Synonyms: Allows synonym matches for topics.
• ROUGE-TopicUniq: Credits each unique topic only once, penalizing redundancy.
• ROUGE-TopicUniq+Synonyms: Applies uniqueness constraint and synonym enhancement.

Formally, for topic-based recall:

$$\text{ROUGE-Topic Recall} = \frac{|T_{ref} \cap T_{cand}|}{|T_{ref}|}$$

In synonym variants, the intersection is extended over token–synonym sets:

$$|T_{ref}^{syn} \cap T_{cand}^{syn}| / |T_{ref}^{syn}|$$

Evaluation shows improved correlation with human assessment, particularly for summaries with lexical and thematic diversity. The uniqueness variants produce metrics that better reflect conciseness and informativeness, aligning with human preferences.

4. Differentiable N-gram ROUGE Objectives

" Differentiable N-gram objectives" [Editor's term] have been proposed to bridge the gap between training and evaluation by formulating ROUGE-like metrics that are compatible with gradient-based optimization (Zhu et al., 2022). These objectives are defined at the probabilistic level, maximizing the cumulative product of token probabilities for matched n-grams, and avoid clipping counts to reference occurrences.

For position-sensitive matching:

$$L_{n\text{-}gram\_rewards}(\theta) = 1 - \frac{ \sum_{t=0}^{T-n} Reward(\cdot) } {T-n+1}$$

where

$$Reward(\cdot) = \begin{cases} \frac{ \prod_{i=1}^n p(y_{t+i}=g_i \mid X) }{ \text{Count}_{match} } & \text{if match} \ 0 & \text{otherwise} \end{cases}$$

Position-agnostic variants similarly reward matches regardless of sequence alignment. Empirical results on CNN/DM and XSum demonstrate substantial increases in ROUGE-L, ROUGE-1, and ROUGE-2, verifying the efficacy of this approach for higher-order objectives (N=2,3,4).

5. Reference Set Sensitivity and Robustness in ROUGE

The sensitivity of traditional ROUGE variants to the choice and diversity of reference summaries has been systematically evaluated (Casola et al., 17 Jun 2025). Wide fluctuations in instance-level ROUGE-L scores occur when varying the reference set, with averages and standard deviations (e.g., 28.5 ± 5 on SummEval). This instability undermines reliability, as model rankings can change based solely on reference selection. ROUGE_avg (averaging over reference scores) yields greater stability than ROUGE_max (maximum across references), but both require multiple references to achieve reliability seen in semantic metrics.

When human judgments are available, correlation with ROUGE degrades for genre-diverse, high-quality outputs, highlighting that simple n-gram overlap inadequately captures true summary quality. It is recommended that higher-order ROUGE variants aggregate over more references and incorporate semantic or syntactic structures to increase robustness against reference set variation.

Dataset	ROUGE-L Avg ± SD
SummEval	28.5 ± 5
GUMSum	27.5 ± 3
DUC2004	24.9 ± 5.3

A plausible implication is that future higher-order ROUGE formulations should include aggregation strategies, uncertainty estimates, and semantic component integration to maintain fair system-level comparisons.

6. Limitations and Recommendations for Higher-order ROUGE

Higher-order ROUGE variants mitigate some deficiencies of the base metric, yet several challenges persist:

• Exact lexical matching restricts applicability in domains with high paraphrasing and terminology variation (Cohan et al., 2016).
• Synonym expansion is fundamentally limited by the completeness of external synonym resources (Ganesan, 2018).
• Topic selection and uniqueness constraints depend on domain-specific definitions and POS-tagging accuracy.
• Reference set variability induces model ranking instability unless sufficient references or semantic aggregation is employed (Casola et al., 17 Jun 2025).
• Integrating semantic similarity via context embeddings (e.g., transformer-based metrics) presents a promising avenue, as direct fine-tuning on semantic similarity tasks greatly increases correlation with human judgment (Kané et al., 2019).

Extensions such as GRouge, ROUGE 2.0, and differentiable objectives exemplify the ongoing evolution of ROUGE from surface overlap toward semantically and contextually robust evaluation. Future work may benefit from integrating dynamic, data-driven similarity measures, cross-linguistic adaptations, uncertainty quantification, and ensemble meta-evaluation frameworks encompassing both higher-order n-gram and semantic matching dimensions.

7. Summary and Current Perspectives

Higher-order ROUGE variants span a range of methodologies, including extended n-gram matching, lexico-semantic fusion, topic-centric aggregation, synonym expansion, and differentiable objectives. Each addresses specific limitations of the original ROUGE design, striving for improved metric reliability, robustness to human variation, and enhanced alignment with manual evaluation. Precise selection and configuration of ROUGE variant and aggregation strategy should be context-dependent, considering the domain, task, and available reference summary diversity. The continued integration of semantic, syntactic, and probabilistic elements in ROUGE-like metrics marks a trend toward multi-faceted, human-aligned evaluation of automatic summarization.