Meaning in Order, Order in Meaning: Semantic R-precision for Keyphrase Evaluation

Abstract: Evaluating the quality of automatically generated keyphrases remains a complex challenge. Traditional metrics either rely on exact lexical matching or consider semantic similarity while ignoring prediction ranking, both of which misalign with how humans judge informativeness and relevance. We introduce Semantic R-Precision (SemR-p), a novel evaluation metric that integrates semantic similarity into the rank-aware R-Precision framework. Designed from a human-centric perspective and inspired by Information Retrieval metrics, SemR-p rewards semantically relevant keyphrases that appear early in the output list. We conducted extensive analyses to assess its semantic sensitivity, ranking awareness, and discriminative power across models and datasets. The results suggest that SemR-p offers a complementary lens for evaluating keyphrase predictions, helping to better reflect user-centred notions of relevance alongside traditional lexical and semantic matching metrics.

• The paper introduces SemR-p, a new metric that integrates semantic similarity and ranking to better evaluate keyphrase extraction models.
• It leverages mean-pooled neural embeddings and a hierarchical scoring protocol to align system predictions with user-focused retrieval.
• Empirical analysis across diverse datasets demonstrates SemR-p's ability to differentiate model performance and improve practical keyphrase generation.

Semantic R-Precision for Keyphrase Evaluation: Integrating Order and Meaning

Motivation and Context

Evaluation of automatic keyphrase extraction and generation remains a deeply challenging problem due to the discrepancy between surface-level lexical overlap and the true semantic alignment perceived by end users. Historically, widely adopted metrics such as F1, R-Precision, and their approximate-matching variants emphasize exact word-level matches between system predictions and human keyphrase annotations. Although semantically robust alternatives like BERTScore and embedding-based semantic F1 metrics have emerged, these approaches lack explicit sensitivity to the order in which keyphrases are presented—a crucial factor, since users profoundly prioritize results ranked at the top due to cognitive factors such as the primacy effect and attentional biases. The paper "Meaning in Order, Order in Meaning: Semantic R-precision for Keyphrase Evaluation" (2606.07057) introduces Semantic R-Precision (SemR-p), a metric explicitly engineered to combine semantic similarity assessment with a rank-aware framework for evaluating keyphrase generation.

Metric Definition and Methodological Details

SemR-p builds upon the R-Precision paradigm from Information Retrieval, which evaluates only the top-$R$ prediction slots (with $R$ denoting the number of ground-truth references for the document) and models realistic user attention distribution. The distinctive innovation of SemR-p is injecting phrase-level semantic similarity, derived from mean-pooled neural embeddings, directly into the ranking evaluation pipeline.

For each predicted keyphrase among the top $R$, SemR-p follows a hierarchical scoring protocol:

1. If a prediction exactly matches a reference phrase (after stemming), it is credited fully.
2. Otherwise, it receives a semantic score: the average embedding cosine similarity with its top-$k$ closest reference phrases.
3. The document-level SemR-p is simply the mean of these scores over the selected $R$ predictions.

The tunable parameter $k$ operationalizes the width of the semantic neighborhood considered and thus governs the metric's bias between "best possible" hit and "broad semantic coverage."

Experimental Design

A comprehensive suite of analyses probes SemR-p’s capabilities along three axes: semantic sensitivity, rank-awareness, and discriminative utility. Two diverse datasets are considered: kp20k (scientific abstracts) and kptimes (news articles), with output from eight representative unsupervised, supervised, and LLM-based keyphrase models. Baselines include F1@M, rank-based classical metrics (NDCG, AP), approximate-matching R-Precision, and strong embedding-based metrics (Semantic F1, SemP, SemR, BERTScore variants) for a robust comparison.

Empirical Results

Effect of the Semantic Neighborhood ($k$)

Ablation experiments show SemR-p’s absolute scores drop as $k$ increases, reflecting strictness when aggregating over more reference phrases. However, the metric's core semantic discriminative power remains stable, with Spearman $\rho$ above $0.6$ versus core semantic baselines regardless of $R$0, and only marginal practical effect sizes. Notably, at larger $R$1, SemR-p aligns more with exact and rank-focused baselines, likely due to a broader semantic focus.

Figure 1: Score distributions of SemR-p across datasets as $R$2 is changed; larger $R$3 introduces stricter, more holistic assessment.

Figure 2: Cliff's Delta effect size for changes in Spearman correlation between SemR-p and baseline metrics as $R$4 increases from 1 to 3, demonstrating increased alignment with set/rank-based metrics.

On the basis of this analysis, $R$5 is recommended as a robust, balanced default.

Model and Domain Discrimination

A two-way ANOVA on over 48,000 scores shows the model identity explains $R$6 of SemR-p variance—indicating strong sensitivity to actual keyphrase system performance. Dataset/domain and interaction effects, though statistically significant, account for substantially less variance, reinforcing SemR-p’s fidelity for benchmarking model quality.

Agreement with System-Level Rankings

SemR-p's model-level rankings demonstrate consistent positive correlation (Spearman $R$7) across all baselines, with strongest alignment observed with set-quality and rank-aware metrics. Interestingly, correlations with BERTScore variants are notably weaker, possibly reflecting methodological differences (token-level vs. phrase-level semantics, aggregate procedure, lack of ranking).

Figure 3: Spearman rank correlation between SemR-p ($R$8) and other metrics when used to rank model performance at system-level across datasets.

Latent Structure via Factor Analysis

Exploratory factor analysis on the 11-metric space yields two orthogonal factors comprising $R$9 of the variance. Factor 1 aggregates both classic and recent semantic metrics (including SemR-p), representing General Effectiveness/Ranking; Factor 2 is dominated by BERTScore, isolating deep contextual semantic similarity. SemR-p uniquely loads substantially on both, reflecting its dual emphasis on ranking and semantic alignment.

Figure 4: Heatmap of evaluation metric loadings by two latent factors from exploratory factor analysis.

Qualitative Divergence Analysis

Multiple case studies expose instances where SemR-p diverges—often usefully—from baselines:

• It credits conceptually similar predictions missed by exact matchers;
• It penalizes correct predictions if they are poorly ranked, unlike rank-agnostic metrics;
• It distinguishes between precise semantic overlap and loose substring matches, outperforming approximate metrics in penalizing generic predictions.

Theoretical and Practical Implications

SemR-p’s unification of phrase-level semantic sensitivity and strict positional evaluation introduces a more user-centric paradigm in keyphrase evaluation, better reflecting actual informativeness as perceived in downstream retrieval or summarization workflows. The strong discriminative power across model families and demonstration of alignment with both tradition and advanced semantic metrics position SemR-p as an indispensable complement to existing benchmarks. Importantly, its sensitivity to the semantic neighborhood parameter implies the potential for tuning evaluation focus depending on the domain, task, or user requirements.

Practically, adoption of SemR-p can incentivize models to optimize not only for generating semantically accurate keyphrases but also for proper ranking. Cases penalized by SemR-p but not by existing metrics commonly involve misplaced, though accurate, predictions—an error source critical to real-world retrieval or summarization pipelines.

Limitations and Directions for Future Research

While empirically robust and cognitively motivated, SemR-p presents several areas for further investigation:

• Direct calibration against human user judgments to solidify external validity.
• Analysis of dependence on embedding architectures and semantic pooling strategies.
• Extension to diversity-aware matching (e.g., integrating one-to-one matching via maximum bipartite assignment) to address potential redundancy tolerance.
• Exploration of dynamic or domain-adaptive $R$0 tuning strategies, possibly informed by reference set structure.

Conclusion

Semantic R-Precision (SemR-p) (2606.07057) establishes a theoretically coherent and empirically validated bridge between semantic similarity and the crucial notion of prediction ranking in keyphrase evaluation. Its robust performance across modeling paradigms, and its nuanced sensitivity to both phrase meaning and list order, recommend it as both a supplementary score and a potential core evaluation metric for future keyphrase extraction and generation research. Its adoption can directly lead to more user-aligned and trustworthy NLP systems in scientific digital libraries, content recommendation, and knowledge management applications.