CupRank: Transparent Scholarly Ranking

• CupRank is a ranking method that uses an experimentally weighted linear combination of citation and network signals.
• It assigns paper importance based on publication metrics, author quality, venue impact, and network analysis like PageRank.
• The approach emphasizes simplicity and transparency in scholarly evaluation while acknowledging limitations in human judgment and data biases.

Searching arXiv for papers and variants of “CupRank” to ground the article. CupRank is most directly associated with the KMi and Mendeley submission of team BletchleyPark to the WSDM Cup 2016 scholarly publication ranking challenge, where it denotes a simple, transparent method for assigning query-independent importance scores to papers in the Microsoft Academic Graph (MAG) (Herrmannova et al., 2016). In that usage, CupRank is not presented as a separately named algorithm inside the paper so much as a ranking method built from an experimentally weighted linear combination of publication-, author-, venue-, institution-, age-, and citation-network signals. The name was later reused in a different domain for the learned ranking module inside LLMCup, a framework for automatic code-comment updating, so the term is context-dependent rather than universally canonical (Ge et al., 11 Jul 2025).

1. Origin and task setting

The 2016 CupRank setting was the WSDM Cup challenge on scholarly publication ranking. The task was to estimate the static, query-independent importance of papers in MAG and compare system outputs against human judgments prepared as pairwise rankings of a subset of papers. The KMi and Mendeley team emphasized from the outset that their objective was not to build a highly complex learned model, but rather a method that was transparent, understandable, and easy to interpret (Herrmannova et al., 2016).

The problem formulation treated publication importance as a function of both intrinsic impact and the status of linked scholarly entities. In practical terms, this meant scoring several entity types in the scholarly graph separately—publications, authors, venues, and institutions—and then combining these signals at the publication level. A closely related description from another WSDM Cup 2016 submission states that the contest required assigning each paper a score in $[0,1]$ and evaluated performance as the fraction of human pairwise comparisons correctly ordered, which clarifies the competition’s ranking-oriented rather than regression-oriented character (Wesley-Smith et al., 2016).

2. Linear ranking framework

The central CupRank model is a weighted linear mixture:

$$score(p) = 2.5 \cdot s_{pub} + 0.1 \cdot s_{age} + 1.0 \cdot s_{pr} + 1.0 \cdot s_{auth} + 0.1 \cdot s_{venue} + 0.01 \cdot s_{inst}$$

The authors state that, in an ideal supervised setting, such weights might be learned automatically, but because no ground-truth training data were available, the coefficients were chosen experimentally (Herrmannova et al., 2016).

Component	Definition	Weight
$s_{pub}$	Thresholded citations per author	2.5
$s_{age}$	Publication year $y_p$	0.1
$s_{pr}$	PageRank on the citation graph	1.0
$s_{auth}$	Mean of authors’ average citations per publication	1.0
$s_{venue}$	Total citations of other papers in the venue	0.1
$s_{inst}$	Average institution-level citation score	0.01

This design embodies the paper’s stated view that publication importance is evidenced both by the paper’s own impact and by the importance of the entities involved in its creation. The weighting also makes the publication-based citation signal dominant, with author quality and PageRank as secondary factors, and venue, age, and institution features acting as weaker adjustments (Herrmannova et al., 2016).

3. Entity-specific scoring signals

The strongest component is the publication score, defined as a citation count normalized by the number of authors and capped by a maximum citation threshold:

$$s_{pub}(p) = \begin{cases} c(p) / |A_p|, & \text{for } c(p) \leq t \ t / |A_p|, & \text{for } c(p) > t \end{cases}$$

where $$score(p) = 2.5 \cdot s_{pub} + 0.1 \cdot s_{age} + 1.0 \cdot s_{pr} + 1.0 \cdot s_{auth} + 0.1 \cdot s_{venue} + 0.01 \cdot s_{inst}$$0 is the number of citations received by publication $$score(p) = 2.5 \cdot s_{pub} + 0.1 \cdot s_{age} + 1.0 \cdot s_{pr} + 1.0 \cdot s_{auth} + 0.1 \cdot s_{venue} + 0.01 \cdot s_{inst}$$1, $$score(p) = 2.5 \cdot s_{pub} + 0.1 \cdot s_{age} + 1.0 \cdot s_{pr} + 1.0 \cdot s_{auth} + 0.1 \cdot s_{venue} + 0.01 \cdot s_{inst}$$2 is the set of its authors, and the threshold was set experimentally to $$score(p) = 2.5 \cdot s_{pub} + 0.1 \cdot s_{age} + 1.0 \cdot s_{pr} + 1.0 \cdot s_{auth} + 0.1 \cdot s_{venue} + 0.01 \cdot s_{inst}$$3. The paper notes that this thresholded form was introduced in the second round of the competition; in the first round, citations above the threshold were effectively discarded. The authors also explored raw citation counts, time-decayed citations, decay on total citation counts, and normalizations by age or number of authors, but report that the thresholded citations-per-author variant performed best (Herrmannova et al., 2016).

A time-decay alternative was tested using

$$score(p) = 2.5 \cdot s_{pub} + 0.1 \cdot s_{age} + 1.0 \cdot s_{pr} + 1.0 \cdot s_{auth} + 0.1 \cdot s_{venue} + 0.01 \cdot s_{inst}$$4

where $$score(p) = 2.5 \cdot s_{pub} + 0.1 \cdot s_{age} + 1.0 \cdot s_{pr} + 1.0 \cdot s_{auth} + 0.1 \cdot s_{venue} + 0.01 \cdot s_{inst}$$5 is the current year and $$score(p) = 2.5 \cdot s_{pub} + 0.1 \cdot s_{age} + 1.0 \cdot s_{pr} + 1.0 \cdot s_{auth} + 0.1 \cdot s_{venue} + 0.01 \cdot s_{inst}$$6 is the publication year of the citing paper. This formulation was intended to weight recent influence more strongly. The authors also tried logarithmic and linear decay over total citations, but found that a citation cap worked better than these more elaborate alternatives (Herrmannova et al., 2016).

The age feature is deliberately simple:

$$score(p) = 2.5 \cdot s_{pub} + 0.1 \cdot s_{age} + 1.0 \cdot s_{pr} + 1.0 \cdot s_{auth} + 0.1 \cdot s_{venue} + 0.01 \cdot s_{inst}$$7

with $$score(p) = 2.5 \cdot s_{pub} + 0.1 \cdot s_{age} + 1.0 \cdot s_{pr} + 1.0 \cdot s_{auth} + 0.1 \cdot s_{venue} + 0.01 \cdot s_{inst}$$8 the publication year. This makes the component a linear recency proxy: more papers receive higher $$score(p) = 2.5 \cdot s_{pub} + 0.1 \cdot s_{age} + 1.0 \cdot s_{pr} + 1.0 \cdot s_{auth} + 0.1 \cdot s_{venue} + 0.01 \cdot s_{inst}$$9, and importance under this term decreases linearly with time (Herrmannova et al., 2016).

For citation-network structure, CupRank added a PageRank feature in the second phase:

$s_{pub}$0

To compute PageRank efficiently, the authors inserted a dummy paper into the citation network that cites and is cited by all publications except itself, allowing weight to be redistributed across the graph and stabilizing the computation. They found PageRank to perform similarly to total citation counts, but retained it as an additional signal (Herrmannova et al., 2016).

The author-based score was derived by computing, for each author of a paper, that author’s average citations per publication, and then averaging those values across the authors of the paper. The venue score was defined as the total citation count of all other papers in the same venue:

$s_{pub}$1

where $s_{pub}$2 is the set of papers published in venue $s_{pub}$3. The institution score averaged citation impact across the unique institutions affiliated with the paper’s authors:

$s_{pub}$4

where $s_{pub}$5 is the set of unique institutions linked to the paper and $s_{pub}$6 is the set of publications by authors affiliated with institution $s_{pub}$7 (Herrmannova et al., 2016).

4. Evaluation and comparative WSDM Cup context

The WSDM Cup evaluation details were only partially disclosed. According to the organizers, systems were scored by percentage agreement with human evaluation data, and those human labels consisted of pairwise rankings of a subset of MAG papers prepared by computer science experts. The data were split into validation and test sets, and participants could repeatedly submit runs through an online evaluation tool. The BletchleyPark team reports submitting more than 270 runs during the training phase. Their method achieved the highest score on the test data while ranking fifth on the validation data; their final second-round submission used the weighted combination described above (Herrmannova et al., 2016).

CupRank emerged within a competitive field of strong citation-centric baselines. The ALEF system, described in "Static Ranking of Scholarly Papers using Article-Level Eigenfactor" (Wesley-Smith et al., 2016), used a citation-network-only eigenvector-centrality method adapted to the directed acyclic structure of citation graphs and reported a final test score of $s_{pub}$8, earning second place. The UFMG/LATIN S-RCR system, described in "Simplified Relative Citation Ratio for Static Paper Ranking: UFMG/LATIN at WSDM Cup 2016" (Ribas et al., 2016), used a single feature based on field-normalized citation behavior over co-citation neighborhoods; it reported $s_{pub}$9 on the public leaderboard and $s_{age}$0 on the private test set, placing third in Phase 1. This constellation of results suggests that the competition strongly rewarded compact methods built around citation-derived evidence, even when those methods differed substantially in formalism.

5. Strengths, interpretation, and criticism

The main stated strength of CupRank is simplicity. The authors explicitly preferred straightforward, interpretable methods over opaque learning-to-rank systems and presented transparency as a substantive virtue in the context of research evaluation (Herrmannova et al., 2016).

Their experiments also exposed a limitation with broader methodological implications: in the challenge setup, simple citation counts outperformed more sophisticated alternatives such as time-decayed citations, PageRank, and $s_{age}$1-index-based author scoring. The authors treat this result cautiously. They do not interpret it as proof that citation counts are universally superior for scholarly assessment. Instead, they argue that the evaluation labels may themselves have been biased toward citation-like signals, or that the task design and metric were too opaque to support more nuanced ranking approaches (Herrmannova et al., 2016).

Several further limitations are listed explicitly. The evaluation dataset and metric were not disclosed in sufficient detail, making it difficult to avoid overfitting. Human judgments are inherently subjective, which complicates their role as a gold standard. Citation counts also ignore differences in citation meaning, topic popularity, skewed citation distributions, and citation lag. The authors further state that they were unsure why the maximum citation threshold improved performance, and they recommend revisiting that design choice (Herrmannova et al., 2016).

6. Full-text evaluation and semantometrics

A major future direction proposed in the CupRank paper is full-text-based evaluation, linked explicitly to semantometrics. The authors contrast semantometrics with bibliometrics, altmetrics, and webometrics: whereas those families rely on external signals from the scholarly communication ecosystem, semantometrics use the manuscript itself to assess value. They report that a pilot study had already shown the feasibility and utility of such methods, and they identify full-text access as the main practical barrier (Herrmannova et al., 2016).

The access problem is described concretely. Full texts were difficult to obtain because of copyright restrictions and paywalls, and the authors investigated sources including Elsevier, Springer, CrossRef, and Mendeley APIs but found coverage and access too limited for their purposes. They therefore propose future work that would enrich MAG with altmetric, webometric, and semantometric data, incorporate additional network measures beyond PageRank, and possibly support a future competition in which full-text and other external signals are available. They also call for a large, open, unbiased dataset of human judgments and a more transparent evaluation framework (Herrmannova et al., 2016).

7. Later reuse of the name

In later literature, "CupRank" was reused in a different technical setting: LLMCup, a framework for automatic comment updating in software repositories. There, CupRank is a learned neural ranking model rather than a bibliometric publication scorer. LLMCup follows an update-then-rank paradigm in which an LLM generates multiple candidate updated comments under different prompt strategies—$s_{age}$2-shot, $s_{age}$3-shot, $s_{age}$4-shot, and $s_{age}$5-shot—and CupRank selects the best candidate (Ge et al., 11 Jul 2025).

This later CupRank is a dual-encoder model operating on a code-change sequence and a comment-change sequence. Each token is represented as an edit token $s_{age}$6, where $s_{age}$7 denotes edit operation type and $s_{age}$8 marks token origin. The model uses CodeBERT token embeddings, bidirectional cross-attention between code and comment changes, a 2-layer Transformer Encoder, max pooling, a projection layer, and cosine similarity as the ranking score. Training uses an augmented ranking dataset with the ground-truth updated comment as a positive and LLM-generated alternatives as negatives, optimized by a listwise loss with temperature parameter $s_{age}$9 (Ge et al., 11 Jul 2025).

This later usage is conceptually distinct from the WSDM Cup 2016 publication-ranking method. A plausible implication is that "CupRank" now functions less as the name of one fixed algorithm than as a competition-derived label that has been repurposed for ranking components in different domains.