Percentile-Based Evaluation Protocol

• Percentile-based evaluation protocols are defined as methods that rank items by their position within a reference distribution rather than by raw magnitude.
• These protocols normalize citation and performance data by comparing items to subject-specific, time-dependent reference sets, reducing distortions from extreme values.
• They address challenges such as tied scores and discrete distributions using rules and fractional scoring, ensuring fair aggregation and benchmark alignment.

Percentile-based evaluation protocol denotes a family of assessment procedures in which an item is valued by its position within a reference distribution rather than by its raw magnitude. In bibliometrics, the canonical case is citation impact: a publication is compared with papers from the same subject area, publication year, and document type, and is then represented by a percentile, a percentile rank class, or a top-$x\%$ indicator. The protocol emerged as a non-parametric alternative to mean-based normalization for highly skewed citation distributions, and it has subsequently been adapted to model assessment, percentile-thresholding in machine learning, matched-reference evaluation of speech behavior, and other settings in which relative standing is more informative than an unnormalized score (Bornmann, 2012, Bornmann et al., 2012, Bérubé et al., 2019).

1. Conceptual basis

The central methodological premise is that the observed value of an item should be interpreted against an appropriate comparison set. In citation analysis, percentiles are derived from the citation distribution of a reference set and therefore express relative citation impact rather than absolute citation counts. This is why percentile protocols are repeatedly presented as preferable to arithmetic means when the underlying distribution is highly skewed: a few very highly cited papers can dominate mean-based indicators, whereas percentile procedures are rank-based and therefore less sensitive to extreme values (Bornmann et al., 2012, Bornmann et al., 2013).

Two representational conventions recur. In one convention, used in InCites-style settings, lower percentile values indicate higher citation impact and the maximum percentile value corresponds to uncited papers. In another convention, the protocol is phrased directly in terms of top classes, such as the proportion of papers in the top $10\%$ or top $1\%$ most frequently cited set. The latter convention underlies indicators such as PP(top $x\%$), which define evaluation in terms of membership in a top citation class rather than a scalar percentile value (Bornmann, 2012, Bornmann et al., 2016).

The protocol also supports aggregation into discrete percentile rank classes. Six-class schemes are especially prominent. One widely used PR(6) formulation distinguishes bottom $50\%$, $50\%$–$75\%$, $75\%$–$90\%$, $90\%$–$10\%$0, $10\%$1–$10\%$2, and top $10\%$3, while another presentation of the same general idea enumerates top-$10\%$4, top-$10\%$5, top-$10\%$6, top-$10\%$7, top-$10\%$8, and bottom-$10\%$9. This class-based formulation makes percentile protocols compatible with integrated indicators such as $1\%$0 and with binary “excellence indicators,” where only two classes are distinguished, for example top-$1\%$1 versus the remainder (Schreiber, 2013, Leydesdorff et al., 2011).

2. Reference sets, normalization, and expected benchmarks

A percentile protocol is only as well defined as its reference set. In bibliometric applications, the standard design is to compare each publication with papers from the same field, publication year, and document type. This field- and year-normalized construction is the reason percentiles are used for cross-field comparison: the protocol does not compare raw counts across incomparable citation cultures, but relative standing within matched distributions (Bornmann, 2012, Bornmann et al., 2013).

The best-known benchmark is PP(top $1\%$2), defined as the proportion of papers in a unit that belong to the top $1\%$3 most frequently cited papers in the relevant fields and publication years. Under the idealized expectation, if papers are randomly sampled from the database, then the share in the top-$1\%$4 class should equal $1\%$5. The same logic yields expected PP(top $1\%$6, PP(top $1\%$7, and PP(top $1\%$8. Empirical work, however, shows that database-specific population values can deviate slightly from these nominal benchmarks because of normalization and classification choices. In an in-house Max Planck Digital Library database derived from SCI-E, SSCI, and AHCI, covering 1980–2010 and $1\%$9 papers, the observed population values were PP(top $x\%$0, PP(top $x\%$1, and PP(top $x\%$2 (Bornmann et al., 2016).

That departure from exact nominal values is not accidental. In the same study, the database did not fractionally assign impact to subject categories; if a paper was assigned to multiple subject categories, an average citation impact was used instead. A related controversy appears in other percentile systems: some implementations assign multi-category papers their best percentile across categories, while alternative proposals recommend a weighted percentile rank using subject-category sizes. This suggests that reference-set design is not merely a bookkeeping detail but part of the evaluation protocol itself (Bornmann et al., 2016, Bornmann et al., 2020).

Reference-set dependence also has direct strategic consequences. When percentile normalization is based on journal subject categories, the same citation count can map to very different percentile positions in different categories. Empirical examples show that a paper can obtain a markedly better percentile standing in one subject category than in another, even when its citation count is held fixed. A plausible implication is that journal choice can materially affect percentile-based evaluation because the protocol normalizes against the chosen reference distribution, not against an abstract disciplinary constant (Schreiber, 2014).

3. Ties, discreteness, and fractional scoring

The main technical difficulty in percentile-based evaluation is the discrete nature of citation distributions combined with the prevalence of ties. Percentile thresholds often fall inside a block of papers with identical citation counts, creating ambiguity in class assignment. A simple field with $x\%$3 publications—$x\%$4 with $x\%$5 citations, $x\%$6 with $x\%$7 citations, and $x\%$8 with $x\%$9 citations—illustrates the problem for PPtop $50\%$0. If the tied $50\%$1-citation papers are excluded, the result is $50\%$2; if they are included, the result is $50\%$3. Neither outcome reproduces the intended $50\%$4 benchmark (Waltman et al., 2012).

Several tie rules have been proposed. The “lower citation rates” rule assigns tied papers to the lower class and tends to underpopulate higher classes. The “lower or equal citation rates” rule assigns them to the higher class and tends to overpopulate higher classes. Uncertainty-interval rules, average percentiles, and averaged weights moderate the problem but do not eliminate it completely. In a six-class protocol with theoretical $50\%$5, empirical comparisons across four datasets showed totals below the benchmark under lower-citation assignment, above the benchmark under lower-or-equal assignment, close but not exact under average-percentile and averaged-weight schemes, and exactly $50\%$6 only under fractional scoring (Schreiber, 2013).

Fractional scoring treats a tied citation block as spanning an interval of the percentile distribution and splits that interval proportionally across adjacent percentile classes. In the $50\%$7-paper example, the $50\%$8 papers with $50\%$9 citations already cover $50\%$0 of the distribution, so the remaining $50\%$1 of the top-$50\%$2 region must be drawn fractionally from the $50\%$3 papers with $50\%$4 citations, each of which therefore receives a top-$50\%$5 weight of $50\%$6. The resulting indicator is exactly $50\%$7 (Waltman et al., 2012).

A formal version of this idea defines percentile intervals $50\%$8, class scores $50\%$9, citation-count blocks $75\%$0, and an overlap function

$75\%$1

The score of papers with $75\%$2 citations is then the overlap-weighted average of the class scores, and the percentile-based indicator for a unit is the average of those publication scores. This construction has a strong fairness property: when applied to all publications in a field, the resulting field-wide value depends only on the percentile design and not on the field’s citation distribution. For PPtop $75\%$3, the whole field therefore has exactly $75\%$4 top-$75\%$5 papers under the protocol (Waltman et al., 2012).

The tie problem is not limited to bibliometrics. A related argument in quantile-based assessment distinguishes the linear problem of assigning fractional percentile ranks from the nonlinear problem of aggregating those ranks into coarse classes such as PR6. The proposed resolution is to perform fractional attribution at the level of hundred percentiles or quantiles first, and only then aggregate. This separation is motivated as both statistically cleaner and computationally lighter (Leydesdorff, 2012).

4. Aggregation, benchmarking, and statistical inference

Percentile-based evaluation rarely ends with paper-level ranks. The protocol usually aggregates individual positions into class frequencies or weighted totals. PR(2) schemes reduce the problem to a binary high-impact classification, typically top $75\%$6 versus all others. PR(6) schemes describe the full citation structure more finely. In one university-comparison design, expected PR(6) proportions under random selection were $75\%$7, $75\%$8, $75\%$9, $75\%$0, $75\%$1, and $75\%$2 for the successive classes from top $75\%$3 to below median. Observed deviations from those proportions were used to identify whether a unit had fewer low-impact papers or more highly cited papers than expected (Bornmann, 2012).

Integrated indicators aggregate percentile classes by weighting them. One widely used formulation is

$75\%$4

where $75\%$5 denotes a percentile rank class and $75\%$6 the number of publications in that class. In the six-class framework, the normalized score is

$75\%$7

and the theoretical value under the ideal PR(6) class proportions is $75\%$8. Excellence indicators such as top-$75\%$9 can be interpreted as special binary cases of the same general architecture (Bornmann, 2012, Schreiber, 2013, Leydesdorff et al., 2011).

Because percentile distributions are generally non-normal, non-parametric and categorical inferential tools are standard. For distributions of continuous percentile values across units, the Kruskal–Wallis $90\%$0-test and Bonferroni-corrected pairwise comparisons have been used. For contingency tables of percentile rank classes, chi-square testing and chi-square decomposition identify the classes that contribute most strongly to group differences. For binary excellence indicators and impact indicators, the $90\%$1-test for independent proportions has been explicitly recommended. Percentile or quartile values can also be compared with expected benchmark values using the Wilcoxon signed-rank test (Bornmann, 2012, Leydesdorff et al., 2011, Leydesdorff, 2012).

An important methodological caution follows from these benchmarked comparisons. Random-sampling experiments on PP(top $90\%$2), PP(top $90\%$3), and PP(top $90\%$4) show that the mean over many random samples is very close to the population value, but single samples can deviate substantially, especially for small sample sizes and especially for PP(top $90\%$5). In the reported example of $90\%$6 random samples of size $90\%$7, mean PP(top $90\%$8 was close to the population value $90\%$9, yet the minimum was $90\%$0 and the maximum $90\%$1. The protocol therefore supports expectation only as an average property over repeated random sampling, not as a deterministic target for every empirical unit (Bornmann et al., 2016).

5. Alternative constructions and interpretive controversies

Percentile-based evaluation has generated a large methodological literature because the percentile itself is not uniquely defined. Plotting-position formulas such as $90\%$2, $90\%$3, and Hazen’s $90\%$4 differ in endpoint behavior and in whether the median falls exactly at $90\%$5. Hazen’s rule has often been preferred because it gives the median paper a percentile of $90\%$6 and treats the two tails symmetrically in the cited examples, while Blom’s and Gringorten’s formulas embody alternative assumptions about the underlying distribution (Bornmann et al., 2012).

A more radical alternative is P100, which ranks unique citation values rather than all papers and assigns

$90\%$7

so that the lowest unique citation count receives $90\%$8 and the highest $90\%$9. P100 was intended to provide fixed endpoints and unambiguous handling of equal citation counts, but empirical analysis found counterintuitive behavior. Small changes in one paper’s citation count can create or remove a unique citation level and thereby shift many other papers’ P100 values. Comparisons across fields or publication years can also become problematic because P100 is sensitive to the structure of unique citation counts rather than to the full frequency distribution (Schreiber, 2014).

P100′ incorporated citation frequencies and was presented as a refinement, but subsequent analysis concluded that it is “not much different from standard percentile-based ratings” in ordinary cases and usually lies within the standard uncertainty interval bounded by inverted InCites-style and Rousseau-style percentiles. A further interpolation, P100″, was proposed to remain within that interval while avoiding specific top-tie anomalies. The controversy illustrates a broader point: percentile protocols can differ substantially in formal definition while remaining operationally close, or they can appear elegant but behave poorly under realistic tie structures (Schreiber, 2014).

Another line of development rejects plotting positions in favor of cumulative frequencies in percentages. CP-IN assigns to a citation count the cumulative percentage including the papers in the current row; CP-EX excludes the current row and thus represents the percentage strictly below that citation count. These approaches are presented as more directly interpretable because they map cleanly to “percentage of papers at or below” or “percentage of papers below” a citation threshold, use the complete citation distribution, and support estimation of citation thresholds for predefined percentiles. They also provide weighted aggregation formulas for papers assigned to multiple subject categories or multiple units (Bornmann et al., 2020).

Empirical comparisons among Hazen, InCites, SCImago, CWTS, and P100 found that the methods differ not only in percentile semantics but also in predictive behavior. InCites was reported to overestimate citation impact because it uses the highest percentile rank when a paper is assigned to more than one subject category; SCImago showed higher power in predicting long-term citation impact in early years; P100, despite analytic appeal, showed a disadvantage in predictive ability. This suggests that percentile-based protocols should be evaluated not only for formal consistency but also for empirical stability under multi-category classification, early citation windows, and downstream ranking tasks (Bornmann et al., 2013).

6. Generalizations beyond citation analysis

Although the modern vocabulary of percentile-based evaluation was shaped in bibliometrics, the protocol now appears in several other technical domains. In model assessment, percentile-based residuals replace standardized residuals that rely only on predictive means and standard deviations. For a continuous predictive distribution $10\%$00, the residual is

$10\%$01

and for discrete distributions a one-half continuity correction is used. Under the null $10\%$02, these residuals are exactly standard normal, which gives better-calibrated outlier detection and diagnostic plots in hierarchical, discrete, skewed, or otherwise non-Gaussian settings (Bérubé et al., 2019).

In research assessment, percentile-based double rank analysis maps local publication counts to world citation percentiles and finds that the resulting distribution is well approximated by a power law,

$10\%$03

Because the fit is built from all publications rather than only the high-citation tail, it can be used to estimate the likelihood of producing very highly cited papers at percentiles such as top $10\%$04, top $10\%$05, and top $10\%$06. This challenges the misconception that a single top-$10\%$07 or top-$10\%$08 cutoff exhausts percentile-based evaluation; the full percentile curve can encode materially different breakthrough-producing capacities (Brito et al., 2017).

Recent work in speech-to-speech evaluation defines a matched-reference percentile protocol for behavioral plausibility in conversational prosody and rhythm. Using more than $10\%$09 hours of English dyadic conversation from the Seamless Interaction dataset, the protocol constructs reference regimes for $10\%$10 mean, $10\%$11 expressivity, speech rate, articulation rate, pause ratio, and mean pause duration. A system waveform is evaluated against the closest matched human stratum, and outputs are reported as percentile deviations or $10\%$12th–$10\%$13th percentile out-of-regime flags. On held-out human rows, pooled references over-flagged state-conditioned $10\%$14 expressivity and rhythm, whereas matched references returned flag rates closer to the nominal $10\%$15 and made the direction of deviation interpretable (Hallur et al., 30 Jun 2026).

Percentile logic has also entered industrial machine learning. PEARL defines a user-relative engagement percentile as $10\%$16 and shows that pairwise comparison against a sample from the user’s own historical distribution yields an unbiased estimator of that percentile. PercentMatch, in multi-label semi-supervised learning, uses percentile thresholds $10\%$17 and $10\%$18 to derive class-specific score thresholds for positive and negative pseudo-labels from an EMA histogram of unlabeled predictions. In both cases, percentiles are used not as a post hoc descriptive statistic but as the supervisory object or thresholding mechanism itself (Gella et al., 20 May 2026, Huang et al., 2022).

Across these domains, the recurrent methodological lesson is consistent. A percentile-based evaluation protocol is not merely a way of rescaling numbers to the unit interval. It is a reference-dependent procedure that requires explicit decisions about comparison sets, tie handling, orientation, aggregation, and uncertainty. When those choices are specified precisely, percentile protocols provide robust rank-based evaluation under skewed, heterogeneous, or context-sensitive distributions. When they are left implicit, results from different studies or systems may not be directly comparable (Bornmann et al., 2012, Bornmann et al., 2016).