Precision at R (P@R): Definition & Evaluation

• Precision at R (P@R) is a metric that quantifies the proportion of relevant items among the top R predictions, with R defined by the number of true relevant items.
• It adapts to variable ground-truth sizes and penalizes ranking errors, thereby aligning evaluation with user-centric relevance in tasks like keyphrase extraction and classification.
• Optimization techniques using quantile-based surrogates and convex losses enable efficient estimation in imbalanced settings, while semantic extensions like SemR-p capture nuanced relevance.

Precision at R (P@R), also known as R-Precision, is a core metric in information retrieval, classification, and keyphrase evaluation, quantifying the proportion of relevant items among the top-R results produced by a model, where R is determined by the number of ground-truth relevant items. The versatility of P@R arises from its direct alignment with user-centric evaluation: it explicitly focuses attention on the most highly ranked predictions, tailoring the cutoff per instance or document and penalizing both ranking errors and spurious matches. This article systematically presents the foundational definition, formal properties, algorithmic framework, optimization techniques, and recent semantic extensions of P@R.

1. Formal Definition and Notation

Let $Y = \{y_1, ..., y_R\}$ denote the set of $R$ reference (i.e., relevant) items for a given instance (e.g., all true keyphrases for a document), and let $\hat Y = (\hat y_1, ..., \hat y_m)$ be an ordered list of $m$ model predictions. The canonical definition of Precision at R is: $$\mathrm{P@}R = \frac{1}{R}\sum_{i=1}^{R} \mathbf{1}\left[\mathrm{match}(\hat y_i, Y)\right]$$ where $\mathbf{1}[\cdot]$ is the indicator function and $\mathrm{match}(\hat y_i, Y)$ evaluates to 1 if $\hat y_i$ matches any $y_j \in Y$ according to the chosen matching criterion (e.g., exact, stemmed, or approximate match) (Venturini et al., 5 Jun 2026).

Alternatively, in classification with real-valued instance scores $\{r_i\}$ and a global acceptance threshold $R$0, define $R$1. For a desired rejection rate $R$2, set $R$3, and compute: $R$4 where $R$5 and $R$6 denote true and false positives among the accepted items (Fischer et al., 2023).

2. Statistical Properties and Estimation

Population and empirical properties of P@R are formalized as follows (Hughes-Oliver, 2018):

• The population-level Precision at recall $R$7 is given by:

$R$8

$R$9 and $\hat Y = (\hat y_1, ..., \hat y_m)$0 are the class priors; $\hat Y = (\hat y_1, ..., \hat y_m)$1, $\hat Y = (\hat y_1, ..., \hat y_m)$2 are class-conditional cumulative distributions of the score.

• The empirical estimator is:

$\hat Y = (\hat y_1, ..., \hat y_m)$3

with empirical CDFs for finite labeled data.

• Asymptotic normality holds under continuity and regularity conditions, but if scores are discrete, $\hat Y = (\hat y_1, ..., \hat y_m)$4 can be inconsistent at recall levels with ties, leading to multimodal estimates. As the class imbalance increases (smaller $\hat Y = (\hat y_1, ..., \hat y_m)$5), both P@R and the overall PR curve systematically decrease, reflecting the challenge of rare-positive detection.

3. Algorithmic and Optimization Framework

The task of maximizing P@R can be framed as a constrained learning problem (Mackey et al., 2018, Tasche, 2018, Kar et al., 2015). Given samples with scores $\hat Y = (\hat y_1, ..., \hat y_m)$6 and labels $\hat Y = (\hat y_1, ..., \hat y_m)$7, the core combinatorial program is:

$\hat Y = (\hat y_1, ..., \hat y_m)$8

This is generally intractable for large $\hat Y = (\hat y_1, ..., \hat y_m)$9 due to the discrete constraint. Two main solution principles emerge:

1. Quantile-based Surrogates: Replace the hard threshold with a (smoothed or convexified) quantile estimator and the indicator function with a convex surrogate loss $m$0. The loss is then minimized via SGD:

$\mathbf{1}[\cdot]$1 (Mackey et al., 2018)

1. Optimal Thresholding on Posterior: For binary classification, thresholding at the $m$1-quantile of the positive class posterior probability $m$2 asymptotically maximizes P@R for a given recall, under regularity assumptions (Tasche, 2018).
2. Convex Surrogates for Ranking: Precision@R is non-convex. Ramp, max, and avg surrogates provide convex upper bounds—$m$3—with associated perceptron and SGD algorithms that generalize under margin conditions (Kar et al., 2015):

Surrogate	Consistency Condition	Algorithmic Use
$m$4	Weak $m$5-margin	Perceptron@k
$m$6	Strong $m$7-margin	Perceptron@k-Fast
$m$8	$m$9-margin	Perceptron@k/SGD@k

4. P@R in Practical Evaluation and Usage

P@R is used in information retrieval, keyphrase extraction, and class-imbalance classification:

• In keyphrase generation, $$\mathrm{P@}R = \frac{1}{R}\sum_{i=1}^{R} \mathbf{1}\left[\mathrm{match}(\hat y_i, Y)\right]$$0 is document-specific, enabling fair adaptation to varying lengths and complexity (Venturini et al., 5 Jun 2026).
• In reject option classifiers, P@R enables precise control of the trade-off between retained instance count and precision: only the most confident predictions up to the desired acceptance fraction are considered (Fischer et al., 2023).
• Under class imbalance, the tight coupling of P@R to the positive class renders it more sensitive than accuracy-based rejection approaches, which can be inflated by the majority class.

A typical computation involves ranking instances by prediction confidence, choosing the top $$\mathrm{P@}R = \frac{1}{R}\sum_{i=1}^{R} \mathbf{1}\left[\mathrm{match}(\hat y_i, Y)\right]$$1 (or top $$\mathrm{P@}R = \frac{1}{R}\sum_{i=1}^{R} \mathbf{1}\left[\mathrm{match}(\hat y_i, Y)\right]$$2 corresponding to recall $$\mathrm{P@}R = \frac{1}{R}\sum_{i=1}^{R} \mathbf{1}\left[\mathrm{match}(\hat y_i, Y)\right]$$3), and computing the fraction of positives among them.

5. Semantic Extensions: From P@R to SemR-p

Traditional P@R is based on surface-form matching, missing semantic equivalence (e.g., “deep learning” vs. “neural networks”). The introduction of Semantic R-Precision (SemR-p) augments the metric with graded, embedding-based semantic similarity (Venturini et al., 5 Jun 2026):

• For each of the top-$$\mathrm{P@}R = \frac{1}{R}\sum_{i=1}^{R} \mathbf{1}\left[\mathrm{match}(\hat y_i, Y)\right]$$4 predictions:
  • If its stem matches a reference, assign a score of 1.0.
  • Otherwise, score it by averaging the cosine similarity of its embedding with the $$\mathrm{P@}R = \frac{1}{R}\sum_{i=1}^{R} \mathbf{1}\left[\mathrm{match}(\hat y_i, Y)\right]$$5 closest reference keyphrase embeddings.
• SemR-p is then the mean of these scores for the top $$\mathrm{P@}R = \frac{1}{R}\sum_{i=1}^{R} \mathbf{1}\left[\mathrm{match}(\hat y_i, Y)\right]$$6 predictions.

This approach preserves ranking-awareness, still restricting credit to the early predictions, while capturing nuanced semantic overlap. Empirical results demonstrate that SemR-p is robust to the semantic–lexical tradeoff parameter $$\mathrm{P@}R = \frac{1}{R}\sum_{i=1}^{R} \mathbf{1}\left[\mathrm{match}(\hat y_i, Y)\right]$$7, sensitive to actual model differences, and more aligned with human assessment than purely lexical or order-ignorant metrics.

6. Empirical Behavior, Considerations, and Limitations

Extensive analysis reveals several key findings:

• P@R is monotonic non-increasing in acceptance rate (or non-decreasing in rejection); selecting higher $$\mathrm{P@}R = \frac{1}{R}\sum_{i=1}^{R} \mathbf{1}\left[\mathrm{match}(\hat y_i, Y)\right]$$8 generally reduces precision unless the model ranks all positives highest (Fischer et al., 2023).
• In population terms, the estimator's bias and variance are controlled by the score distributions' regularity; for discrete outputs, consistency can fail at quantile boundaries (Hughes-Oliver, 2018).
• For very high rejection rates (low retention), P@R estimates are noisy due to small sample counts.
• Surrogates and quantile-approximation yield efficient and scalable online algorithms that retain theoretical guarantees in margin-rich settings (Kar et al., 2015, Mackey et al., 2018).
• Comparative studies show that optimizing for set-based metrics (such as unordered $$\mathrm{P@}R = \frac{1}{R}\sum_{i=1}^{R} \mathbf{1}\left[\mathrm{match}(\hat y_i, Y)\right]$$9) or full-list metrics (such as NDCG) can obscure ranking errors or overgeneration, while P@R directly penalizes misplaced or untimely relevant predictions (Venturini et al., 5 Jun 2026).

7. Connections, Generalizations, and Outlook

P@R generalizes fixed-cutoff precision ($\mathbf{1}[\cdot]$0) by adapting the cutoff dynamically per instance based on the number of relevant items, making it especially suitable in domains with variable ground-truth set sizes. Its close relationship to the precision–recall curve enables its use both as a point metric and as a diagnostic across operating regimes (Hughes-Oliver, 2018, Fischer et al., 2023).

Recent semantic extensions, such as SemR-p, demonstrate the ongoing relevance of P@R as a foundation for metrics that intertwine order and content, advancing model evaluation along axes that better reflect user-centric relevance while retaining rigorous statistical properties (Venturini et al., 5 Jun 2026).

P@R continues to play a central role in ranking and decision-making tasks where early relevance, variable ground-truth cardinality, and robustness to semantic variation are critical. Ongoing work extends its utility through improved surrogates, semantic and contextual integration, and theoretical refinements in imbalanced and discrete settings.