Popularity Quantile Calibration

• The paper introduces a quantile-based framework that reframes popularity bias as an alignment challenge between historical user profiles and recommendations.
• It leverages user-specific quantile thresholds to calibrate the popularity distribution of recommended items, yielding measurable calibration errors.
• Empirical findings reveal that users with niche preferences experience higher miscalibration, highlighting the method’s potential to enhance fairness and supplier exposure.

Popularity Quantile Calibration (PQC) is a user-centric measurement framework that quantifies how well a recommender’s popularity profile aligns with each individual user’s historical preferences for popular versus niche content. In its most explicit formulation, it reframes popularity bias as an alignment problem and operationalizes alignment via quantile-based calibration between the popularity distribution of a user’s consumed items and that of the model’s recommendations (Schirmer et al., 1 Apr 2026). Earlier recommendation work had already formulated closely related ideas under the names popularity calibration, miscalibration, and popularity lift, showing that recommendation lists can be systematically more popular than users’ profiles and that this distortion is unevenly distributed across user groups (Abdollahpouri et al., 2020); (Abdollahpouri et al., 2020).

1. Conceptual scope and historical development

Popularity Quantile Calibration emerged from two adjacent lines of work. The first studied calibration in recommendation as a match between the distribution of item attributes in a user’s profile and the distribution of the same attributes in recommendations. The second studied popularity bias as the tendency of recommendation algorithms to amplify existing popularity differences across items. Their conjunction produced the idea that a recommender should not merely reduce global popularity exposure, but should match each user’s own preference for head, mid, or tail content (Abdollahpouri et al., 2019).

A foundational formulation of this idea appears in work on popularity calibration, where the recommendation list is calibrated to match the distribution of popularity groups in each user’s profile. In that setting, the catalog is partitioned into three groups by cumulative share of ratings—Head, Mid, and Tail—and re-ranking is used to optimize a trade-off between relevance and the divergence between the profile distribution and the recommendation distribution (Abdollahpouri et al., 2020). Related work on fairness and calibration showed that groups less interested in popular items tend to experience both higher popularity lift and higher miscalibration, linking popularity bias to unequal user experience (Abdollahpouri et al., 2020).

The specific term Popularity Quantile Calibration is introduced as a measurement framework in “Aligning Recommendations with User Popularity Preferences” (Schirmer et al., 1 Apr 2026). That paper shifts the emphasis from system-level debiasing to user–recommender alignment. Rather than asking whether recommendations are globally less popular, it asks whether the popularity levels in the recommendation list match the user’s historical popularity profile. This distinguishes PQC from metrics such as average recommendation popularity or exposure concentration, which summarize aggregate popularity but do not directly quantify user-level alignment (Schirmer et al., 1 Apr 2026).

Subsequent work instantiated the same basic principle in multiple recommendation settings. Point-of-interest recommendation used long-tail bins Tail, Mid, and Head derived from check-in frequencies and optimized list-level alignment to user popularity profiles (Forster et al., 4 Jul 2025). Session-based recommendation treated popularity calibration as prediction of the proportion of tail items appropriate for the current session, using two bins—head and tail—and aligning the list composition to the session’s own popularity composition (Chen et al., 2021). This suggests that PQC is best understood as a family of calibration formalisms whose common object is the alignment between historical and recommended popularity distributions.

2. Formalization and measurement

In the quantile-based formulation, item popularity is defined as the total number of interactions a catalog item receives across all users and all histories in the evaluation corpus: $s(i) \;=\; \sum_{u\in \mathcal{U}}\sum_{j=1}^{T_u} \mathbbm{1}\{\mathcal{H}_u(j) = i\}.$ For a user $u$, the historical distribution $p(s\mid u)$ is the empirical distribution of popularity scores $s(i)$ for items in the user history $\mathcal{H}_u$, and the recommender’s distribution $q(s\mid u)$ is the empirical distribution of popularity scores for items in the top-$K$ list $\mathcal{R}_u$ (Schirmer et al., 1 Apr 2026).

PQC uses per-user quantile thresholds derived from $q(s\mid u)$, not from global popularity. Let $F_q$ be the cumulative distribution function of $u$0, and let $u$1 be the corresponding quantile function: $u$2 For a grid $u$3, the user history is then evaluated at the recommender’s thresholds $u$4, yielding

$u$5

The points $u$6 define the popularity calibration curve. Perfect calibration is the identity line $u$7; curves above the diagonal indicate that recommendations are too popular relative to the user’s history, while curves below the diagonal indicate that they are too niche (Schirmer et al., 1 Apr 2026).

The aggregate PQC score is the Popularity Calibration Error: $u$8 Because PQC uses quantiles and ranks, it is described as scale-invariant and robust to outliers; the calibration curve captures direction and dispersion, while PCE captures magnitude but not direction (Schirmer et al., 1 Apr 2026).

Earlier popularity-calibration work used bin-based rather than quantile-threshold formulations. A widely used construction defines item popularity as

$u$9

partitions items into popularity groups or quantiles, and compares the user’s historical distribution to the recommendation distribution across those groups (Abdollahpouri et al., 2020). The miscalibration measure is often the Hellinger distance,

$p(s\mid u)$0

where $p(s\mid u)$1 is the user’s profile distribution across popularity quantiles and $p(s\mid u)$2 is the recommendation distribution (Abdollahpouri et al., 2020). In Calibrated Popularity, the divergence is Jensen–Shannon divergence over Head, Mid, and Tail bins, producing the user-level User Popularity Deviation (UPD) and item-level Item Popularity Deviation (IPD) (Abdollahpouri et al., 2020).

Popularity lift remains a separate but complementary quantity. For a group $p(s\mid u)$3, the profile and recommendation average popularity are

$p(s\mid u)$4

and the amplification measure is

$p(s\mid u)$5

Positive $p(s\mid u)$6 means the algorithm increases exposure to popular items beyond what is in users’ profiles (Abdollahpouri et al., 2019).

3. Algorithmic formulations and mitigation strategies

PQC can function purely as a measurement instrument, but much of the literature couples it with re-ranking or inference-time control. The standard list-level formulation maximizes a trade-off between relevance and popularity alignment: $p(s\mid u)$7 where $p(s\mid u)$8 is the user’s target popularity distribution, $p(s\mid u)$9 is the popularity distribution of the recommendation list, and $s(i)$0 is a bounded divergence on probability distributions in $s(i)$1 (Forster et al., 4 Jul 2025). A greedy implementation updates the list one item at a time, using incremental changes in divergence. In the POI setting, this yields complexity $s(i)$2, with $s(i)$3 bins and $s(i)$4 candidate items (Forster et al., 4 Jul 2025).

The three-bin Head/Mid/Tail version is especially prominent. In Calibrated Popularity, item groups are defined by cumulative popularity mass, the user propensity distribution $s(i)$5 is estimated from the profile, the recommendation distribution $s(i)$6 is estimated from the current list, and Jensen–Shannon divergence is minimized jointly with relevance (Abdollahpouri et al., 2020). In POI recommendation, the bins are Tail, Mid, and Head, corresponding to bottom 20%, middle 60%, and top 20% of items by normalized check-in frequency; two calibration modes are reported: CP$s(i)$7, which is accuracy-oriented, and CP$s(i)$8, which is calibration-oriented (Forster et al., 4 Jul 2025).

Session-based recommendation replaces long-term user profiles with session-level targets. For a session $s(i)$9, the target popularity composition is

$\mathcal{H}_u$0

with $\mathcal{H}_u$1 indicating head or tail membership. A calibration module predicts a two-bin distribution $\mathcal{H}_u$2 from the session representation $\mathcal{H}_u$3, and training alternates between predicting the composition of the current top-$\mathcal{H}_u$4 list and aligning that prediction to the session target, using KL-based losses and a two-stage curriculum strategy (Chen et al., 2021). This construction does not apply explicit inference-time re-ranking; instead, the calibrated backbone directly emits scores whose top-$\mathcal{H}_u$5 composition is better aligned with session preferences (Chen et al., 2021).

The most recent mitigation approach, SPREE, is inference-time rather than re-ranking-based. It first identifies a popularity direction in representation space using contrastive sets of high-popularity and low-popularity sequences, then computes a signed user-level popularity bias around the median recommender quantile,

$\mathcal{H}_u$6

and finally steers activations as

$\mathcal{H}_u$7

The sign of $\mathcal{H}_u$8 determines whether to steer toward or away from popular content, and the magnitude scales the correction to the estimated misalignment (Schirmer et al., 1 Apr 2026). Unlike global popularity reduction, this explicitly targets alignment rather than uniformly reducing popularity.

4. Empirical findings, group disparities, and fairness

A consistent empirical finding is that user groups with lower interest in popular items receive recommendations with both higher popularity amplification and higher miscalibration. In MovieLens 1M and Yahoo Movies, users were divided into ten groups $\mathcal{H}_u$9 by the average popularity of items in their profiles, with $q(s\mid u)$0 the most niche-focused and $q(s\mid u)$1 the most blockbuster-focused. For MovieLens Top-10 recommendations, ItemKNN produced $q(s\mid u)$2 versus $q(s\mid u)$3, alongside $q(s\mid u)$4 versus $q(s\mid u)$5; Most-popular produced $q(s\mid u)$6 versus $q(s\mid u)$7, alongside $q(s\mid u)$8 versus $q(s\mid u)$9 (Abdollahpouri et al., 2020). The same qualitative pattern appears on Yahoo Movies, and the reported interpretation is direct: “the more a group is affected by algorithmic popularity bias, the more their recommendations are miscalibrated” (Abdollahpouri et al., 2020).

Later studies extended this pattern to accuracy disparities. On LastFm, MovieLens, and MyAnimeList, users were split into LowPop, MedPop, and HighPop groups. The reported result is that users with little interest in popular content receive the worst recommendation accuracy, and that this is aligned with miscalibration and popularity lift (Kowald et al., 2023). In MyAnimeList, the paper further reports that particular genres contribute disproportionately to inconsistency in recommendation performance, especially in terms of miscalibration (Kowald et al., 2023).

The user-alignment framing yields different conclusions from global debiasing metrics. In the PQC/SPREE study, uniform popularity reduction methods such as IPR, PopSteer, and Random Neighbors often worsened user-level alignment, while SPREE shifted calibration curves toward the identity and improved PCE@100 while largely preserving NDCG@100 (Schirmer et al., 1 Apr 2026). Under a constraint of at most 10% NDCG drop, SPREE reduced PCE@100 across all reported datasets: fs-tky from $K$0 to $K$1, ml-1m from $K$2 to $K$3, ml-20m from $K$4 to $K$5, and RateBeer from $K$6 to $K$7 (Schirmer et al., 1 Apr 2026).

In multistakeholder evaluation, popularity calibration also has supplier-side effects. On MovieLens and Last.fm, the Calibrated Popularity re-ranker achieved the lowest UPD and the best Equity of Attention Supplier Fairness (ESF) and Supplier Popularity Deviation (SPD), even though suppliers were not directly optimized in the objective (Abdollahpouri et al., 2020). This suggests that calibrating recommendations to users’ popularity preferences can redistribute exposure across suppliers more evenly as a secondary effect.

POI recommendation studies report the same relevance–alignment trade-off. Calibration techniques were found to effectively align recommendation popularity with user preferences, provided there is a careful balance between accuracy and bias mitigation, and the combination of calibration and context-awareness yielded recommendations that balance accuracy and close alignment with the users’ popularity profiles (Forster et al., 4 Jul 2025).

5. Relation to general quantile calibration

Outside recommendation, quantile calibration is a mature concept in probabilistic regression. A probabilistic regressor outputs a predictive CDF $K$8, and with the Probability Integral Transform $K$9, perfect probabilistic calibration requires $\mathcal{R}_u$0. Equivalently, quantile calibration requires

$\mathcal{R}_u$1

with central prediction intervals having nominal coverage $\mathcal{R}_u$2 (Dheur et al., 2024). In that literature, calibration is assessed with quantities such as PIT calibration error, Quantile Expected Calibration Error, and the Integral Calibration Index (Dheur et al., 2024).

This is not identical to recommendation-oriented PQC. In recommender systems, the object being calibrated is not a predictive target variable $\mathcal{R}_u$3 but the distribution of recommendation popularity levels relative to each user’s historical popularity profile. Nevertheless, one paper makes the connection explicit: “Popularity Quantile Calibration” is not a term used in the paper, but if it is interpreted as quantile calibration under popularity-based covariate structure, then subgroup calibrators $\mathcal{R}_u$4 can be defined over popularity strata $\mathcal{R}_u$5, and one can evaluate coverage within strata $\mathcal{R}_u$6 (Dheur et al., 2024). This suggests that popularity-stratified recommendation calibration and subgroup quantile calibration in regression share a common monotone-calibration perspective.

Other quantile-calibration papers contribute transferable methodology. Quantile Regularization defines a trainable regularizer that penalizes deviations of the distribution of $\mathcal{R}_u$7 from Uniform$\mathcal{R}_u$8, rather than relying on a post-hoc calibration set (Utpala et al., 2020). Distribution Calibration for Regression distinguishes marginal quantile calibration from the stronger notion of distribution calibration and introduces instance-specific monotone transformations of CDF values via a GP–Beta calibration map (Song et al., 2019). Multi-Level Quantile Tracker provides simultaneous calibration of multiple quantile levels under adversarial distribution shifts, while preserving quantile ordering by projection onto the isotonic cone (Ding et al., 29 Dec 2025). Regularization Strategies for Quantile Regression uses deep lattice networks monotone in the quantile input $\mathcal{R}_u$9 and applies rate constraints to improve calibration on specified subsets (Narayan et al., 2021).

A plausible implication is that future recommender formulations of PQC may borrow more directly from this regression literature: subgroup-specific monotone calibrators, end-to-end quantile regularizers, and constrained calibration on subsets of interest are already described in a compatible vocabulary (Dheur et al., 2024).

6. Limitations, design choices, and open directions

PQC depends on reliable estimation of a user’s historical popularity preference. Cold-start users require some history to estimate $q(s\mid u)$0; suggested fallbacks include a minimum history size threshold, cohort-level calibration, or global priors (Schirmer et al., 1 Apr 2026). Cold-start items and catalog shifts also affect popularity scores, and the PQC formulation explicitly does not apply smoothing or time-windowing beyond the dataset’s temporal split (Schirmer et al., 1 Apr 2026). In production settings, this makes periodic recomputation of $q(s\mid u)$1 or popularity bins an operational concern.

The choice of bins or quantile resolution is consequential. Some work uses per-user equal-mass quantile bins derived from $q(s\mid u)$2, with typical grids such as $q(s\mid u)$3 or $q(s\mid u)$4 (Schirmer et al., 1 Apr 2026). Other work uses coarse bins such as Head/Mid/Tail or Head/Tail because they are robust and interpretable (Abdollahpouri et al., 2020); (Chen et al., 2021). Finer quantiles provide more detail, but this suggests greater sensitivity to candidate scarcity and data sparsity. Several papers note that calibration cannot realize the desired target if the base model fails to retrieve enough items from the relevant popularity regions (Forster et al., 4 Jul 2025).

A second limitation is the relevance–calibration trade-off. In POI recommendation, stronger calibration improves alignment and reduces ARP and PopLift, but can reduce nDCG, especially in the more calibration-oriented setting (Forster et al., 4 Jul 2025). In session-based recommendation, too large a calibration weight $q(s\mid u)$5 hurts both accuracy and long-tail metrics (Chen et al., 2021). In SPREE, aggressive steering can harm alignment when the base recommender is already well calibrated, as reported for fs-tky (Schirmer et al., 1 Apr 2026).

Fairness and multistakeholder effects remain partly indirect. Popularity calibration reduces user misalignment and can improve supplier fairness, but some papers note that tail under-exposure persists because the candidate pool itself is popularity-skewed (Abdollahpouri et al., 2020). This implies that PQC alone does not eliminate all long-tail effects; retrieval, candidate generation, and exposure dynamics still matter.

Open directions are stated explicitly across the literature. PQC has been evaluated in recommendation domains such as movies, music, anime, POIs, and sequential recommendation, but further evaluation in other domains is identified as future work (Kowald et al., 2023). More granular user grouping, temporal evaluation, position-aware calibration, multi-objective optimization, and multiple quantile anchors for tail-specific correction are all proposed or implied next steps (Kowald et al., 2023); (Forster et al., 4 Jul 2025); (Schirmer et al., 1 Apr 2026). Across these variants, the central technical problem remains stable: calibrating recommendation popularity not in the aggregate, but relative to each user’s own popularity profile.