---
title: Popularity Quantile Calibration
url: https://www.emergentmind.com/topics/popularity-quantile-calibration
type: topic
---

# Popularity Quantile Calibration

Popularity Quantile Calibration (PQC) is a user-centric measurement framework that quantifies how well a recommender’s popularity profile aligns with each individual user’s historical preferences for popular versus niche content. In its most explicit formulation, it reframes popularity bias as an alignment problem and operationalizes alignment via quantile-based calibration between the popularity distribution of a user’s consumed items and that of the model’s recommendations [2604.01036]. Earlier recommendation work had already formulated closely related ideas under the names *popularity calibration*, *miscalibration*, and *popularity lift*, showing that recommendation lists can be systematically more popular than users’ profiles and that this distortion is unevenly distributed across user groups [2007.12230]; [2008.09273].

## 1. Conceptual scope and historical development

Popularity Quantile Calibration emerged from two adjacent lines of work. The first studied calibration in recommendation as a match between the distribution of item attributes in a user’s profile and the distribution of the same attributes in recommendations. The second studied popularity bias as the tendency of recommendation algorithms to amplify existing popularity differences across items. Their conjunction produced the idea that a recommender should not merely reduce global popularity exposure, but should match each user’s own preference for head, mid, or tail content [1910.05755].

A foundational formulation of this idea appears in work on *popularity calibration*, where the recommendation list is calibrated to match the distribution of popularity groups in each user’s profile. In that setting, the catalog is partitioned into three groups by cumulative share of ratings—Head, Mid, and Tail—and re-ranking is used to optimize a trade-off between relevance and the divergence between the profile distribution and the recommendation distribution [2007.12230]. Related work on fairness and calibration showed that groups less interested in popular items tend to experience both higher popularity lift and higher miscalibration, linking popularity bias to unequal user experience [2008.09273].

The specific term *Popularity Quantile Calibration* is introduced as a measurement framework in “Aligning Recommendations with User Popularity Preferences” [2604.01036]. That paper shifts the emphasis from system-level debiasing to user–recommender alignment. Rather than asking whether recommendations are globally less popular, it asks whether the popularity levels in the recommendation list match the user’s historical popularity profile. This distinguishes PQC from metrics such as average recommendation popularity or exposure concentration, which summarize aggregate popularity but do not directly quantify user-level alignment [2604.01036].

Subsequent work instantiated the same basic principle in multiple recommendation settings. Point-of-interest recommendation used long-tail bins Tail, Mid, and Head derived from check-in frequencies and optimized list-level alignment to user popularity profiles [2507.03503]. Session-based recommendation treated popularity calibration as prediction of the proportion of tail items appropriate for the current session, using two bins—head and tail—and aligning the list composition to the session’s own popularity composition [2112.02581]. This suggests that PQC is best understood as a family of calibration formalisms whose common object is the alignment between historical and recommended popularity distributions.

## 2. Formalization and measurement

In the quantile-based formulation, item popularity is defined as the total number of interactions a catalog item receives across all users and all histories in the evaluation corpus:
\[
s(i) \;=\; \sum_{u\in \mathcal{U}}\sum_{j=1}^{T_u} \mathbbm{1}\{\mathcal{H}_u(j) = i\}.
\]
For a user \(u\), the historical distribution \(p(s\mid u)\) is the empirical distribution of popularity scores \(s(i)\) for items in the user history \(\mathcal{H}_u\), and the recommender’s distribution \(q(s\mid u)\) is the empirical distribution of popularity scores for items in the top-\(K\) list \(\mathcal{R}_u\) [2604.01036].

PQC uses per-user quantile thresholds derived from \(q(s\mid u)\), not from global popularity. Let \(F_q\) be the cumulative distribution function of \(q(s\mid u)\), and let \(F_q^{-1}\) be the corresponding quantile function:
\[
F_q(s) \;=\; \mathbb{P}_{x\sim q(s\mid u)}(x \le s),
\qquad
F_q^{-1}(\tau) \;=\; \inf\{s: \tau \le F_q(s)\}.
\]
For a grid \(\{\tau_j\}_{j=1}^m\), the user history is then evaluated at the recommender’s thresholds \(s^{(\tau_j)} = F_q^{-1}(\tau_j)\), yielding
\[
\hat{\tau}_j \;=\; \frac{1}{|\mathcal{H}_u|}\sum_{i\in\mathcal{H}_u} \mathbbm{1}\{\, s(i) \le s^{(\tau_j)} \,\}.
\]
The points \((\tau_j,\hat{\tau}_j)\) define the popularity calibration curve. Perfect calibration is the identity line \(\hat{\tau}_j=\tau_j\); curves above the diagonal indicate that recommendations are too popular relative to the user’s history, while curves below the diagonal indicate that they are too niche [2604.01036].

The aggregate PQC score is the Popularity Calibration Error:
\[
\mathrm{PCE}(u) \;=\; \frac{1}{m}\sum_{j=1}^{m} \big(\tau_j - \hat{\tau}_j\big)^2,
\qquad
\mathrm{PCE} \;=\; \frac{1}{|\mathcal{U}|}\sum_{u\in\mathcal{U}} \mathrm{PCE}(u).
\]
Because PQC uses quantiles and ranks, it is described as scale-invariant and robust to outliers; the calibration curve captures direction and dispersion, while PCE captures magnitude but not direction [2604.01036].

Earlier popularity-calibration work used bin-based rather than quantile-threshold formulations. A widely used construction defines item popularity as
\[
\theta(i) = \frac{|\{u \in U : i \in \Gamma_u\}|}{|U|},
\]
partitions items into popularity groups or quantiles, and compares the user’s historical distribution to the recommendation distribution across those groups [2008.09273]. The miscalibration measure is often the Hellinger distance,
\[
MC_u^{pop} = \frac{1}{\sqrt{2}} \lVert \sqrt{\pi_u} - \sqrt{\rho_u} \rVert_2,
\]
where \(\pi_u\) is the user’s profile distribution across popularity quantiles and \(\rho_u\) is the recommendation distribution [2008.09273]. In *Calibrated Popularity*, the divergence is Jensen–Shannon divergence over Head, Mid, and Tail bins, producing the user-level User Popularity Deviation (UPD) and item-level Item Popularity Deviation (IPD) [2007.12230].

Popularity lift remains a separate but complementary quantity. For a group \(G\), the profile and recommendation average popularity are
\[
GAP_p(G) = \frac{1}{|G|} \sum_{u \in G} \frac{1}{|\Gamma_u|} \sum_{i \in \Gamma_u} \theta(i),
\qquad
GAP_q(G) = \frac{1}{|G|} \sum_{u \in G} \frac{1}{|\Lambda_u|} \sum_{i \in \Lambda_u} \theta(i),
\]
and the amplification measure is
\[
PL(G) = \frac{GAP_q(G) - GAP_p(G)}{GAP_p(G)}.
\]
Positive \(PL(G)\) means the algorithm increases exposure to popular items beyond what is in users’ profiles [1910.05755].

## 3. Algorithmic formulations and mitigation strategies

PQC can function purely as a measurement instrument, but much of the literature couples it with re-ranking or inference-time control. The standard list-level formulation maximizes a trade-off between relevance and popularity alignment:
\[
L^*_u = \arg\max_{L_u,\ |L_u|=n} \big[(1-\lambda)\cdot Rel(L_u)-\lambda\cdot \Im(P_u,Q(L_u))\big],
\]
where \(P_u\) is the user’s target popularity distribution, \(Q(L_u)\) is the popularity distribution of the recommendation list, and \(\Im\) is a bounded divergence on probability distributions in \([0,1]\) [2507.03503]. A greedy implementation updates the list one item at a time, using incremental changes in divergence. In the POI setting, this yields complexity \(O(K\cdot |C_u| \cdot Q)\), with \(Q=3\) bins and \(|C_u|=150\) candidate items [2507.03503].

The three-bin Head/Mid/Tail version is especially prominent. In *Calibrated Popularity*, item groups are defined by cumulative popularity mass, the user propensity distribution \(P\) is estimated from the profile, the recommendation distribution \(Q\) is estimated from the current list, and Jensen–Shannon divergence is minimized jointly with relevance [2007.12230]. In POI recommendation, the bins are Tail, Mid, and Head, corresponding to bottom 20%, middle 60%, and top 20% of items by normalized check-in frequency; two calibration modes are reported: CP\(_H\), which is accuracy-oriented, and CP\(_\Im\), which is calibration-oriented [2507.03503].

Session-based recommendation replaces long-term user profiles with session-level targets. For a session \(s=\{x_1,\dots,x_n\}\), the target popularity composition is
\[
q_s = \frac{1}{n}\sum_{x_i\in s} pop(x_i),
\]
with \(pop(x)\in\{0,1\}^2\) indicating head or tail membership. A calibration module predicts a two-bin distribution \(\hat p_s\) from the session representation \(h_s\), and training alternates between predicting the composition of the current top-\(N\) list and aligning that prediction to the session target, using KL-based losses and a two-stage curriculum strategy [2112.02581]. This construction does not apply explicit inference-time re-ranking; instead, the calibrated backbone directly emits scores whose top-\(N\) composition is better aligned with session preferences [2112.02581].

The most recent mitigation approach, SPREE, is inference-time rather than re-ranking-based. It first identifies a popularity direction in representation space using contrastive sets of high-popularity and low-popularity sequences, then computes a signed user-level popularity bias around the median recommender quantile,
\[
e(u)=\hat{\tau}_{\mathrm{med}}-\tau_{\mathrm{med}},\qquad \tau_{\mathrm{med}}=0.5,
\]
and finally steers activations as
\[
h^{*}_{t,\ell} \;=\; h_{t,\ell} \;+\; \lambda \,\hat{f}(h_{t,\ell}) \, v_{t,\ell}.
\]
The sign of \(\hat f(h_{t,\ell})\) determines whether to steer toward or away from popular content, and the magnitude scales the correction to the estimated misalignment [2604.01036]. Unlike global popularity reduction, this explicitly targets alignment rather than uniformly reducing popularity.

## 4. Empirical findings, group disparities, and fairness

A consistent empirical finding is that user groups with lower interest in popular items receive recommendations with both higher popularity amplification and higher miscalibration. In MovieLens 1M and Yahoo Movies, users were divided into ten groups \(G_1,\dots,G_{10}\) by the average popularity of items in their profiles, with \(G_1\) the most niche-focused and \(G_{10}\) the most blockbuster-focused. For MovieLens Top-10 recommendations, ItemKNN produced \(PL(G_1)=12.19\) versus \(PL(G_{10})=0.458\), alongside \(MC(G_1)=0.418\) versus \(MC(G_{10})=0.250\); Most-popular produced \(PL(G_1)=15.7\) versus \(0.563\), alongside \(MC(G_1)=0.501\) versus \(0.277\) [2008.09273]. The same qualitative pattern appears on Yahoo Movies, and the reported interpretation is direct: “the more a group is affected by algorithmic popularity bias, the more their recommendations are miscalibrated” [2008.09273].

Later studies extended this pattern to accuracy disparities. On LastFm, MovieLens, and MyAnimeList, users were split into LowPop, MedPop, and HighPop groups. The reported result is that users with little interest in popular content receive the worst recommendation accuracy, and that this is aligned with miscalibration and popularity lift [2303.00400]. In MyAnimeList, the paper further reports that particular genres contribute disproportionately to inconsistency in recommendation performance, especially in terms of miscalibration [2303.00400].

The user-alignment framing yields different conclusions from global debiasing metrics. In the PQC/SPREE study, uniform popularity reduction methods such as IPR, PopSteer, and Random Neighbors often worsened user-level alignment, while SPREE shifted calibration curves toward the identity and improved PCE@100 while largely preserving NDCG@100 [2604.01036]. Under a constraint of at most 10% NDCG drop, SPREE reduced PCE@100 across all reported datasets: fs-tky from \(0.210\) to \(0.200\), ml-1m from \(0.211\) to \(0.176\), ml-20m from \(0.418\) to \(0.382\), and RateBeer from \(0.321\) to \(0.293\) [2604.01036].

In multistakeholder evaluation, popularity calibration also has supplier-side effects. On MovieLens and Last.fm, the Calibrated Popularity re-ranker achieved the lowest UPD and the best Equity of Attention Supplier Fairness (ESF) and Supplier Popularity Deviation (SPD), even though suppliers were not directly optimized in the objective [2007.12230]. This suggests that calibrating recommendations to users’ popularity preferences can redistribute exposure across suppliers more evenly as a secondary effect.

POI recommendation studies report the same relevance–alignment trade-off. Calibration techniques were found to effectively align recommendation popularity with user preferences, provided there is a careful balance between accuracy and bias mitigation, and the combination of calibration and context-awareness yielded recommendations that balance accuracy and close alignment with the users’ popularity profiles [2507.03503].

## 5. Relation to general quantile calibration

Outside recommendation, quantile calibration is a mature concept in probabilistic regression. A probabilistic regressor outputs a predictive CDF \(F_\theta(\cdot\mid x)\), and with the Probability Integral Transform \(U=F_\theta(Y\mid X)\), perfect probabilistic calibration requires \(U\sim Uniform(0,1)\). Equivalently, quantile calibration requires
\[
P(Y \le q_p(X)\mid X)=p,
\]
with central prediction intervals having nominal coverage \(1-\alpha\) [2403.11964]. In that literature, calibration is assessed with quantities such as PIT calibration error, Quantile Expected Calibration Error, and the Integral Calibration Index [2403.11964].

This is not identical to recommendation-oriented PQC. In recommender systems, the object being calibrated is not a predictive target variable \(Y\) but the distribution of recommendation popularity levels relative to each user’s historical popularity profile. Nevertheless, one paper makes the connection explicit: “Popularity Quantile Calibration” is not a term used in the paper, but if it is interpreted as quantile calibration under popularity-based covariate structure, then subgroup calibrators \(\Phi_S\) can be defined over popularity strata \(S\), and one can evaluate coverage within strata \(P(Y \le q_p(X)\mid X\in S)\approx p\) [2403.11964]. This suggests that popularity-stratified recommendation calibration and subgroup quantile calibration in regression share a common monotone-calibration perspective.

Other quantile-calibration papers contribute transferable methodology. Quantile Regularization defines a trainable regularizer that penalizes deviations of the distribution of \(U=[F(X)](Y)\) from Uniform\([0,1]\), rather than relying on a post-hoc calibration set [2002.12860]. Distribution Calibration for Regression distinguishes marginal quantile calibration from the stronger notion of distribution calibration and introduces instance-specific monotone transformations of CDF values via a GP–Beta calibration map [1905.06023]. Multi-Level Quantile Tracker provides simultaneous calibration of multiple quantile levels under adversarial distribution shifts, while preserving quantile ordering by projection onto the isotonic cone [2512.23671]. Regularization Strategies for Quantile Regression uses deep lattice networks monotone in the quantile input \(\tau\) and applies rate constraints to improve calibration on specified subsets [2102.05135].

A plausible implication is that future recommender formulations of PQC may borrow more directly from this regression literature: subgroup-specific monotone calibrators, end-to-end quantile regularizers, and constrained calibration on subsets of interest are already described in a compatible vocabulary [2403.11964].

## 6. Limitations, design choices, and open directions

PQC depends on reliable estimation of a user’s historical popularity preference. Cold-start users require some history to estimate \(p(s\mid u)\); suggested fallbacks include a minimum history size threshold, cohort-level calibration, or global priors [2604.01036]. Cold-start items and catalog shifts also affect popularity scores, and the PQC formulation explicitly does not apply smoothing or time-windowing beyond the dataset’s temporal split [2604.01036]. In production settings, this makes periodic recomputation of \(s(i)\) or popularity bins an operational concern.

The choice of bins or quantile resolution is consequential. Some work uses per-user equal-mass quantile bins derived from \(q(s\mid u)\), with typical grids such as \(m=6\) or \(m=11\) [2604.01036]. Other work uses coarse bins such as Head/Mid/Tail or Head/Tail because they are robust and interpretable [2007.12230]; [2112.02581]. Finer quantiles provide more detail, but this suggests greater sensitivity to candidate scarcity and data sparsity. Several papers note that calibration cannot realize the desired target if the base model fails to retrieve enough items from the relevant popularity regions [2507.03503].

A second limitation is the relevance–calibration trade-off. In POI recommendation, stronger calibration improves alignment and reduces ARP and PopLift, but can reduce nDCG, especially in the more calibration-oriented setting [2507.03503]. In session-based recommendation, too large a calibration weight \(\lambda\) hurts both accuracy and long-tail metrics [2112.02581]. In SPREE, aggressive steering can harm alignment when the base recommender is already well calibrated, as reported for fs-tky [2604.01036].

Fairness and multistakeholder effects remain partly indirect. Popularity calibration reduces user misalignment and can improve supplier fairness, but some papers note that tail under-exposure persists because the candidate pool itself is popularity-skewed [2007.12230]. This implies that PQC alone does not eliminate all long-tail effects; retrieval, candidate generation, and exposure dynamics still matter.

Open directions are stated explicitly across the literature. PQC has been evaluated in recommendation domains such as movies, music, anime, POIs, and sequential recommendation, but further evaluation in other domains is identified as future work [2303.00400]. More granular user grouping, temporal evaluation, position-aware calibration, multi-objective optimization, and multiple quantile anchors for tail-specific correction are all proposed or implied next steps [2303.00400]; [2507.03503]; [2604.01036]. Across these variants, the central technical problem remains stable: calibrating recommendation popularity not in the aggregate, but relative to each user’s own popularity profile.

Source: https://www.emergentmind.com/topics/popularity-quantile-calibration