RMBR: Unified Bias Metric in Recommenders

• RMBR is a unified metric that quantifies how recommender systems over- or under-represent items or groups in their top-K outputs.
• It generalizes specific measures like recency bias, exposure disparity, and category bias to facilitate cross-model and cross-domain analysis.
• The framework informs practical mitigation strategies via post-hoc corrections, training regularization, and exposure-aware reward adjustments.

A Recommendation Model Bias Rate (RMBR) is a formal metric framework for quantifying how a recommender system systematically over- or under-represents certain items, item categories, temporal positions, or user groups in its outputs. This approach generalizes various specific bias measures—such as recency bias, exposure disparity, or mainstreamness gap—into a unified rate-based structure, facilitating comparisons across models, datasets, and domains. RMBR metrics inform both diagnosis and mitigation of fairness and diversity issues in collaborative, sequential, and online learning recommendation environments.

1. Formal Definition and General Framework

RMBR quantifies the frequency with which a recommender’s top-K outputs systematically amplify or suppress a specific source of bias. For a given bias class, define a set $B(s)$ of bias-relevant candidate items for each context $s$ (e.g., a session, user, or round). The general RMBR at cutoff $K$ is

$$\mathrm{RMBR}_B@K = \frac{1}{|Q|} \sum_{s\in Q} \mathbf{1}\bigl(\exists\,i\in B(s)\,\text{s.t.}\,\mathrm{rank}_s(i)\le K\bigr)$$

where $Q$ is an evaluation set of contexts, $\mathrm{rank}_s(i)$ is the ranking position of $i$ in $s$, and $\mathbf{1}(\cdot)$ is the indicator function. Each instantiation of $B(s)$ maps to a separate bias concern:

• Last-item recency bias: $s$0.
• Popularity bias: $s$1 is the set of globally most popular items.
• Position bias: $s$2.
• Category bias: $s$3 as historically preferred categories for the user/session.

This hit-rate-based approach can be further generalized for item exposure frequency or category coverage, forming the backbone for recent advancements in fairness-aware recommender evaluation (Oh et al., 2024, Mansoury et al., 2019, Mansoury et al., 2024).

2. Instantiations of RMBR: Recency, Exposure, and Group Disparity

Recency Bias (HRLI):

The Hit Rate of the Last Item (HRLI@K) is a canonical instantiation:

$s$4

where $s$5 is the last item of session $s$6. High HRLI@K values indicate strong recency bias, i.e., the tendency of the model to rank the most recent interaction disproportionately highly. This pattern is prevalent in self-attention and Transformer-based sequential recommenders, as well as in recurrent models, and has been shown to suppress serendipity and long-term preference modeling (Oh et al., 2024).

Group Category Bias (Bias Disparity):

RMBR also captures group-wise bias via category over- or under-representation. For a group $s$7 and category $s$8, Bias Disparity is:

$s$9

with

$K$0

where $K$1 ($K$2) is the fraction of group-$K$3 interactions or recommendations in $K$4, and $K$5 is the catalog prevalence of $K$6. Aggregated over categories, this yields a group RMBR as in

$K$7

This quantifies overall deviation between recommendations and actual historical group preferences (Mansoury et al., 2019).

Exposure Bias and Online Fairness:

In online and contextual bandit environments, RMBR operationalizes exposure bias through the Gini index of item exposure distributions:

$K$8

$K$9

where $$\mathrm{RMBR}_B@K = \frac{1}{|Q|} \sum_{s\in Q} \mathbf{1}\bigl(\exists\,i\in B(s)\,\text{s.t.}\,\mathrm{rank}_s(i)\le K\bigr)$$0 may count binary inclusions or be weighted by position and item merit. This formulation accommodates temporally evolving bias and is applicable to online learning to rank and bandit models (Mansoury et al., 2024).

3. Properties, Interpretation, and Methodological Best Practices

RMBR-type metrics possess the following critical properties:

• Range: $$\mathrm{RMBR}_B@K = \frac{1}{|Q|} \sum_{s\in Q} \mathbf{1}\bigl(\exists\,i\in B(s)\,\text{s.t.}\,\mathrm{rank}_s(i)\le K\bigr)$$1 by normalization.
• Sensitivity: Tailored to the particular bias class encoded by $$\mathrm{RMBR}_B@K = \frac{1}{|Q|} \sum_{s\in Q} \mathbf{1}\bigl(\exists\,i\in B(s)\,\text{s.t.}\,\mathrm{rank}_s(i)\le K\bigr)$$2 or $$\mathrm{RMBR}_B@K = \frac{1}{|Q|} \sum_{s\in Q} \mathbf{1}\bigl(\exists\,i\in B(s)\,\text{s.t.}\,\mathrm{rank}_s(i)\le K\bigr)$$3. For example, HRLI@K is specifically sensitive to over-recommendation of the most recent session item and is agnostic to deeper history (Oh et al., 2024).
• Comparability: Appropriate normalization (e.g., by category prevalence or item merit) is essential for legitimate cross-category or cross-model comparisons (Mansoury et al., 2019, Mansoury et al., 2024).
• Aggregate vs. Fine-grained Reporting:
  • Group-wise RMBR (e.g., gender, mainstreamness) should be reported alongside overall population rates to expose intersectional effects.
  • Position-weighted and merit-weighted variants separate "simple exposure" from "high-value exposure" (e.g., top-slot recommendations, high-utility groups) (Mansoury et al., 2024).

The decision on $$\mathrm{RMBR}_B@K = \frac{1}{|Q|} \sum_{s\in Q} \mathbf{1}\bigl(\exists\,i\in B(s)\,\text{s.t.}\,\mathrm{rank}_s(i)\le K\bigr)$$4, exposure notion, and normalizer must match the application’s operational or fairness requirements. Coverage, long-tail exposure, and popularity distortion are all accessible via targeted RMBR configurations.

4. Empirical Evaluation and Observed Model Behavior

Empirical studies consistently show high RMBR for deep sequential models on recency (HRLI@K up to 0.99), and non-trivial group/category RMBR for collaborative and trust-aware models (Oh et al., 2024, Mansoury et al., 2019). Key findings:

• HRLI@10 tends to far exceed true Hit@10 (next-item prediction), exposing recency over-weighting.
• Removing the last item from consideration during evaluation ("post-hoc mitigation") can increase NDCG@5 by up to 43% in highly biased models (Oh et al., 2024).
• In collaborative group/category bias experiments, social-regularization methods (SoReg), sparsity-aware models (SLIM), and trust-based kNN outperform pure matrix factorization in minimizing bias disparity under equal nDCG (Mansoury et al., 2019).
• Online bandit algorithms accumulate exposure bias over time unless reward models incorporate position-weighting; adaptations such as the Exposure-Aware reward guarantee bounded regret and steady improvements in exposure fairness without appreciable utility loss (Mansoury et al., 2024).

5. Bias Mitigation Strategies

RMBR analysis motivates a suite of mitigation approaches, which fall into evaluation-time and training-time modifications:

• Post-hoc evaluation correction: Exclude or down-rank bias-related items during evaluation to reveal "true" model utility (e.g., remove the last seen item, or top-N populars) (Oh et al., 2024).
• Training regularization: Penalize the model’s score for over-represented items, categories, or positions via explicit loss terms or regularization. For recency, proposed strategies include recency dropout (masking recent interactions), or explicit penalties for high-scored last items (Oh et al., 2024).
• Cost-sensitive loss re-weighting: Weight the loss function towards under-served users or categories based on low accuracy or group disadvantage. For mainstreamness bias, weighting each user's cross-entropy by a learned cost (decreasing in mainstreamness) achieved >40% reduction in the group accuracy gap with negligible global accuracy loss (Li, 2023).
• Exposure-aware reward optimization: In online/interactive settings, adjust reward assignment and updates to penalize repeated exposure and promote under-recommended items—formally, using nonuniform position-based weights, merit-normalized scores, and explicit fairness objectives (Mansoury et al., 2024).

6. Theoretical Guarantees and Guidelines for Use

For online and bandit-based recommenders, regret analyses show that introducing exposure-aware rewards (including bias constraints) preserves the optimal $$\mathrm{RMBR}_B@K = \frac{1}{|Q|} \sum_{s\in Q} \mathbf{1}\bigl(\exists\,i\in B(s)\,\text{s.t.}\,\mathrm{rank}_s(i)\le K\bigr)$$5 regret scaling, thus fairness/anti-bias interventions do not fundamentally limit learning efficiency—provided penalty hyperparameters are properly controlled (Mansoury et al., 2024).

Best practices for RMBR application include:

• Normalizing by group- or item-type prevalence ($$\mathrm{RMBR}_B@K = \frac{1}{|Q|} \sum_{s\in Q} \mathbf{1}\bigl(\exists\,i\in B(s)\,\text{s.t.}\,\mathrm{rank}_s(i)\le K\bigr)$$6, item "merit") to prevent artifacts from rare categories (Mansoury et al., 2019, Mansoury et al., 2024).
• Ensuring metrics are reported at comparable accuracy levels (e.g., fixing nDCG@K) for proper fairness evaluation.
• Reporting both binary and position-weighted RMBR, as these reflect distinct fairness goals (e.g., aggregate exposure vs. top-slot opportunity) (Mansoury et al., 2024).
• Including item-coverage and group-level disparity analysis for comprehensive auditability (Mansoury et al., 2019).
• Dynamic monitoring of RMBR over time in online contexts to verify that cumulative bias does not increase unboundedly (Mansoury et al., 2024).

RMBR thus provides a principled, extensible scaffold for quantifying, diagnosing, and addressing systematic biases in both static and online recommendation models. The approach is adaptable to emerging fairness concerns and is grounded in both theoretical regret bounds and empirical performance analysis.