Bradley-Terry Rankings for Recommender Systems Across Dataset Taxonomies

Abstract: The ranking of recommendation algorithms is a challenging problem since model performance is sensitive to dataset characteristics such as sparsity, sequential structure, and scale. This drives a demand for a proper methodology for fair comparison between algorithms. Naive aggregation of performance metrics (e.g., averaging NDCG over benchmarks) can yield misleading rankings, undermining practical selection. To address this problem, we introduce a novel, data-driven ranking methodology based on Bradley-Terry (BT) model. We demonstrate that the obtained ranking depends on key dataset statistics. Additionally, we propose a novel metric for evaluating ranking consistency and demonstrate robustness of our ranking to incomplete data. Finally, we introduce a dataset-specific methodology for ranking algorithms on unseen datasets without running the models, relying on extensions of the Bradley-Terry framework, including BT trees and BT models with covariates.

• The paper proposes a novel extension of the Bradley-Terry model by incorporating dataset covariates and non-parametric BT trees to improve ranking stability.
• It applies maximum likelihood and Bayesian MCMC-based inference to achieve robust probabilistic rankings that address missing data and heterogeneity.
• Results demonstrate that BT-based rankings outperform naive aggregation methods, ensuring reproducibility and reliable recommender system evaluation.

Bradley-Terry Rankings for Recommender Systems Across Dataset Taxonomies

Introduction

The study "Bradley-Terry Rankings for Recommender Systems Across Dataset Taxonomies" (2606.07492) introduces a robust methodology for ranking recommendation algorithms using the probabilistic Bradley-Terry (BT) model and its extensions. The motivation emerges from the heterogeneity of recommender systems benchmarks, where naive performance aggregation methods (such as averaging or summing metrics across datasets) yield rankings that are sensitive to dataset properties and can mislead practical model selection. The work argues for principled, probabilistic ranking incorporating pairwise comparisons, dataset-aware extensions, and a new measure for ranking stability, aiming to provide a more reliable foundation for algorithm selection and benchmarking in recommender systems.

BT Model Framework and Estimation

The core of the approach utilizes the classical Bradley-Terry model, which assigns each algorithm a non-negative strength parameter $p_i$ so that the probability that algorithm $i$ outperforms $j$ is $\frac{p_i}{p_i + p_j}$. Given the observed win matrix $W$, maximum likelihood estimation (MLE), Bayesian MCMC-based inference [wainer2023bayesianbt], and the rank centrality spectral method [oh2017rank] are all exploited to estimate these parameters. The methodology is adjusted for the recommender systems context by: (i) accounting for ties based on overlapping confidence intervals in per-dataset metric estimates; (ii) synthesizing listwise performance through the Plackett-Luce model; and (iii) generating probabilistic, rather than ordinal, model rankings to allow for uncertainty quantification.

Stability and Consistency of Rankings

To address instability induced by dataset incompleteness and missing values, the paper introduces the "transitive triplets" metric, computing the proportion of ordered algorithm triplets that maintain strict transitivity across datasets. This enables a granular assessment of both ranking validity and robustness to missing data, beyond correlations such as Kendall’s $\tau$. Strong numerical results demonstrate that BT-based rankings preserve a high triplet ratio even as the fraction of missing pairwise results increases, while naive aggregation methods rapidly degrade.

Figure 1: The BT-based ranking maintains high transitivity even as the fraction of missing metric entries grows, while naive methods lose consistency.

Figure 2: BT and Plackett-Luce model weights are robust under data sparsity, in contrast to aggregation baselines.

Dataset Taxonomies and Heterogeneous Leaderboards

The study systematically demonstrates that algorithm performance and induced rankings strongly depend on dataset-specific characteristics, such as sparsity, sequentiality, interaction counts, and user-item ratios. BT rankings are computed for contrasting dataset subgroups, uncovering clear context-dependencies:

• SASRec and GASATF dominate on sequential datasets but plummet in rank on non-sequential ones, where classic models (ALS, UltraGCN) excel.
• LightGCN is the most robust across taxonomies, but certain methods (e.g., Seq-KNN, User-KNN) see pronounced improvements under specific conditions (e.g., long user history, sparse setups).
• No algorithm demonstrates universal optimality; the strongest performers cluster differently across feature combinations.

Figure 3: Pairwise win probability matrix across all datasets: strong models consistently dominate lower-tier baselines.

These phenomena are visualized via pairwise win probability heatmaps, showing that performance clusters are highly dataset-dependent and that context-aware evaluation is critical for robust algorithm selection.

Extension: Dataset-Aware Ranking via Covariate Models

A central claim is that simple BT rankings, even if statistically grounded, do not suffice for new, previously unseen datasets. To address this, the study extends the BT framework with dataset covariate information, applying:

• Covariate-adjusted BT models: The strength parameter for each algorithm becomes a function of dataset features, fit via penalized likelihood with fusion penalties [schauberger2019btllasso].
• BT trees: Recursive partitioning identifies subpopulations of datasets with homogeneous relative rankings, enabling interpretable, covariate-dependent predictions [zeileis2008modelpartitioning].

BT trees reveal that sequentiality, number of users, and interactions drive splits in ranking structure:

Figure 4: BT tree splits reveal the primary axes (sequentiality, user/item counts) that distinguish contexts with different optimal algorithms.

Empirical Evaluation: Holdout Predictions

Systematic evaluation using holdout datasets confirms that covariate-augmented BT models and BT trees achieve superior top-1 accuracy in predicting the true best algorithm, especially when the training pool is limited. However, the classic global BT ranking suffices for recovery of a strong baseline set (top-$k$ overlap), making it highly effective for practical scenario-based model selection.

Theoretical and Practical Implications

The work provides substantial evidence that classical aggregation-based rankings are inadequate for fair and robust model comparison in recommendation settings with diverse dataset taxonomies. The BT-based approach ensures stability, transitivity, and uncertainty awareness, particularly under missing data and significant heterogeneity. The extension to covariate-driven BT models and non-parametric partitioning enables effective zero-shot baseline selection for new data scenarios—an essential capability for scalable, trustworthy meta-benchmarking in recommender systems.

Conclusion

This study establishes the Bradley-Terry framework, with covariate-augmented and non-parametric extensions, as a theoretically sound and empirically validated methodology for ranking recommender algorithms across diverse datasets. The analysis reveals the necessity of dataset-aware evaluation and probabilistic aggregation for both research benchmarking and practical deployment. The resulting tools, metrics, and dataset-taxonomy analysis inform reproducibility efforts and future developments in robust offline evaluation protocols for recommenders.