MARS: Magnitude-Aware Rank Statistics

Abstract: Comprehensive evaluation of machine learning models is the key to make sure that they perform as robustly and consistently as desired. In order to summarize the experimental results and pick a winner, Critical Difference (CD) diagrams are used. Standard CD diagrams rely on discrete ranks, discarding the magnitude of performance gaps between models, raising an issue which we call magnitude-blindness. In order to address this issue, we propose Magnitude-Aware Rank Statistics (MARS) that incorporates a relative margin coefficient as a weight for the discrete ranks. This coefficient scales ranks based on the distance between the best and worst performers, with a dynamic projection to handle boundary cases. Followed by the calculation of a CD value, MARS results in a more realistic statistical representation of differences of model performances and more insights on how methods actually perform in vast and extensive experimental settings.

• The paper introduces MARS, a magnitude-aware rank method that weights performance by normalized within-dataset gaps while preserving cross-dataset comparability.
• MARS uses rescaled critical differences, permutation testing, and Wilcoxon-Holm comparisons to distinguish robust methods from models that win often by negligible margins or fail catastrophically.
• Synthetic experiments show MARS can separate stable, superior methods from noisy or volatile competitors, but real-world validation and guidance on metric choice remain open challenges.

Motivation: magnitude-blindness in rank-based evaluation

The standard pipeline for comparing $k$ methods over $N$ datasets—Friedman's test followed by Nemenyi or Wilcoxon-Holm post-hoc analyses, visualized via Critical Difference (CD) diagrams—rests on discrete ordinal ranks. The authors identify a structural weakness of this convention, which they term magnitude-blindness: rank aggregation discards the cardinal size of performance gaps, so a marginal victory of 0.01% is treated identically to a dominance of 50%. This is a strong claim with practical consequences: models that win frequently by negligible margins (the "Winner's Curse") are statistically rewarded over robust models that win less often but never fail catastrophically. The paper motivates this concern in high-stakes settings such as medical diagnostics and autonomous driving, as well as in LLM benchmarking, where frequent marginal wins can mask catastrophic failures on complex tasks.

At the same time, the authors acknowledge the reason ranks exist in the first place: raw performance values are not comparable across heterogeneous datasets. Any remedy must therefore preserve the ordinal stability of rank aggregation while reintroducing sensitivity to normalized performance gaps.

The MARS method

MARS converts discrete ranks into continuous, magnitude-aware scores through a relative margin coefficient. For method $j$ on dataset $i$ with metric value $y_{i,j}$, the weight is defined as:

$$w_{i,j} = \frac{y_{i,\max} - y_{i,\min}}{y_{i,j} - y_{i,\min}}$$

so that methods performing close to the worst performer receive large penalties, while the best performer receives weight 1. To handle boundary cases where $y_{i,j} = y_{i,\min}$ (division by zero), the method applies a dynamic penalty derived from the largest observed gap among the remaining weights, ensuring the worst performer is penalized relative to the most significant performance difference on that dataset. The final score $\hat{R}_j$ averages the weighted ranks across datasets.

Because these scores lie in $[1, \infty)$ rather than $[1, k]$, the critical difference must be rescaled accordingly:

$N$0

where the ratio of the empirical standard deviation of the weighted ranks to the theoretical standard deviation of uniform integer ranks adapts the significance threshold to the volatility of performance gaps. Global significance is assessed with a non-parametric permutation test ($N$1 permutations) on the variance of the weighted rank scores, complemented by Wilcoxon-Holm pairwise testing for $N$2-values. Notably, the authors concede that because metric values enter the global test, it becomes more sensitive to the number of observations than standard rank statistics; they therefore recommend treating the permutation $N$3-value as complementary information and relying on Wilcoxon-Holm for strict rejection decisions.

Empirical scenarios

The paper validates MARS on six synthetic scenarios constructed to expose failure modes of standard CD analysis. A representative summary:

Scenario	Standard analysis outcome	MARS outcome
Magnitude of performance	A and B indistinguishable	A isolated as superior
Consistency/robustness	Rewards A's 75% win frequency	Favors robust B; groups A with C
Stability vs. volatility	Groups steady and volatile methods together	Separates them
Noisy superiority	Declares B leader	Reveals B's inflated rank from noise-level wins
Survivor (edge case)	Prefers B and C	Isolates A as clear winner

In each case, the standard pipeline either merges methods into spurious significance cliques or elevates models whose wins are negligible while their losses are severe. In the "noisy superiority" scenario, for instance, Method B wins by a margin of 0.0001 on 30 of 40 datasets yet collapses to 0.15 accuracy elsewhere; standard CD diagrams crown B as leader, whereas MARS identifies A as statistically superior.

Scenario 6, a realistic competition among eight methods across 40 datasets with layered Gaussian noise ($N$4 per dataset, $N$5 per model), illustrates a subtler property. MARS recovers the same ranking hierarchy as standard analysis but draws different significance boundaries: Methods A and D (base accuracies 0.92 vs. 0.85) are not significantly different under MARS despite a large rank-score gap (1.53 vs. 5.25), reflecting the possibility of rank swaps under the noise model. Conversely, E and G, in an analogous pairwise situation but closer to the worst performer, remain separated—the penalty intensifies near the bottom of the performance range.

Limitations and open questions

The paper is candid about several constraints. First, MARS remains sensitive to ranking frequency: more observations yield more certain results, so MARS does not eliminate the dependence on sample size, though the scenario analyses suggest it extracts more information from limited observations than unweighted ranks. Second, like standard rank statistics, MARS remains susceptible to dataset selection; choosing datasets at comparable difficulty stays the researcher's responsibility. Third, all empirical evidence is synthetic—the six scenarios are deliberately constructed edge cases—and no evaluation on real benchmark suites (e.g., published classifier comparisons or LLM leaderboards) is reported, leaving open whether the qualitative gains transfer to natural experimental distributions. Fourth, the weighting scheme assumes higher-is-better metrics and a meaningful within-dataset normalization; behavior under error-rate-style metrics or heavily skewed performance distributions is not analyzed. Finally, the question of whether MARS should replace or merely accompany standard rank statistics is explicitly left to the researcher, with no formal guidance on when each is preferable.

Conclusion

MARS augments classical rank statistics with a relative-margin weighting that preserves the cross-dataset comparability of ranks while restoring sensitivity to the magnitude of performance differences, paired with an empirically calibrated critical difference and permutation-based global testing. Through targeted synthetic scenarios, the paper demonstrates concrete cases where standard Friedman-Nemenyi and Wilcoxon-Holm pipelines produce misleading conclusions—rewarding noisy superiority, masking catastrophic failures, and conflating volatile with stable methods—that MARS resolves. Its principal open issues are validation on real-world benchmarks and principled guidance on when magnitude-awareness should override the conservatism of traditional post-hoc tests.