Score Range Bias: Analysis & Mitigation

• Score Range Bias is a systematic distortion caused by score distribution properties that undermines fair model comparisons in various evaluation settings.
• Analytical proofs and simulation studies reveal that the bias increases with factors like cluster size and scoring scales, affecting metrics such as H-score and factor analysis outcomes.
• Mitigation strategies, including score normalization, contrastive decoding, and bias-adjusted statistical models, improve reliability in both automated and human evaluations.

Score range bias refers to the systematic distortion introduced into evaluations, rankings, or model outputs as a result of how numerical scores are distributed, bounded, or assigned within a specified range. This phenomenon arises across a variety of machine learning and statistical contexts where the properties of scoring functions, evaluation scales, or their dependence on extraneous parameters (such as bicluster size, measurement error, or predefined Likert ratings) can cause spurious, misleading, or unfair results. Score range bias affects model selection, comparison of clusters, density estimation, LLM judgment, and human ratings, and is now a critical topic in high-stakes and automated evaluation scenarios.

1. Mathematical Characterization of Score Range Bias

Score range bias frequently manifests as a systematic dependency of the evaluation score on factors extrinsic to the intended signal, leading to misleading comparisons and invalid inferences.

• Biclustering and H-score: In biclustering, the H-score quantifies within-bicluster homogeneity. The paper shows the average H-score increases monotonically with the number of rows (or columns) based solely on bicluster size, not on signal strength. For an additive-noise model $a_{ij} = \mu + \alpha_i + \beta_j + \epsilon_{ij}$, the expected H-score for a bicluster with $n$ rows, $\bar{H}_n$, obeys a recurrence:

$\bar{H}_{n+1} = \bar{H}_n \cdot \frac{n^2}{n^2-1}$

This renders raw H-scores non-comparable across bicluster sizes (Iorio et al., 2019).

• Ordered Score Functions in Factor Analysis: Regression factor scores maximize determinacy but fail to preserve latent inter-factor correlations. The regression predictor’s correlation matrix overestimates the model’s true correlations, introducing bias in the effective range of factor scores depending on the type of predictor used (Beauducel et al., 2023).
• Density Estimation: Classical kernel density estimators (KDE) suffer from leading order bias $O(h^4)$ because their smoothing fails to account for local gradient changes; this can be construed as a ‘score range bias’ whereby the estimation error grows due to under-correction for local density structure (Epstein et al., 27 Apr 2025).
• LLM-as-a-Judge: The numerical outputs of LLMs acting as scoring judges depend sharply on the range and labeling of the scoring scale (e.g., 0–4, 1–5, 2–6, 3–7), with observed preferences toward specific values, independent of content quality. This leads to unstable and unreliable automated evaluation (Fujinuma, 21 Oct 2025).

2. Analytical and Simulation Evidence

Score range bias is supported by analytical results and empirical simulation:

• Analytical Proofs: In biclustering, Theorem 1 analytically derives the bias formula, showing its independence from noise or signal. The correction factor for H-score normalization across bicluster sizes is a product of $(i^2/(i^2-1))$ over the size difference (Iorio et al., 2019).
• Simulation Studies: Simulations consistently show that as the bicluster size increases, the average H-score increases exactly as predicted theoretically. For small sizes, the bias is pronounced (e.g., simulation yields $r_{2,3} \approx 1.33$ matching $2^2/(2^2-1)$) and diminishes for larger $n$.
• Statistical Genetics Methods in Data Scoring: Investigation of example difficulty scores shows that variance across scores is reduced by averaging across training runs, but systematic variation due to architectural inductive bias remains, affecting the range of scores assigned to the same data (Kwok et al., 2024).

3. Correction and Mitigation Strategies

Various correction mechanisms have been developed:

• H-score Normalization: Adjust bicluster H-scores by dividing by the analytically derived correction factor, yielding “corrected” scores comparable across cluster sizes. Thresholds and selection criteria should be calibrated using corrected, not raw, H-scores (Iorio et al., 2019).
• Contrastive Decoding for LLM Judges: Mitigation of LLM scoring range bias is achieved by contrastive decoding, adjusting the output logits as

$$\log p_\mathrm{main} - \lambda \log p_\mathrm{asst}$$

with a carefully chosen $\lambda$ and temperature scaling for the assistant model. This removes shared bias directions between models of the same family and yields up to 11.3% improvement in correlation with human judgments across varied score ranges (Fujinuma, 21 Oct 2025).

• Score Function Adjustment in Statistical Models: Bias-reducing adjustments to the score function, as in the Dirichlet parameter estimation work, reduce the mean bias from $O(n^{-1})$ to $O(n^{-2})$. For example, Firth’s bias-reducing adjustment is implemented as

$\tilde{U}(\alpha) = U(\alpha) + A^*(\alpha)$

where $A^*(\alpha)$ is a bias-correcting term computed from the expected information and higher moments (Gioia et al., 2021).

• Item-Level Statistical Modeling: In autograder and LLM-judge evaluation, Bayesian generalized linear models explicitly include grader/item interactions and cutpoint estimation, providing measurement of where the scoring range is stretched or compressed and quantifying systematic bias in specific ranges (Dubois et al., 4 Jul 2025).

4. Consequences for Model Selection, Evaluation, and Fairness

Score range bias has direct and substantial implications:

• Model Comparison and Selection: In biclustering and ranking tasks, algorithms that rely on uncorrected scores are structurally biased to select smaller clusters or candidates with compressed latent signal, resulting in suboptimal, misleading interpretations (Iorio et al., 2019, Boehmer et al., 2023).
• Automated Judging with LLMs: The presence of scoring range bias in LLM-as-a-judge settings undermines the search for a universally optimal score range and disrupts the reliability of automatic evaluation, with different judge models favoring different positions within the range based on prompt formatting or rubric labeling (Fujinuma, 21 Oct 2025, Li et al., 27 Jun 2025).
• Human Rating and MOS: In human listening tests for speech synthesis, range-equalizing bias (“rubber ruler” effect) means that presented sample quality context determines how raters use the scale, so MOS ratings cannot be interpreted as context-independent measures of quality; absolute ratings become unreliable when the system range presented is restricted (Cooper et al., 2023).
• Statistical Inference: In finite mixture models, the appearance of mixture probabilities in $[0,1]$ for group allocation induces a negative bias in the score function; thus, MLE may become inconsistent, and inferential procedures relying on the unbiasedness of scores fail (Labouriau, 2020).
• Fairness and Representation: Voting rules and subset selection with narrow score ranges (e.g., SNTV) amplify bias and require exponentially more rankings to recover unbiased latent quality under representational constraints, compared to rules with a broader score range (e.g., Borda) (Boehmer et al., 2023).

5. Limitations and Open Challenges

Several limitations and challenges associated with existing approaches are prominent:

• Metric Dependency: Many corrections and diagnostic tools are score-function-specific. A correction effective for the H-score may not transfer to other biclustering coherence scores or to clustering in very high dimensions.
• Incomplete Downstream Correlation: While intrinsic bias scores (e.g., SAME) may correlate with downstream bias, correlations are often only moderate; other sources of bias outside the measured score distribution can strongly influence real-world outcomes (Schröder et al., 2022).
• Context Sensitivity and Robustness: In human evaluations and automated LLM scoring, practical implementations are sensitive to test design, prompt construction, and even to the labeling convention of scores (Li et al., 27 Jun 2025, Cooper et al., 2023). This context sensitivity challenges the generalizability of corrected scores.
• Trade-offs in Correction: In factor analysis, transforming regression scores into correlation-preserving scores eliminates score range bias at the cost of a slight reduction in factor score determinacy; the optimal trade-off is application-specific and needs to be accounted for in subsequent analyses (Beauducel et al., 2023).
• Computational Overhead: Diagnostics such as B-score (computed from multi-turn LLM outputs) or ensemble and averaging approaches require repeated sampling or paired model runs, resulting in computational burdens inappropriate for some real-time scenarios (Vo et al., 24 May 2025).

6. Implications for Practice and Future Directions

A robust understanding and mitigation of score range bias are essential for the validity of both human and automated evaluation in modern machine learning pipelines. Practitioners are advised to:

• Employ normalized, bias-corrected scores when comparing clusters/subsets of different sizes or evaluating outputs on shifted scales.
• Use explicitly designed prompt templates and calibration methods for LLM-as-a-judge systems, incorporating mitigation strategies such as contrastive decoding or ensemble corrections.
• Quantify the impact of measurement error and score scaling by integrating model-based corrections, particularly in high-stakes or regulatory settings (e.g., education, fairness auditing).
• Exercise caution when interpreting raw evaluation scores, especially in benchmarking or reporting “human-level” performance.
• Support further research into universal, context-independent scoring functions, improved calibration procedures, and evaluation pipeline redesigns that are robust to score range artifacts.

Future work is expected to focus on extending debiasing techniques to more complex or high-dimensional score functions, on unifying intrinsic and downstream bias measurements in generalized evaluation frameworks, and on the development of scalable and efficient bias diagnostics for use in continuous or production ML evaluation systems.