Automatic Pairwise Judge

• Automatic pairwise judge is a system that transforms raw pairwise comparisons into an interpretable continuous scale using probabilistic models like Thurstone Case V.
• It employs maximum likelihood estimation and bootstrap methods to derive reliable quality rankings with confidence intervals and significance tests.
• Applications include image quality assessment and multi-criteria decision-making, offering automated, statistically rigorous analysis of subjective data.

An automatic pairwise judge is a system—algorithmic or software-based—that processes collections of pairwise comparative judgments between objects, conditions, or candidate solutions and systematically aggregates them into a latent measurement scale or global ranking. The primary goal is to transform raw subjective or empirical pairwise data into an interpretable, continuous evaluation metric that can guide decisions, scientific analysis, or quality assessment, often with robust statistical guarantees. Core methodologies leverage probabilistic modeling (notably the Thurstone Case V model), maximum likelihood estimation, and advanced statistical tools for confidence estimation and hypothesis testing. Automatic pairwise judging systems have become essential across image quality assessment, experimental psychology, multi-criteria decision-making, and any domain where ordinal or absolute scoring is impractical or unreliable.

1. Methodological Foundations of Pairwise Judging

The fundamental experimental unit is a pairwise trial in which an observer chooses the “better” of two presented conditions under a prescribed criterion. The aggregate data are cast into a comparison matrix $C$, where each $c_{ij}$ records the frequency with which condition $O_i$ is preferred to $O_j$. The empirical probability of preference, $\hat{p}_{ij}$, is then simply

$\hat{p}_{ij} = \frac{c_{ij}}{c_{ij} + c_{ji}}$

To infer a continuous scale, the response model typically adopts the Thurstone Case V model, positing that each condition’s perceived quality is an independent, normally distributed latent variable (shared variance $\sigma^2$). The crucial relationship is

$q_i - q_j = \sigma_{ij} \, \Phi^{-1}(\hat{p}_{ij})$

where $\Phi^{-1}$ is the probit (inverse cumulative normal distribution) and $\sigma_{ij} = \sigma \sqrt{2}$. Thus, ordinal pairwise counts are mapped to continuous distances (“just-objectionable-differences,” or JODs), which form the backbone of quantitative quality scales (Perez-Ortiz et al., 2017).

2. Statistical Inference and Scaling Algorithms

Direct least squares regression can be used to fit the quality scores $\hat{q}$ to the pairwise distances. However, least squares is unreliable under unanimous choices, where observed distances can be infinite. The maximum likelihood estimation (MLE) framework supersedes least squares by defining, for each comparison, a binomial likelihood: $$\mathcal{L}(q_i - q_j | c_{ij}, n_{ij}) = \binom{n_{ij}}{c_{ij}} [\Phi((q_i - q_j)/\sigma_{ij})]^{c_{ij}} [1 - \Phi((q_i - q_j)/\sigma_{ij})]^{n_{ij} - c_{ij}}$$ The full likelihood is the product over all observed pairs. The numerical maximization yields the latent quality vector, naturally accommodating the certainty (number of comparisons) and resolving estimation in the presence of unanimous responses.

Advanced pipelines also provide:

• Outlier analysis: Observer contributions deviating substantially in likelihood can be flagged (e.g., via functions such as pw_outlier_analysis). Visualization and likelihood-based scoring help diagnose and remove anomalous raters.
• Confidence interval estimation: Nonparametric bootstrap resampling (sampling observer judgments with replacement) generates empirical confidence intervals for quality scores.
• Significance testing: Covariance structure between estimated scores allows for formal two-tailed hypothesis testing, with the comparison-difference variance given by $v_{ij} = \Sigma_{ii} + \Sigma_{jj} - 2\Sigma_{ij}$.
• Finite distance prior: Introduction of a prior regularizes the estimation under low-observer or highly consistent (unanimous) regimes, mitigating estimation bias.

3. Automated Software Implementation

Practical deployment is enabled through dedicated software—exemplified by the public Matlab toolbox presented in (Perez-Ortiz et al., 2017). This system ingests tabulated experiment data (CSV-formatted observer/session/condition/selection), automatically constructs the pairwise comparison matrix, applies the MLE (and least squares) scaling routines, and outputs ranked scores with associated statistics.

Feature summary:

Feature	Description	Implementation Example
Outlier Detection	Likelihood/plot-based anomaly detection	pw_outlier_analysis
Confidence Interval Estimation	Bootstrap resampling over observers	Empirical JOD intervals
Significance Testing	Error covariance–based two-tailed tests	pw_plot_ranking_triangles
Prior for Estimation Error Control	Finite distance prior to reduce bias	Applied when $n$ is low

This encapsulation yields a nearly “automatic” analysis pipeline that addresses unbalanced designs, incomplete pairings, and experimental ties, offering robust, reproducible statistical analysis for subjective experiments.

4. Representative Applications

While methodologically widely applicable, the automatic pairwise judge is detailed primarily in the context of image quality assessment. For example, in a video tone-mapping paper, observer-wise paired selection data (Observer, Session, Scene, Condition1, Condition2, Selection) are processed to output a continuous JOD scale. Bar plots visualize estimated scores with confidence intervals, indicating, for instance, that $1$ JOD roughly equates to a $75\%$ preference rate between two stimuli.

Further, triangle-plot visualizations delineate statistically significant quality differences among conditions, solid or dashed lines indicating the outcome of pairwise significance tests.

The approach yields high-sensitivity, interpretable scales for perceptual experiments and can be extended to other domains where direct rating is unreliable or ordinal data dominates.

5. Strengths and Constraints

Automatic pairwise judge systems offer several methodological advantages:

• Cognitive simplicity: Participants make a direct binary choice, lessening rater effort and error.
• Cardinal measurement: Outputs both ranking and an interpretable difference metric (JODs), quantifying the subjective gap.
• Robustness: MLE and prior incorporation enable the method to handle unanimity, incomplete data, and observer anomalies.
• Statistical rigor: Outlier detection, confidence estimates, and formal hypothesis testing are integral.

However, key limitations are inherent to the foundational assumptions:

• Thurstone Case V model: A single, unidimensional Gaussian observer model may not represent complex dependencies across raters or repeated measurements.
• Large scale differences: For pairwise distances exceeding about $2$ JODs, estimation fidelity degrades—necessitating alternative scaling.
• Ties and bias: The system includes ad hoc modeling of “no-preference” judgments (ties), but without specialized calibration, these can induce underestimation bias.
• Manual intervention: Outlier detection, while largely automated, still benefits from expert review, especially near detection thresholds.

6. Future Perspectives and Generalizability

The described methodologies—especially when integrated into automated analysis tools—form a foundational baseline for constructing reliable, scalable, and interpretable judging systems in behavioral research, perceptual quality assessment, decision science, and related fields. Ongoing limitations regarding model flexibility and handling large qualitative gaps suggest areas for methodological innovation, such as multidimensional scaling or advanced mixture models.

The automation of data ingestion, handling of experimental irregularities, and statistical reporting positions the automatic pairwise judge paradigm as a best-practice analytical tool for the synthesis of ordinal comparative data into robust, reproducible scientific measurements.