Pairwise Comparison Procedures

• Pairwise comparison procedures are a set of methods that generate rankings or weight vectors from complete or partial pairwise judgments, emphasizing consistency and accuracy.
• They utilize techniques such as the eigenvector, geometric mean, and least squares methods to derive precise metrics from comparison matrices.
• Recent developments integrate statistical scaling, optimization, and robust error handling to manage uncertainty, missing data, and potential manipulations.

Pairwise comparison procedures constitute a class of methods that extract a ranking or a set of weights for a collection of items, alternatives, or populations based on the complete or partial comparison of every pair. These methods are foundational in statistics, psychometrics, decision theory, and algorithmic evaluation—serving both as direct inference tools (as in subjective assessments) and as components of complex aggregation or testing procedures. The theory and methodology cover precise issues of consistency and discrepancy, choice of aggregation technique, statistical scaling, advanced optimization, treatment of uncertainty and missing data, and vulnerability to manipulation.

1. Mathematical Frameworks and Consistency

The standard mathematical model for pairwise comparisons is the pairwise comparison matrix (PC matrix), often square, with elements $a_{ij}$ encoding the preference or dominance of item $i$ over $j$. In ratio-scale (multiplicative) models, $a_{ij} > 0$ and the reciprocal $a_{ji} = 1/a_{ij}$ holds. The comparative judgments are consistent if and only if there exists a positive weight vector $w$ such that $a_{ij} = w_i/w_j$, equivalently satisfying the transitive closure $a_{ik} = a_{ij} a_{jk}$ for all $i,j,k$ (Koczkodaj et al., 2016). The conversion to and from additive forms—$b_{ij} = \log a_{ij} = u_i - u_j$—enables analytical techniques such as least squares projection for finding the nearest consistent matrix (i.e., “consistencization”). In mathematical terms, this is the minimizer of

$$\min_{u \in \mathbb{R}^n} \sum_{i, j} [b_{ij} - (u_i - u_j)]^2,$$

which projects $b$ onto the additive subspace corresponding to the weight simplex; the solution $u^*$ then yields $w_i = \exp(u^*_i)$ (Koczkodaj et al., 2016).

2. Priority Derivation and Discrepancy

Several methods are established for deriving weight vectors from a PC matrix:

• Eigenvector Method: Solve $A w = \lambda_{\max} w$ and normalize $w$ (Kułakowski, 2013, Kułakowski, 2014, Krivulin et al., 17 Jan 2024).
• Geometric Mean Method: For each item, compute $w_i = \bigl(\prod_j a_{ij}\bigr)^{1/n}$ and normalize (Herman et al., 2015, Krivulin et al., 17 Jan 2024).
• Least Squares and Chi-Square Methods: Minimize deviations between $a_{ij}$ and the ratios $w_i/w_j$ in the (logarithmically) transformed space (Kułakowski, 2014).

Empirical comparison shows that for “not-so-inconsistent” matrices, geometric mean and principal eigenvector solutions differ little ("maximum average deviation" $< 0.0036$ in Tchebychev metric), with the geometric mean slightly better for Euclidean error and the eigenvector slightly better for maximum error (Herman et al., 2015, Krivulin et al., 17 Jan 2024).

The discrepancy between input judgments and the derived ranking is quantified via local parameters $\epsilon(i,j) = a_{ji} \cdot (w_i/w_j)$ (should be 1 in case of perfect consistency) and aggregate measures such as the global ranking discrepancy $\mathcal{D}(A,w) = \max_{i,j} \mathcal{E}(i,j)$, with $$\mathcal{E}(i,j) = \max\{\epsilon(i,j)-1, 1/\epsilon(i,j)-1\}$$ (Kułakowski, 2013, Kułakowski, 2014). Output properties, such as regularity (zero discrepancy if the input is consistent) and sensitivity to inconsistency (output discrepancy decreases with reduced input inconsistency), are formalized (Kułakowski, 2014).

Conditions of order preservation (COP)—the requirement that derived weights respect both stated order ($a_{ij} > 1 \implies w_i > w_j$) and intensity—are only ensured when both input inconsistency and output discrepancy are below explicit thresholds (Kułakowski, 2013, Kułakowski, 2014).

3. Statistical Scaling and Subjective Evaluation

For subjective or perceptual evaluation, pairwise comparisons convert sets of comparative judgments into calibrated quality scores. The foundational model is Thurstone Case V, where the probability of $i$ beating $j$ is $\Phi\left(\frac{q_i - q_j}{\sigma_{ij}}\right)$, and the observed win probabilities are inverted to compute scale distances:

$q_i - q_j = \sigma_{ij} \Phi^{-1}(P(r_i > r_j))$

(Perez-Ortiz et al., 2017). Maximum likelihood estimation, often with a finite distance prior, handles statistical uncertainty and unanimous responses. Probabilistic models such as Bradley–Terry are employed, with likelihood functions of the form:

$$L(q_i - q_j | c_{ij}, n_{ij}) = \binom{n_{ij}}{c_{ij}} \Phi^{c_{ij}}\Bigl(\frac{q_i - q_j}{\sigma_{ij}}\Bigr) [1 - \Phi\Bigl(\frac{q_i - q_j}{\sigma_{ij}}\Bigr)]^{n_{ij} - c_{ij}}$$

The procedures are augmented by bootstrapping for confidence intervals, outlier analysis, and software toolboxes (Perez-Ortiz et al., 2017). For crowdsourced settings, Elo scoring systems have been used to aggregate pairwise outcomes, with updates after each comparison and demonstrated reduction in bias and error compared to majority voting, at a comparison cost scaling as $O(N\log N)$ for $N$ items (Narimanzadeh et al., 2023).

4. Consistency Indices, Interval, and Random Models

Consistency indices (e.g., Saaty's CI: $\mathcal{S}(A) = \frac{\lambda_{\max}-n}{n-1}$; Koczkodaj's $\mathcal{K}$ based on triad deviations) serve both to reject or revise input data and to bound discrepancies in output (Kułakowski, 2013, Kułakowski, 2014).

Interval-valued methods generalize the theory to interval pairwise comparison matrices (IPCMs). Each comparison is modeled as an interval $[a_{ij}^-, a_{ij}^+]$ rather than a single value, with Abelian linearly ordered group structure, generalizing operation, reciprocity, and consistency conditions. Metrics for consistency and indeterminacy are defined in this group-theoretic context, such as

$$I_{[\mathcal{G}]}(\tilde{A}) = \left(\bigodot_{i<j<k} d_{[\mathcal{G}]}(\tilde{a}_{ijk},\tilde{a}_{ikj})\right)^{1/|T|}, \qquad \Delta_{[\mathcal{G}]}(\tilde{A}) = \left(\bigodot_{i\neq j}\delta(\tilde{a}_{ij})\right)^{1/{n(n-1)}}$$

with distance $d$, and all previous operations as special cases (multiplicative, additive, or fuzzy) (Cavallo et al., 2017).

Random PC matrices permit each entry to be random. Stochastic consistency, reciprocity, and total inconsistency indices are the expectation of their deterministic analogs, and procedures such as optimal transport (Wasserstein distance minimization) and expectation functionals extend the notion of “nearest consistent matrix” to the probability-measure setting (Magnot, 2023).

5. Algorithmic and Optimization Approaches

Tropical optimization provides effective tools for log-Chebyshev (max-norm) consistency correction, leading to solution forms such as $x = B_{\mu}^* u$, where $B_\mu$ is a normalized and symmetrized version of the input and $B_\mu^*$ is its Kleene star. This unifies both multiplicative and additive comparison scales and underpins tropical versions of methods such as analytic hierarchy process (AHP) (Krivulin, 2015, Krivulin et al., 17 Jan 2024).

Orthogonalization with respect to the Frobenius or generalized Frobenius inner product allows for efficient projection of the log-transformed matrix onto the (linear) subspace of consistent matrices, with the geometric mean method as a special case when the Frobenius product is used. The method generalizes naturally to weighted inner products to reflect reliability or importance of assessments (Benitez et al., 18 Mar 2024, Koczkodaj et al., 2020). The choice of inner product is significant: different metrics yield different approximations and derived weight vectors (Koczkodaj et al., 2020).

For incomplete matrices, lexicographically optimal completion prioritizes minimizing the maximal local (triad) inconsistency, followed by the next, ensuring ordinal consistency that is not guaranteed by CR/GCI-optimal completions for the eigenvector or LLSM (Csató, 2023).

Recent developments in active sampling and ranking for subjective evaluation explore Bayesian or information-theoretic sampling, Swiss tournaments, tree-based schedules, and the MST-based “Sort-MST” approach, which builds minimum spanning trees from Elo-score–ranked pairs to select the most informative and balanced comparisons. This approach converges rapidly, is computationally less demanding than full Bayesian active sampling, and achieves state-of-the-art ranking accuracy (Webb et al., 25 Aug 2025).

6. Extensions, Limitations, and Practical Considerations

Pairwise comparison procedures extend to binary-only judgments (“simple pairwise comparison”), where weights are fixed solely by the number of items, and the increments in weights are uniform $2/(k(k-1))$ (for $k$ criteria), making the scale robust to subjective variations (Lörcks, 2020). In majority voting on graphs (majority domination), pairwise comparison methods underlie heuristics with explicit error and convergence bounds for structured graphs (Shushko et al., 10 Jun 2025).

Practitioners should be aware of vulnerabilities: manipulation is possible through iterative orthogonal projections to force ties or promote a particular alternative (“greedy” and “bubble” manipulation algorithms). Such attacks are not mitigated by high input inconsistency; each manipulation can lower the ranking stability and ease subsequent manipulative moves, suggesting the need for alternative detection metrics (Szybowski et al., 21 Mar 2024).

Strict ranking (no ties) requires tailored conditions (the "R-condition") and minimization over the non-tied locus, as standard consistencization can destroy injectivity of the final ranking (Magnot, 11 Dec 2024). Moreover, the interval property is central in multiple hypothesis testing involving pairwise comparisons: residual-based stepwise procedures ensure monotonicity, convexity, and avoid reversals, assumptions violated by naive step-up/step-down methods (Cohen et al., 2012).

---

Table: Summary of Leading Methods for Priority Derivation

Method	Core Formula / Approach	Consistency Correction
Principal Eigenvector	$A w = \lambda_{\max} w$, normalize $w$	Sensitivity to input inconsistency
Geometric Mean	$w_i = (\prod_j a_{ij})^{1/n}$, normalize	Solution coincides with Frobenius projection
Least Squares	Minimize $\sum_{ij} [b_{ij} - (u_i-u_j)]^2$	Log-space projection to consistent subspace
Log-Chebyshev/Tropical	Minimize $\max_{ij} |\log a_{ij} - \log(w_i/w_j)|$	Kleene star method for optimal correction
Lexicographic Completion	Iteratively minimize maximal triad inconsistency	Guarantees ordinal consistency

7. Impact, Domains, and Directions

Pairwise comparison procedures are core to applied statistics, machine learning (especially in subjective or preference labeling), AHP-based decision support, voting, and aggregation problems. The interplay between model choice, inconsistency management, sampling/survey design, statistical inference, and robustness to manipulation continues to motivate methodological research (Kułakowski, 2013, Kułakowski, 2014, Perez-Ortiz et al., 2017, Webb et al., 25 Aug 2025). Extensions to interval and random frameworks, formal guarantees for ordinal preservation, scalable optimization, and computationally efficient algorithms for large or uncertain data remain central directions. Practitioners leveraging these techniques must balance computational efficiency, statistical reliability, consistency, and resistance to both noise and strategic manipulation.