---
title: 'EZ-Sort: Efficient Pairwise Ranking'
url: https://www.emergentmind.com/topics/ez-sort
type: topic
---

# EZ-Sort: Efficient Pairwise Ranking

Searching arXiv for the specified paper and a few closely related references to ground the article.
arxiv_search(query="2508.21550", max_results=5)
EZ-Sort is a human-in-the-loop pairwise ranking method that combines zero-shot CLIP-based pre-ordering, bucket-aware Elo initialization, and uncertainty-guided MergeSort to reduce the annotation burden of subjective or visually ambiguous sorting tasks. It is designed for settings in which pairwise comparison is more reliable than absolute rating or fixed ordinal classification, but exhaustive pairwise annotation is prohibitively expensive because it requires \(O(n^2)\) comparisons, exactly \(n(n-1)/2\). EZ-Sort preserves the \(O(n \log n)\) comparison schedule of sorting-based active sampling while further reducing human effort by pre-ordering items semantically and automating low-uncertainty comparisons; on FGNET with \(n=100\), it used 467 human comparisons versus 4,950 for exhaustive pairwise comparison and 582 for a prior sorting-based method, while improving or maintaining inter-rater reliability [2508.21550].

## 1. Problem formulation and motivation

Pairwise comparison is often preferred in subjective annotation problems because human annotators are generally better at deciding whether item \(A\) should precede item \(B\) than at assigning calibrated absolute scores. The paper situates EZ-Sort in tasks such as perceptual quality assessment, face-age estimation, historical image chronology, and medical image triage, where labels may be ambiguous or weakly anchored and where direct classification can underperform pairwise judgment [2508.21550].

The central cost bottleneck is combinatorial. Exhaustive pairwise comparison over \(n\) items requires \(n(n-1)/2\) judgments. For \(n=100\), this is 4,950 comparisons. Sorting-based active sampling reduces this to \(O(n \log n)\) by embedding comparisons inside a comparison-based sort such as MergeSort; for \(n=100\), the paper gives the canonical order-of-magnitude estimate \(100 \times \log_2(100) \approx 664\) comparisons. EZ-Sort operates within this sorting regime rather than replacing it. A common misunderstanding is that the method changes the asymptotic comparison complexity; it does not. Its contribution is to reduce *human* annotation within the \(O(n \log n)\) schedule by deciding easy comparisons automatically and escalating only uncertain comparisons to human annotators [2508.21550].

This design choice is consequential because the comparison budget is not used uniformly. Human annotation is concentrated on ambiguous, cross-bucket, or low-confidence pairs, whereas high-confidence pairs are resolved automatically from current Elo ratings. This suggests a division of labor in which pretrained multimodal semantics provide a coarse prior and humans correct the cases most likely to be semantically fragile.

## 2. Three-stage architecture and decision logic

EZ-Sort consists of three core stages plus a decision layer for automated handling of trivial cases. First, it performs CLIP-based zero-shot hierarchical pre-ordering. Second, it initializes bucket-aware Elo ratings. Third, it runs an uncertainty-guided human-in-the-loop MergeSort. At comparison time, the method decides whether to query a human or to use the current Elo ordering directly [2508.21550].

The pre-ordering stage replaces flat \(k\)-way classification with a hierarchy of binary prompt decisions. At each level \(\ell = 1,2,\dots,d\), the method uses binary prompts \(P_\ell = \{p_{\ell,0}, p_{\ell,1}\}\) to split the current group into visually distinguishable subgroups. Prompt generation is described as automated, domain- and range-aware, and oriented toward observable visual features while avoiding contextual clues. For image \(i\), similarities \(s_{i,\ell,j}\) are computed against the two prompts, the winning bit is selected by \(\arg\max\), and confidence is computed by a softmax with temperature \(\tau = 0.1\). The resulting binary decisions are packed into a group index
\[
g_i = \sum_{\ell=1}^{d_i} c_{i,\ell} 2^{\ell-1},
\]
where \(d_i\) is the final depth for image \(i\), with adaptive stopping when further splits are unreliable; the reported typical depth is 3–5 levels [2508.21550].

These fine-grained groups are then merged into \(k\) coarse buckets using
\[
M(g) = \left\lfloor g \cdot \frac{k}{2^d} \right\rfloor,
\]
which the paper states preserves ordinal relationships. The reported bucket settings are \(k=5\) for FGNET and \(k=3\) for DHCI and EyePACS. The distinction reflects task-dependent noise and CLIP compatibility: FGNET is described as CLIP-friendly, while DHCI and EyePACS are noisier domains [2508.21550].

The second stage initializes Elo scores using bucket identity and CLIP confidence. Items receive ratings
\[
r_i = r_{\text{base}}(b_i) + \eta_i \bigl(1.5 - \text{conf}_i\bigr),
\]
where \(r_{\text{base}}(b)\) is linearly distributed in \([1200, 1800]\) across buckets, \(\eta_i \sim \mathcal{U}(-\delta_b, \delta_b)\), and \(\delta_b = 75\). Confidence weighting reduces motion for high-confidence items and allows more movement for low-confidence items. This initialization differs from uniform-prior ranking schemes because it encodes both a coarse CLIP-derived ordering and a measure of pre-ordering uncertainty [2508.21550].

The final stage runs MergeSort over the initially pre-ordered list. For each candidate comparison \((i,j)\), EZ-Sort computes an uncertainty score from the current Elo state and decides whether to solicit a human judgment. If the uncertainty is at least the current threshold \(\theta_t\), a human is queried; otherwise the pair is decided automatically by the sign of the Elo difference. Ties are resolved conservatively in favor of human input [2508.21550].

## 3. Formal model: CLIP similarity, Elo dynamics, and uncertainty routing

The method uses CLIP encoders \(f_{\text{img}}\) and \(f_{\text{txt}}\) to obtain image and text embeddings. For image \(I_i\) and prompt \(t\),
\[
e_{\text{img}}(i) = f_{\text{img}}(I_i), \qquad e_{\text{txt}}(t) = f_{\text{txt}}(t),
\]
and the similarity is cosine similarity,
\[
s(i,t) = \frac{\langle e_{\text{img}}(i), e_{\text{txt}}(t) \rangle}{\|e_{\text{img}}(i)\| \, \|e_{\text{txt}}(t)\|}.
\]
At each hierarchy level, the class bit is
\[
c_{i,\ell} = \arg\max_{j \in \{0,1\}} s_{i,\ell,j},
\]
and the confidence is computed by a temperature-scaled softmax with \(\tau = 0.1\) [2508.21550].

Pairwise ranking is modeled with standard Elo probabilities. For items \(i\) and \(j\) with ratings \(r_i\) and \(r_j\),
\[
E_i = \frac{1}{1 + 10^{(r_j-r_i)/400}}, \qquad E_j = 1 - E_i.
\]
After a human or automated decision, ratings are updated by
\[
R_i \leftarrow R_i + K(S_i - E_i), \qquad R_j \leftarrow R_j + K(S_j - E_j),
\]
with \(K=32\). The paper notes that \(S_i \in \{1, 0.5, 0\}\) can encode win, tie, or loss, although ties were not reported in the benchmark tasks [2508.21550].

The uncertainty measure is based on information gain. Let
\[
p_{ij} = \frac{1}{1 + 10^{(r_j-r_i)/400}},
\]
and define \(P_{\text{before}} = [p_{ij}, 1-p_{ij}]\). Information gain is the KL-divergence from the uniform distribution,
\[
\operatorname{InfoGain}(i,j) =
\operatorname{KL}\!\left(P_{\text{before}} \,\|\, P_{\text{uniform}}\right)
= \sum_{k \in \{i,j\}} p_k \log \frac{p_k}{0.5}.
\]
This is multiplied by a bucket factor and a confidence factor:
\[
\operatorname{Priority}(i,j) =
\operatorname{InfoGain}(i,j)\cdot \gamma(b_i,b_j)\cdot \phi(\text{conf}_i,\text{conf}_j),
\]
with \(\gamma(b_i,b_j)=1.2\) for cross-bucket pairs and \(1.0\) otherwise, and
\[
\phi(\text{conf}_i,\text{conf}_j) = 2.0 - \text{avg\_conf}.
\]
The normalized uncertainty is
\[
\operatorname{uncertainty}(i,j) = 1 - \frac{\operatorname{Priority}(i,j)}{\log 2}.
\]
Human annotation is requested when \(\operatorname{uncertainty}(i,j) \ge \theta_t\); otherwise the comparison is automated using the current Elo sign [2508.21550].

The threshold is adaptive:
\[
\theta_t = \theta_0 \Bigl(1 + \alpha \,\frac{\text{remaining}}{\text{total}}\Bigr)\, \beta^{\operatorname{accuracy}_t},
\]
with \(\theta_0 = 0.15\), \(\alpha = 0.3\), and \(\beta = 0.9\). The paper states that the threshold is updated after each batch of human comparisons or every \(k\) merges, whichever occurs first. As empirical accuracy improves, \(\beta^{\operatorname{accuracy}_t}\) shrinks \(\theta_t\), enabling more automation; if budget becomes tight, \(\theta_t\) increases, routing more cases to humans [2508.21550].

The method is also linked explicitly to Bradley–Terry-style probabilistic ranking. The paper presents the Bradley–Terry formulation
\[
P(i \succ j) = \frac{e^{\beta_i}}{e^{\beta_i}+e^{\beta_j}},
\]
and notes that Elo uses a practical logistic approximation. This places EZ-Sort within standard latent-strength ranking interpretations while retaining the operational convenience of Elo updates [2508.21550].

## 4. Empirical performance and reliability

Evaluation was conducted on three datasets. FGNET contains 1,002 face images with continuous age labels from 0 to 69 years. DHCI contains 450 historical images with five classes spanning the 1930s to the 1970s. EyePACS contains 28,792 retinal images with 3-level quality grading [2508.21550].

On FGNET, the human comparison counts show the principal efficiency result. For \(n=30\), exhaustive pairwise comparison required 435 comparisons, the prior sorting-based method required 126 human comparisons, and EZ-Sort required 89. For \(n=50\), the counts were 1,225, 240, and 142. For \(n=100\), they were 4,950, 582, and 467. The corresponding reductions for \(n=100\) were 90.5% fewer human comparisons than exhaustive pairwise comparison and 19.8% fewer than the prior sorting-based method; equivalently, EZ-Sort used 9.4% of the exhaustive human annotation cost [2508.21550].

The human-versus-automated split further clarifies where the savings arise. Across sizes in FGNET, human annotation was used for 23.1% of the comparisons at \(n=30\), 18.4% at \(n=50\), and 31.2% at \(n=100\); the remainder were resolved automatically. From an information-theoretic perspective, the paper reports that EZ-Sort used approximately \(0.87\times\), \(0.73\times\), and \(1.01\times\) the \(n \ln n\) lower bound at \(n=30\), \(50\), and \(100\), respectively. At \(n=100\), 467 human queries are reported as approximately 90% of the 520-query lower limit [2508.21550].

Inter-rater reliability was measured on 30 images per dataset using three experts, with \(p < 0.05\). On EyePACS, classification achieved Spearman 0.53 \((\pm 0.06)\), Kendall 0.46 \((\pm 0.07)\), Pearson 0.54 \((\pm 0.07)\), and ICC 0.75; the prior sorting method achieved Spearman 0.72 \((\pm 0.07)\), Kendall 0.56 \((\pm 0.06)\), Pearson 0.72 \((\pm 0.07)\), and ICC 0.89; EZ-Sort achieved Spearman 0.85 \((\pm 0.09)\), Kendall 0.76 \((\pm 0.14)\), Pearson 0.85 \((\pm 0.09)\), and ICC 0.94. On DHCI, classification produced Spearman 0.39, Kendall 0.33, Pearson 0.42, and ICC 0.68; sorting produced Spearman 0.47, Kendall 0.35, Pearson 0.47, and ICC 0.78; EZ-Sort produced Spearman 0.47, Kendall 0.39, Pearson 0.47, and ICC 0.73. On FGNET, classification produced Spearman 0.92, Kendall 0.85, Pearson 0.94, and ICC 0.97; sorting produced Spearman 0.97, Kendall 0.88, Pearson 0.97, and ICC 0.99; EZ-Sort produced Spearman 0.96, Kendall 0.91, Pearson 0.96, and ICC 0.99 [2508.21550].

These results indicate that the reduction in annotation cost did not come from simply weakening the task. On EyePACS, EZ-Sort substantially exceeded both classification and the earlier sorting approach in all reported reliability measures. On FGNET, it matched the prior sorting method in ICC and remained close in the correlation metrics while using fewer human comparisons. On DHCI, the results were mixed: Spearman and Pearson matched the prior sorting method, Kendall improved, and ICC was lower. This suggests that the gains are strongest in CLIP-friendly or visually discriminative domains and more modest in noisier settings [2508.21550].

## 5. Ablations and relation to prior ranking methods

The principal ablation concerns prompting strategy. Two flat baselines were tested: a simple template, “a photograph of a [class],” and a flat detailed template generated with GPT-4 attributes, “a photograph of [class] with [attributes].” Hierarchical prompting outperformed the flat variants across datasets, with up to 2.0 MAE improvement. On FGNET, hierarchical prompting achieved Spearman correlation 0.90 versus 0.83 for the flat variant, which the paper reports as an 8.4% relative gain [2508.21550].

EZ-Sort is positioned against two families of prior work. The first is classical pairwise models such as Bradley–Terry–Luce and active learning methods that assume uniform priors and therefore require many comparisons to obtain good rankings. The second is sorting-based active pairwise comparison, which already reduces the human query count to \(O(n \log n)\) but begins from scratch. EZ-Sort departs from both by introducing a semantically informed prior through zero-shot CLIP pre-ordering, carrying that prior into bucket-aware Elo initialization, and then using KL-based uncertainty routing to allocate human effort selectively [2508.21550].

The head-to-head comparison on FGNET with \(n=100\) is the clearest quantitative summary: the prior sorting-based method required 582 human comparisons, whereas EZ-Sort required 467, a reduction of 19.8%, with comparable or higher inter-rater reliability. The paper therefore frames the method not as a replacement for sorting-based active sampling, but as an augmentation of it with pretrained multimodal semantics and explicit uncertainty control [2508.21550].

A plausible implication is that the effectiveness of EZ-Sort depends strongly on whether the CLIP prior is aligned with the ranking criterion. The ablation results support this interpretation because the hierarchical prompt design, rather than CLIP usage alone, appears central to obtaining a useful coarse ordering.

## 6. Limitations, implementation parameters, and usage conditions

The paper identifies several limitations and failure modes. CLIP priors can be biased, and domain shift may degrade pre-ordering quality, especially in historical or medical imagery. Mis-bucketed items may therefore introduce early errors, even though EZ-Sort increases review priority for cross-bucket and low-confidence pairs. Hierarchical prompting may also stall or produce shallow trees when domain distinctions are subtle and difficult to capture in prompts. Noisy annotators and imbalanced datasets can perturb Elo updates, and integrating annotator reliability models such as Whitehill et al. is described as future work. Preliminary trials with Bayesian Elo variants such as TrueSkill did not improve results in the reported tasks. The paper also notes ethical concerns: because CLIP is trained on web-scale data, societal biases may propagate, so human oversight is essential in high-stakes domains [2508.21550].

The reported implementation uses CLIP ViT-B/32, with average CPU preprocessing time of approximately 39 ms per image. Prompt depth is typically 3–5 levels, and splitting stops when GPT-4 flags unreliable splits. The bucket settings are \(k=5\) for FGNET and \(k=3\) for DHCI and EyePACS. Elo uses \(K=32\), \(r_{\text{base}}\) linearly spaced in \([1200,1800]\), and \(\delta_b=75\). The uncertainty components use \(\tau=0.1\), \(\gamma=1.2\) for cross-bucket comparisons and \(1.0\) otherwise, \(\phi = 2.0 - \text{avg\_conf}\), and threshold parameters \(\theta_0 = 0.15\), \(\alpha=0.3\), and \(\beta=0.9\). The recommended interface batches uncertain comparisons, updates \(\theta_t\) periodically, tracks \(\operatorname{accuracy}_t\) empirically, and keeps tie-breaking pro-human [2508.21550].

In computational terms, the method preserves the exact comparison schedule of classical MergeSort, with \(O(1)\) uncertainty checks per comparison and total complexity \(O(n \log n)\). Pre-ordering is reported as \(O(kn)\) for fixed \(k\), followed by \(O(n)\) Elo initialization. The paper summarizes expected human annotation with a simple cost model of approximately \(c n \log n\) human comparisons with \(c<1\), domain dependent; for FGNET at \(n=100\), \(c \approx 467/664 \approx 0.70\) [2508.21550].

The method is therefore most effective in domains with visible, discriminative features and sufficiently informative CLIP prompts. In those settings, the reported results indicate that zero-shot semantic priors and uncertainty-aware routing can materially reduce human pairwise annotation while preserving or improving reliability [2508.21550].

Source: https://www.emergentmind.com/topics/ez-sort