Noise-Tolerant Ranking Loss

• Noise-tolerant ranking loss comprises objectives designed to maintain ranking consistency by mitigating various noise types through order-preservation and unbiased corrections.
• Pairwise margin approaches, such as the Quadratic Disparity Ranking loss, leverage smooth gradients to robustly separate positive from negative instances even under low-SNR conditions.
• Advanced techniques integrate distributionally robust optimization and group-wise aggregation to enhance ranking performance across domains under challenging noise and label perturbations.

Noise-tolerant ranking loss denotes a class of ranking objectives and surrogate-risk constructions designed to preserve ranking quality when the supervision, comparisons, or observations are corrupted by class-conditional label noise, incomplete labels, Gaussian or adversarial perturbations, noisy pairwise comparisons, or severe acoustic noise. Recent formulations realize this goal through several distinct mechanisms: order-preserving label-symmetric losses for learning to rank (Haddad, 2022), inverse-noise correction for ordinal losses (Garg et al., 2019), distributionally robust optimization over Wasserstein balls (Sotudian et al., 2021), variance-aware least-squares estimation from noisy pairwise votes (Christoforou et al., 2020), bag-sampled and group-wise contrastive objectives for incompletely labeled retrieval (Li et al., 2022), ranked-positive contrastive objectives for blurry positive/negative boundaries (Hoffmann et al., 2022), and squared-margin pairwise objectives such as the quadratic disparity ranking loss used in voice activity detection (Wang et al., 28 Aug 2025).

1. Formal criteria for robustness

A central formalization of noise tolerance in learning to rank is order preservation under class-conditional label noise. For clean risk $R(f)$, noisy risk $\tilde R(f)$, and corruption accuracy $\gamma>1/2$, a loss is order-preserving if, for every two scorers $f,g$,

$\tilde R(f)-\tilde R(g)=(2\gamma-1)\,[R(f)-R(g)].$

Equivalently, $\arg\min \tilde R=\arg\min R$. A sufficient condition is label symmetry:

$\ell(\alpha)+\ell(-\alpha)=c,$

for all $\alpha\neq 0$, which implies

$\tilde R(f)=(2\gamma-1)R(f)+c(1-\gamma).$

The same line of work shows that $0$-$\tilde R(f)$0, hinge, $\tilde R(f)$1, and AUC-pairwise $\tilde R(f)$2-$\tilde R(f)$3 losses are label-symmetric, while logistic, exponential, and RankNet are not; symmetrized logistic and symmetrized RankNet are therefore introduced as noise-tolerant analogs (Haddad, 2022).

A second formalization is unbiased noise correction. In robust ordinal regression, the observed noisy label $\tilde R(f)$4 is generated from the true label $\tilde R(f)$5 through a class-conditional noise matrix $\tilde R(f)$6, and the corrected loss vector is defined by

$\tilde R(f)$7

This yields

$\tilde R(f)$8

so the surrogate is unbiased with respect to the clean loss. The same work proves rank consistency, meaning the threshold order $\tilde R(f)$9 is preserved at optimum, and shows that SGD preserves ordering in expectation when the step-size satisfies $\gamma>1/2$0 (Garg et al., 2019).

A third criterion is structure consistency for supervised ranking from partial preferences. The asymptotic theory of ranking establishes that a surrogate is Fisher-consistent if and only if it is structure-consistent, equivalently if the local suboptimality gap

$\gamma>1/2$1

remains positive for every $\gamma>1/2$2. This is important because the same analysis proves that many common pairwise convex surrogates fail this requirement even in low-noise settings (Duchi et al., 2012).

2. Pairwise margin losses under noisy labels and low-SNR observations

A prominent modern example is the Quadratic Disparity Ranking (QDR) loss introduced in "SincQDR-VAD" (Wang et al., 28 Aug 2025). For positive frames $\gamma>1/2$3, negative frames $\gamma>1/2$4, sigmoid scores $\gamma>1/2$5, and margin $\gamma>1/2$6, the loss is

$\gamma>1/2$7

Training uses the hybrid objective

$\gamma>1/2$8

where $\gamma>1/2$9 is the binary cross-entropy over the mini-batch and $f,g$0.

QDR is explicitly pairwise: every speech frame is forced to outscore every non-speech frame by at least $f,g$1. Its robustness claim is tied to the quadratic margin penalty, which yields smooth gradients even when the margin is severely violated. The paper contrasts this with hinge ranking loss, which is linear beyond the margin and can produce zero gradient when the violation is large, and with pairwise logistic ranking, which can saturate and be less sensitive near the margin. By averaging over all $f,g$2 pairs, QDR also acts as an implicit regularizer against individual outliers. In the VAD setting, this pairwise formulation is directly linked to AUROC optimization, class-imbalance handling, and resistance to spurious score flips under heavy noise. Empirically, removing QDR drops AUROC on clean AVA-Speech from $f,g$3 to $f,g$4, and on noisy variants from $f,g$5 to $f,g$6; the reported SNR breakdown shows gains of $f,g$7–$f,g$8 AUROC at each noise level, with the biggest relative improvements at $f,g$9 dB. Practical guidance in the same work sets $\tilde R(f)-\tilde R(g)=(2\gamma-1)\,[R(f)-R(g)].$0, $\tilde R(f)-\tilde R(g)=(2\gamma-1)\,[R(f)-R(g)].$1, uses mini-batches of size $\tilde R(f)-\tilde R(g)=(2\gamma-1)\,[R(f)-R(g)].$2, and recommends negative subsampling when memory is tight (Wang et al., 28 Aug 2025).

A closely related construction appears in recommender-system multi-task learning as the PWiseR loss (Durmus et al., 2024). Here conversion examples $\tilde R(f)-\tilde R(g)=(2\gamma-1)\,[R(f)-R(g)].$3 are treated as “definitive positives,” click-only examples $\tilde R(f)-\tilde R(g)=(2\gamma-1)\,[R(f)-R(g)].$4 as noisier negatives, and no-click examples $\tilde R(f)-\tilde R(g)=(2\gamma-1)\,[R(f)-R(g)].$5 as an additional negative class. PWiseR forms two sets of pairwise comparisons between conversion anchors and CTR scores: click-only versus conversion, and no-click versus conversion. Both terms use a squared-margin hinge. The full objective is

$\tilde R(f)-\tilde R(g)=(2\gamma-1)\,[R(f)-R(g)].$6

This construction is motivated by the observation that a conversion necessitates a preceding click, so conversions are treated as higher-trust supervision for CTR ranking. In the reported Alibaba-US MMoE results, CTR AUC improves from $\tilde R(f)-\tilde R(g)=(2\gamma-1)\,[R(f)-R(g)].$7 to $\tilde R(f)-\tilde R(g)=(2\gamma-1)\,[R(f)-R(g)].$8 and CTCVR from $\tilde R(f)-\tilde R(g)=(2\gamma-1)\,[R(f)-R(g)].$9 to $\arg\min \tilde R=\arg\min R$0; on the industrial dataset, MMoE moves from $\arg\min \tilde R=\arg\min R$1 to $\arg\min \tilde R=\arg\min R$2 on CTR AUC and from $\arg\min \tilde R=\arg\min R$3 to $\arg\min \tilde R=\arg\min R$4 on CTCVR. The paper recommends ensuring that each mini-batch contains at least one conversion example and notes that pair sampling may be needed to control the $\arg\min \tilde R=\arg\min R$5 growth (Durmus et al., 2024).

These pairwise-margin methods share a common pattern: they retain pointwise BCE for calibration while adding a ranking term that gives privileged status to cleaner positives or cleaner separations. This suggests that pairwise robustness is often treated as complementary to, rather than a replacement for, per-item probability estimation.

3. Distributionally robust and explicitly noise-corrected objectives

"Distributionally Robust Multi-output Regression Ranking" formulates robustness as a min-max problem over a Wasserstein ball around the empirical distribution (Sotudian et al., 2021). With random example $\arg\min \tilde R=\arg\min R$6, empirical distribution $\arg\min \tilde R=\arg\min R$7, radius $\arg\min \tilde R=\arg\min R$8, and multi-output loss $\arg\min \tilde R=\arg\min R$9, the objective is

$\ell(\alpha)+\ell(-\alpha)=c,$0

Strong duality yields the equivalent convex program

$\ell(\alpha)+\ell(-\alpha)=c,$1

where $\ell(\alpha)+\ell(-\alpha)=c,$2. For $\ell(\alpha)+\ell(-\alpha)=c,$3, the regularizer is the spectral norm of $\ell(\alpha)+\ell(-\alpha)=c,$4. The scoring function is multi-output: each document receives a $\ell(\alpha)+\ell(-\alpha)=c,$5-dimensional Ground-Truth Deviation vector $\ell(\alpha)+\ell(-\alpha)=c,$6, intended to encode NDCG-swap deviations, positional-deviation scores, and a per-document importance weight. At test time, each feature vector is mapped to a predicted deviation vector $\ell(\alpha)+\ell(-\alpha)=c,$7, and the final ranking is extracted greedily by cycling through columns.

The robustness interpretation is explicit. Gaussian feature noise, adversarial perturbations such as FGSM, and label poisoning are all described as shifts of the empirical distribution within, or relative to, a Wasserstein ball; the worst-case objective therefore constrains sensitivity in $\ell(\alpha)+\ell(-\alpha)=c,$8-$\ell(\alpha)+\ell(-\alpha)=c,$9 space through the matrix-norm penalty. On OHSUMED, mean NDCG@5 over five folds is reported as $\alpha\neq 0$0 for DRMRR versus $\alpha\neq 0$1 for LambdaMART_NDCG and $\alpha\neq 0$2 for XE-MART. Under Gaussian noise, DRMRR’s AP@5 and NDCG@5 degrade by only $\alpha\neq 0$3, whereas baselines lose $\alpha\neq 0$4–$\alpha\neq 0$5; under universal adversarial FGSM it remains within $\alpha\neq 0$6–$\alpha\neq 0$7 of its clean score while baselines drop by $\alpha\neq 0$8–$\alpha\neq 0$9; under black-box adversaries it shows a $\tilde R(f)=(2\gamma-1)R(f)+c(1-\gamma).$0 drop at $\tilde R(f)=(2\gamma-1)R(f)+c(1-\gamma).$1, while baselines suffer $\tilde R(f)=(2\gamma-1)R(f)+c(1-\gamma).$2–$\tilde R(f)=(2\gamma-1)R(f)+c(1-\gamma).$3; under label-flip attack it loses $\tilde R(f)=(2\gamma-1)R(f)+c(1-\gamma).$4 in NDCG@5, while baselines lose $\tilde R(f)=(2\gamma-1)R(f)+c(1-\gamma).$5–$\tilde R(f)=(2\gamma-1)R(f)+c(1-\gamma).$6 (Sotudian et al., 2021).

Robust ordinal regression addresses a different regime: class-conditional label noise in ordinal targets (Garg et al., 2019). Two base ordinal losses are considered, the implicit margin-constrained loss

$\tilde R(f)=(2\gamma-1)R(f)+c(1-\gamma).$7

and cumulative-logit cross-entropy

$\tilde R(f)=(2\gamma-1)R(f)+c(1-\gamma).$8

The paper’s noise model uses an inversely-decaying transition matrix, with nearby ranks more likely to be confused. Noise-corrected losses are formed by multiplying the clean-loss vector by $\tilde R(f)=(2\gamma-1)R(f)+c(1-\gamma).$9. The resulting objectives are proved unbiased, rank-consistent, and compatible with deep implementations. Empirically, under inversely-decaying noise with $0$0, $0$1 reduces MAE on the synthetic dataset from $0$2 to $0$3 when $0$4 is known, improves California from $0$5 to $0$6, and improves MSLR from $0$7 to $0$8. The same study notes that estimation error limits gains on very small datasets such as Boston and Abalone (Garg et al., 2019).

4. Aggregation, group-wise ranking, and partial-preference regimes

When the data consist of noisy pairwise votes rather than clean labels, one robust formulation is the graph-based least-squares ranking loss (Christoforou et al., 2020). Each edge $0$9 in a comparison graph receives $\tilde R(f)$00 Bernoulli comparisons with success probability $\tilde R(f)$01, where $\tilde R(f)$02 is the hidden quality of item $\tilde R(f)$03. After estimating $\tilde R(f)$04 and inverting the link function to obtain $\tilde R(f)$05, ranking is defined by

$\tilde R(f)$06

A Taylor expansion shows that the transformed error $\tilde R(f)$07 has approximately zero mean and variance

$\tilde R(f)$08

This motivates weights $\tilde R(f)$09, so very noisy edges are down-weighted. Under mild spectral conditions, the method is $\tilde R(f)$10-PAC with total comparison complexity

$\tilde R(f)$11

The two-round adaptive version can cut the total number of comparisons by $\tilde R(f)$12–$\tilde R(f)$13 while retaining the same order guarantee (Christoforou et al., 2020).

For incompletely labeled document ranking, "Information retrieval for label noise document ranking by bag sampling and group-wise loss" replaces individual-document supervision by bag sampling and group-wise Localized Contrastive Estimation (LCE) (Li et al., 2022). For each query, the retrieved top $\tilde R(f)$14 candidates are divided into $\tilde R(f)$15 contiguous bags, and $\tilde R(f)$16 documents are sampled from each bag to form a group. The rationale is that MS MARCO-style corpora only mark one positive per query, so many unlabeled but truly relevant documents near the top would otherwise be treated as negatives. Each sampled group is encoded as a matrix of document representations, passed through a 1D convolution and max-pooling, and scored by a linear layer. If $\tilde R(f)$17 denotes the group containing the known positive, the group-wise loss is

$\tilde R(f)$18

With $\tilde R(f)$19, $\tilde R(f)$20, $\tilde R(f)$21, kernel size $\tilde R(f)$22, and $\tilde R(f)$23 filters, the reported model reaches dev MRR@100 $\tilde R(f)$24, compared with $\tilde R(f)$25 for ANCE + LongP, and eval MRR@100 $\tilde R(f)$26 (Li et al., 2022).

Contrastive learning produces another family of noise-tolerant ranking objectives. RINCE generalizes InfoNCE from binary positives to ranked positives $\tilde R(f)$27, each stage using a higher temperature $\tilde R(f)$28 (Hoffmann et al., 2022). For stage $\tilde R(f)$29,

$\tilde R(f)$30

and

$\tilde R(f)$31

The robustness claim is that low-temperature early stages focus on the most trusted positives, while later stages with higher temperature relax the penalty on noisier higher-rank positives. On CIFAR-100, RINCE-in reports $\tilde R(f)$32 linear accuracy versus $\tilde R(f)$33 for SCL-in and $\tilde R(f)$34 for cross-entropy, with retrieval and OOD-detection gains as well. In unsupervised video learning, RINCE-uni achieves UCF-101 top-1 $\tilde R(f)$35 and HMDB51 top-1 $\tilde R(f)$36, exceeding standard InfoNCE (Hoffmann et al., 2022).

At a broader theoretical level, supervised ranking from partial preferences can also be repaired by $\tilde R(f)$37-statistic aggregation (Duchi et al., 2012). The approach first aggregates $\tilde R(f)$38 partial observations through a structure map $\tilde R(f)$39, then minimizes a surrogate over these aggregates:

$\tilde R(f)$40

Under power-law conditions on the query distribution, bounded Lipschitz surrogates, uniform covering-number control, and suitable growth of $\tilde R(f)$41, the theory proves a uniform law of large numbers and consistency of approximate empirical minimizers (Duchi et al., 2012).

5. Empirical behavior across domains

The empirical record is heterogeneous in modality and metric, but a recurrent pattern is that robust ranking losses either improve the clean ranking metric directly or reduce the degradation curve as noise increases.

Setting	Reported result	Paper
Clean AVA-Speech	AUROC $\tilde R(f)$42, F$\tilde R(f)$43 $\tilde R(f)$44; removing QDR gives AUROC $\tilde R(f)$45, F$\tilde R(f)$46 $\tilde R(f)$47	(Wang et al., 28 Aug 2025)
Noisy AVA-Speech, avg. $\tilde R(f)$48 dB	AUROC $\tilde R(f)$49, F$\tilde R(f)$50 $\tilde R(f)$51; removing QDR gives AUROC $\tilde R(f)$52, F$\tilde R(f)$53 $\tilde R(f)$54	(Wang et al., 28 Aug 2025)
ACAM	F$\tilde R(f)$55 rises from $\tilde R(f)$56 (TinyVAD baseline) to $\tilde R(f)$57 with QDR+Sinc	(Wang et al., 28 Aug 2025)
OHSUMED	NDCG@5 $\tilde R(f)$58 for DRMRR, versus $\tilde R(f)$59 for LambdaMART_NDCG and $\tilde R(f)$60 for XE-MART	(Sotudian et al., 2021)
DRMRR under Gaussian noise	AP@5 and NDCG@5 degrade only $\tilde R(f)$61, while baselines lose $\tilde R(f)$62–$\tilde R(f)$63	(Sotudian et al., 2021)
Robust ordinal regression, synthetic	MAE $\tilde R(f)$64 for baseline CE, $\tilde R(f)$65 for $\tilde R(f)$66 with known $\tilde R(f)$67	(Garg et al., 2019)
Alibaba-US, MMoE	CTR AUC $\tilde R(f)$68; CTCVR $\tilde R(f)$69	(Durmus et al., 2024)
Industrial MMoE	CTR AUC $\tilde R(f)$70; CTCVR $\tilde R(f)$71	(Durmus et al., 2024)
MS MARCO Document Ranking dev	MRR@100 $\tilde R(f)$72 versus $\tilde R(f)$73 for ANCE + LongP	(Li et al., 2022)
CIFAR-100 linear evaluation	RINCE-in $\tilde R(f)$74, SCL-in $\tilde R(f)$75, cross-entropy $\tilde R(f)$76	(Hoffmann et al., 2022)

These results span VAD, medical document retrieval, drug response prediction, ordinal regression benchmarks, recommender systems, long-document ranking, and representation learning. A plausible implication is that noise-tolerant ranking losses are not tied to a single metric family: the literature reports improvements in AUROC, F$\tilde R(f)$77, NDCG@5, AP@5, MAE, MRR@100, retrieval R@1, top-1 accuracy, and OOD AUROC, depending on the ranking task.

6. Limitations, misconceptions, and recurring design principles

A recurring misconception is that pairwise convex surrogates automatically become reliable in low-noise settings. The asymptotic theory of supervised ranking explicitly rejects this: pairwise margin losses of the form

$\tilde R(f)$78

and shifted-margin losses

$\tilde R(f)$79

are shown to be inconsistent even under the reverse-triangle low-noise condition (Duchi et al., 2012). The same work further notes an NP-hardness barrier for optimizing such pairwise surrogates exactly.

Another misconception is that “noise tolerance” refers to a single mathematical property. In practice, the term covers distinct robustness mechanisms. Under class-conditional label noise, it may mean order preservation via label symmetry or unbiased correction via $\tilde R(f)$80 (Haddad, 2022, Garg et al., 2019). Under adversarial or distributional perturbations, it may mean worst-case control over a Wasserstein ball and the associated matrix-norm regularization (Sotudian et al., 2021). Under low-SNR and class-imbalanced observations, it may mean a squared-margin pairwise loss that continues to apply corrective pressure to misordered scores, as in QDR (Wang et al., 28 Aug 2025). Under incomplete labels, it may mean aggregation across bags or groups so that any single mislabeled negative contributes less to the gradient (Li et al., 2022).

Computational trade-offs are equally consistent across the literature. Pairwise objectives scale with $\tilde R(f)$81 in QDR and with $\tilde R(f)$82 pair counts in PWiseR, which is why both papers recommend subsampling negatives or pairs when necessary (Wang et al., 28 Aug 2025, Durmus et al., 2024). The robust ordinal framework incurs $\tilde R(f)$83 extra work per example and an $\tilde R(f)$84 inversion of the noise matrix once at startup (Garg et al., 2019). Group-wise document ranking adds convolution and max-pooling over sampled groups (Li et al., 2022). These are not failures of robustness, but they define the practical cost of obtaining it.

The hyperparameter patterns are also narrow and explicit. QDR uses only two additional parameters, $\tilde R(f)$85 and $\tilde R(f)$86, with the paper setting $\tilde R(f)$87, $\tilde R(f)$88, and recommending a grid search in $\tilde R(f)$89 for $\tilde R(f)$90 and typical $\tilde R(f)$91 (Wang et al., 28 Aug 2025). PWiseR reports $\tilde R(f)$92, $\tilde R(f)$93, and $\tilde R(f)$94 in Alibaba-US tuning (Durmus et al., 2024). RINCE relies on the number of ranks $\tilde R(f)$95 and an increasing temperature schedule, with $\tilde R(f)$96 given as an example for two-rank CIFAR-100 (Hoffmann et al., 2022). DRMRR depends on the Wasserstein radius $\tilde R(f)$97 and the norm pair $\tilde R(f)$98 (Sotudian et al., 2021). This suggests that robust ranking losses are typically parameterized by a small number of quantities that directly encode the assumed noise mechanism.

The literature therefore presents noise-tolerant ranking loss not as a single canonical objective, but as a design space. A plausible implication is that the appropriate loss is determined less by the ranking metric in isolation than by the corruption model: class-conditional noise favors order-preserving or noise-corrected surrogates, distributional shift favors DRO, incompletely labeled retrieval favors aggregation or group-wise objectives, and heavy low-SNR misordering favors squared-margin pairwise ranking.