---
title: Set-List Wise Training Strategy
url: https://www.emergentmind.com/topics/set-list-wise-training-strategy
type: topic
---

# Set-List Wise Training Strategy

The Set-List Wise Training Strategy, also referred to as "ListPL" in some literature, is a collection of list-wise and set-structured optimization techniques designed to address ranking and structured prediction problems where candidate items are naturally annotated with ambiguous or multi-level relevance labels. Unlike classical pairwise or pointwise ranking approaches, set-list wise strategies are tailored to fully exploit the order structure of an entire candidate set, while rigorously accounting for inherent ambiguities in real-world supervised data and emphasizing relevant statistical properties such as ordinal relationships and permutation invariance.

## 1. Modeling Relevance Ambiguity in List-wise Ranking

In supervised learning-to-rank contexts, such as document retrieval, input data is typically organized as a set $D = \{d_1, ..., d_n\}$ corresponding to candidate items associated with a query $q$. Each item is annotated by a discrete relevance label $y_i$ on an ordinal scale (e.g., $\{0,1,2,3,4\}$). Unlike strict total orders, these labels often only produce a partial order, leaving ambiguous tie groups. The Set-List Wise approach partitions $D$ into equivalence classes $G_k = \{d_i \mid y_i = k\}$ for each grade $k$, enforcing that within each group $G_k$ items are exchangeable under the ground-truth—no statistical preference should be enforced for permutations within these sets. This enables models to acknowledge that any total order consistent with the coarser grade-wise order ($G_K \succ G_{K-1} \succ ...$) is equally valid for supervision [1707.07493].

## 2. Permutation-based List-wise Objectives

Set-List Wise strategies are built upon probabilistic permutation models, notably the Plackett-Luce (PL) distribution, to capture the full distribution of correct rankings under label ambiguity. For label vector $Y$ and mapping $\psi(y_i)$ (typically the identity function), the PL probability of a permutation $\pi$ is:
$$
P_{PL}(\pi \mid D; \psi_Y) = \prod_{i=1}^n \frac{\exp(\psi(y_{\pi_i}))}{\sum_{j=i}^n \exp(\psi(y_{\pi_j}))}
$$
Sampling from this distribution at each stochastic gradient step generates a ranking $\pi^*$ that serves as a surrogate "ground-truth" ordering, ensuring that only statistically necessary ordering constraints are enforced by supervision. The list-wise loss is defined as the negative log-likelihood of $\pi^*$ under the model's score-induced permutation model:
$$
L(f; D, Y) = -\log P_{PL}(\pi^* \mid D; f) = -\sum_{i=1}^n [ f(d_{\pi^*_i}) - \log \sum_{j=i}^n \exp(f(d_{\pi^*_j})) ]
$$
By re-sampling $\pi^*$ at each iteration, the loss averages over all valid tie-breaking permutations, regularizing the model against overfitting forced orderings within grade sets [1707.07493].

## 3. Top-Rank Sensitive and Enhanced Loss Functions

In applications such as machine translation or information retrieval, empirical utility is often heavily concentrated at the top of the ranked list. Top-rank enhanced set-list wise losses introduce position-dependent costs $c(j)$ to prioritize accuracy for higher ranked candidates. A canonical weighting is:
$$
c(j) = \frac{2(k - j + 1)}{k(k + 1)}
$$
This up-weights the influence of errors at the head of the list (small $j$), and down-weights errors at lower ranks, better aligning surrogate losses with top-$k$ evaluation metrics such as BLEU or nDCG. Applying this weighting to ListMLE or analogous distributions results in the top-rank enhanced ListMLE and ListNet objectives, which have demonstrated empirical superiority over their base variants in statistical machine translation tasks [1707.05438].

## 4. Extensions to Dense Retrieval and Ordinal Synthetic Data

Set-list wise training has been extended to dense retrieval systems where label ambiguity is modeled via synthetic data generation. Using open-source language models to generate for each query $q$ and relevance grade $\ell$ a set of synthetic documents $\{d_{q,i}\}$ annotated at multiple discrete levels, a ground-truth relevance histogram $u$ is built to reflect the empirical label structure. The predictive distribution $p$ is taken as the model softmax output over all candidates. The loss is then defined via the regularized 1-Wasserstein (earth-mover's) distance between $p$ and $u$, minimizing:
$$
W_C(p, u) = \min_{T \in \mathcal{U}(p,u)} \sum_{i=1}^N \sum_{j=1}^N T_{ij} C_{ij}
$$
with $C_{ij} = |r_i - r_j|^p$, implemented via the Sinkhorn algorithm for tractable entropic regularization. This enables global optimization over all relevance levels, rather than collapsing all non-positives into a single negative class as in InfoNCE [2503.23239].

| Approach               | Label Ambiguity       | Metric Sensitivity    |
|------------------------|----------------------|----------------------|
| ListPL [1707.07493]    | Explicit (sets)      | Uniform/top-$n$      |
| Top-rank Enhanced [1707.05438] | Implicit    | Top-weighted         |
| Wasserstein List-wise [2503.23239] | Synthetic, graded | Ordinal/ground metric |
  
Set-list wise methods handle not only discrete ambiguity at annotation time, but also the complex structure of graded, synthetic training data, providing a unified framework for modern ranking models.

## 5. Training Procedures and Neural Architectures

For neural implementations, list-wise losses such as ListPL are typically paired with multi-layer perceptrons (MLPs) for feature-based ranking or Siamese encoders for dense retrieval. In [1707.07493], a 3-layer MLP (input: 136-dim features, hidden: 2×80 ReLU, output: linear score) was trained on MSLR-WEB10k, using ADAM optimizer and query-sized batches. The loss is backpropagated through the sampled permutation and network, without the need for additional regularization, as the tie-averaging effect of sampling is empirically sufficient. In dense retrieval [2503.23239], end-to-end backpropagation includes differentiating through the Sinkhorn iterations, which propagate gradients from the Wasserstein OT layer back into query and document BERT encoders.

## 6. Empirical Results and Generalization

Set-List Wise methods have demonstrated significant improvements over classical list-wise losses. On MSLR-WEB10k, ListPL achieves nDCG@10 ≈ 0.495, exceeding ListMLE (≈0.490) and ListNet (≈0.487). These gains are statistically significant and are accompanied by enhanced generalization: ListPL does not overfit after approximately 100 epochs, in contrast to the rapid overfitting of ListMLE and ListNet. In statistical machine translation, top-rank enhanced ListMLE yields BLEU increases of up to +1.07 points over strong pairwise and standard list-wise baselines—a substantial effect in large-scale settings [1707.07493, 1707.05438].

## 7. Limitations, Extensions, and Broader Applicability

Computational cost remains a consideration for set-list wise objectives, especially those requiring full permutation modeling as in ListNet. Sampling-based objectives (e.g., ListPL) and entropically-regularized optimal transport (Sinkhorn) alleviate some of this burden. The methodology is robust to "patchy" or heterogeneous candidate sets through instance aggregation. The core set-list wise approach is extensible across structured prediction problems—syntactic parsing, machine translation, n-best reranking—and can be tailored to metric-specific requirements (e.g., adjusting $c(j)$ to match an external evaluation’s utility curve). Wasserstein-based losses further generalize these ideas to accommodate continuous or synthetic label regimes and are particularly effective in zero-shot and low-supervision domains [1707.07493, 1707.05438, 2503.23239].

Source: https://www.emergentmind.com/topics/set-list-wise-training-strategy