Set-List Wise Training Strategy

• Set-List Wise Training Strategy is a method that employs list-wise and set-structured optimization to fully utilize candidate order structures while addressing ambiguous relevance annotations.
• It utilizes permutation-based objectives like the Plackett-Luce distribution and top-rank enhanced loss functions to prioritize critical ranking positions in evaluation metrics.
• Empirical results show improvements in metrics such as nDCG and BLEU, and the approach extends to dense retrieval, machine translation, and other structured prediction tasks.

The Set-List Wise Training Strategy, also referred to as "ListPL" in some literature, is a collection of list-wise and set-structured optimization techniques designed to address ranking and structured prediction problems where candidate items are naturally annotated with ambiguous or multi-level relevance labels. Unlike classical pairwise or pointwise ranking approaches, set-list wise strategies are tailored to fully exploit the order structure of an entire candidate set, while rigorously accounting for inherent ambiguities in real-world supervised data and emphasizing relevant statistical properties such as ordinal relationships and permutation invariance.

1. Modeling Relevance Ambiguity in List-wise Ranking

In supervised learning-to-rank contexts, such as document retrieval, input data is typically organized as a set $D = \{d_1, ..., d_n\}$ corresponding to candidate items associated with a query $q$. Each item is annotated by a discrete relevance label $y_i$ on an ordinal scale (e.g., $\{0,1,2,3,4\}$). Unlike strict total orders, these labels often only produce a partial order, leaving ambiguous tie groups. The Set-List Wise approach partitions $D$ into equivalence classes $G_k = \{d_i \mid y_i = k\}$ for each grade $k$, enforcing that within each group $G_k$ items are exchangeable under the ground-truth—no statistical preference should be enforced for permutations within these sets. This enables models to acknowledge that any total order consistent with the coarser grade-wise order ($G_K \succ G_{K-1} \succ ...$) is equally valid for supervision (Jagerman et al., 2017).

2. Permutation-based List-wise Objectives

Set-List Wise strategies are built upon probabilistic permutation models, notably the Plackett-Luce (PL) distribution, to capture the full distribution of correct rankings under label ambiguity. For label vector $Y$ and mapping $\psi(y_i)$ (typically the identity function), the PL probability of a permutation $\pi$ is:

$$P_{PL}(\pi \mid D; \psi_Y) = \prod_{i=1}^n \frac{\exp(\psi(y_{\pi_i}))}{\sum_{j=i}^n \exp(\psi(y_{\pi_j}))}$$

Sampling from this distribution at each stochastic gradient step generates a ranking $\pi^*$ that serves as a surrogate "ground-truth" ordering, ensuring that only statistically necessary ordering constraints are enforced by supervision. The list-wise loss is defined as the negative log-likelihood of $\pi^*$ under the model's score-induced permutation model:

$$L(f; D, Y) = -\log P_{PL}(\pi^* \mid D; f) = -\sum_{i=1}^n [ f(d_{\pi^*_i}) - \log \sum_{j=i}^n \exp(f(d_{\pi^*_j})) ]$$

By re-sampling $\pi^*$ at each iteration, the loss averages over all valid tie-breaking permutations, regularizing the model against overfitting forced orderings within grade sets (Jagerman et al., 2017).

3. Top-Rank Sensitive and Enhanced Loss Functions

In applications such as machine translation or information retrieval, empirical utility is often heavily concentrated at the top of the ranked list. Top-rank enhanced set-list wise losses introduce position-dependent costs $c(j)$ to prioritize accuracy for higher ranked candidates. A canonical weighting is:

$c(j) = \frac{2(k - j + 1)}{k(k + 1)}$

This up-weights the influence of errors at the head of the list (small $j$), and down-weights errors at lower ranks, better aligning surrogate losses with top-$k$ evaluation metrics such as BLEU or nDCG. Applying this weighting to ListMLE or analogous distributions results in the top-rank enhanced ListMLE and ListNet objectives, which have demonstrated empirical superiority over their base variants in statistical machine translation tasks (Chen et al., 2017).

4. Extensions to Dense Retrieval and Ordinal Synthetic Data

Set-list wise training has been extended to dense retrieval systems where label ambiguity is modeled via synthetic data generation. Using open-source LLMs to generate for each query $q$ and relevance grade $\ell$ a set of synthetic documents $\{d_{q,i}\}$ annotated at multiple discrete levels, a ground-truth relevance histogram $u$ is built to reflect the empirical label structure. The predictive distribution $p$ is taken as the model softmax output over all candidates. The loss is then defined via the regularized 1-Wasserstein (earth-mover's) distance between $p$ and $u$, minimizing:

$$W_C(p, u) = \min_{T \in \mathcal{U}(p,u)} \sum_{i=1}^N \sum_{j=1}^N T_{ij} C_{ij}$$

with $C_{ij} = |r_i - r_j|^p$, implemented via the Sinkhorn algorithm for tractable entropic regularization. This enables global optimization over all relevance levels, rather than collapsing all non-positives into a single negative class as in InfoNCE (Esfandiarpoor et al., 29 Mar 2025).

Approach	Label Ambiguity	Metric Sensitivity
ListPL (Jagerman et al., 2017)	Explicit (sets)	Uniform/top-$n$
Top-rank Enhanced (Chen et al., 2017)	Implicit	Top-weighted
Wasserstein List-wise (Esfandiarpoor et al., 29 Mar 2025)	Synthetic, graded	Ordinal/ground metric

Set-list wise methods handle not only discrete ambiguity at annotation time, but also the complex structure of graded, synthetic training data, providing a unified framework for modern ranking models.

5. Training Procedures and Neural Architectures

For neural implementations, list-wise losses such as ListPL are typically paired with multi-layer perceptrons (MLPs) for feature-based ranking or Siamese encoders for dense retrieval. In (Jagerman et al., 2017), a 3-layer MLP (input: 136-dim features, hidden: 2×80 ReLU, output: linear score) was trained on MSLR-WEB10k, using ADAM optimizer and query-sized batches. The loss is backpropagated through the sampled permutation and network, without the need for additional regularization, as the tie-averaging effect of sampling is empirically sufficient. In dense retrieval (Esfandiarpoor et al., 29 Mar 2025), end-to-end backpropagation includes differentiating through the Sinkhorn iterations, which propagate gradients from the Wasserstein OT layer back into query and document BERT encoders.

6. Empirical Results and Generalization

Set-List Wise methods have demonstrated significant improvements over classical list-wise losses. On MSLR-WEB10k, ListPL achieves nDCG@10 ≈ 0.495, exceeding ListMLE (≈0.490) and ListNet (≈0.487). These gains are statistically significant and are accompanied by enhanced generalization: ListPL does not overfit after approximately 100 epochs, in contrast to the rapid overfitting of ListMLE and ListNet. In statistical machine translation, top-rank enhanced ListMLE yields BLEU increases of up to +1.07 points over strong pairwise and standard list-wise baselines—a substantial effect in large-scale settings (Jagerman et al., 2017, Chen et al., 2017).

7. Limitations, Extensions, and Broader Applicability

Computational cost remains a consideration for set-list wise objectives, especially those requiring full permutation modeling as in ListNet. Sampling-based objectives (e.g., ListPL) and entropically-regularized optimal transport (Sinkhorn) alleviate some of this burden. The methodology is robust to "patchy" or heterogeneous candidate sets through instance aggregation. The core set-list wise approach is extensible across structured prediction problems—syntactic parsing, machine translation, n-best reranking—and can be tailored to metric-specific requirements (e.g., adjusting $c(j)$ to match an external evaluation’s utility curve). Wasserstein-based losses further generalize these ideas to accommodate continuous or synthetic label regimes and are particularly effective in zero-shot and low-supervision domains (Jagerman et al., 2017, Chen et al., 2017, Esfandiarpoor et al., 29 Mar 2025).