LambdaLoss: NDCG-Optimized Ranking

• LambdaLoss is a differentiable ranking objective that optimizes position-based metrics like NDCG for learning-to-rank applications.
• It uses pairwise ΔNDCG weighting to prioritize swaps affecting top-list enrichment, enabling fine-grained, effective ranking improvements.
• Empirical evaluations show that LambdaLoss outperforms standard regression and RankSVM, especially in heterogeneous and multi-assay screening tasks.

LambdaLoss is a loss formulation for learning-to-rank applications that provides a principled, differentiable approximation to optimizing position-based ranking metrics, specifically Discounted Cumulative Gain (DCG) and its normalized variant NDCG. It is designed for use with models such as gradient boosting decision trees (GBDT), particularly in information retrieval and ligand-based virtual screening. LambdaLoss emphasizes proper ranking at the top of candidate lists by assigning pairwise weights that directly correspond to the ΔNDCG impact of swapping item pairs, thereby aligning model optimization more closely with practical enrichment objectives (Furui et al., 2022).

1. Mathematical Formulation

LambdaLoss, also known as “NDCGLoss2” in the LightGBM implementation, operates at the group (query/assay) level, where each group contains $N$ items with integer relevance labels $y = (y_1, ..., y_N)$ and model scores $s = (s_1, ..., s_N)$. The core constructs are:

• Gain: $gain_i = 2^{y_i} - 1$
• Discount: $D_i = \log_2(i + 1)$
• DCG@K: $\mathrm{DCG}@K = \sum_{i=1}^K \frac{gain_i}{D_i}$
• maxDCG@K: Optimal DCG@K with labels sorted descending.
• NDCG@K: $$\mathrm{NDCG}@K = \frac{\mathrm{DCG}@K}{\mathrm{maxDCG}@K}$$
• Expected random DCG: Using mean gain, $randomDCG@K = \sum_{i=1}^K \frac{gain_{mean}}{D_i}$
• NEDCG@K: $$\mathrm{NEDCG}@K = \frac{\mathrm{DCG}@K - randomDCG@K}{maxDCG@K - randomDCG@K}$$, with $0$ as random-level performance and $y = (y_1, ..., y_N)$0 as perfect ranking.

The LambdaLoss objective is formulated via a pairwise cost over all pairs with $y = (y_1, ..., y_N)$1:

$y = (y_1, ..., y_N)$2

Here, $y = (y_1, ..., y_N)$3 is user-settable (typically $y = (y_1, ..., y_N)$4), and the pairwise weight $y = (y_1, ..., y_N)$5 encodes the impact on NDCG:

$y = (y_1, ..., y_N)$6

with the discount swap effect:

$y = (y_1, ..., y_N)$7

These weights ensure that model updates are prioritized for pairs whose swap most influences early list enrichment.

2. Gradient and Hessian Computation in GBDT

Efficient integration into GBDT frameworks such as LightGBM is accomplished by explicit derivation of gradients and Hessians (pseudo-responses). The gradient for item $y = (y_1, ..., y_N)$8 sums contributions from all pairs where $y = (y_1, ..., y_N)$9 should be ranked above or below $s = (s_1, ..., s_N)$0:

$s = (s_1, ..., s_N)$1

The diagonal Hessian is analogously constructed using the term:

$s = (s_1, ..., s_N)$2

These quantities are used as pseudo-responses for second-order boosting, enabling direct optimization of the LambdaLoss objective with GBDT.

3. GBDT Optimization Workflow

The boosting procedure is as follows:

1. Negative gradient and Hessian computation: Using current scores $s = (s_1, ..., s_N)$3.
2. Tree growth: Fit a regression tree $s = (s_1, ..., s_N)$4 to the negative gradients, using the Hessian as instance weights. Leaf output minimizes:

$s = (s_1, ..., s_N)$5

with $s = (s_1, ..., s_N)$6 for L₂ regularization.

1. Leaf output update: $s = (s_1, ..., s_N)$7
2. Score update: $s = (s_1, ..., s_N)$8, using learning rate $s = (s_1, ..., s_N)$9.
3. Validation and early-stopping: Based on metrics such as NDCG@10 on a held-out set; boosting halts if no improvement over $gain_i = 2^{y_i} - 1$0 rounds.

This framework allows LambdaLoss to be efficiently and scalably used in practical multi-stage ranking pipelines.

4. Empirical Evaluation and Comparative Performance

Experiments focused on ligand-based virtual screening with complex multi-assay and simple single-assay settings, using compound/protein features and several learning-to-rank methods.

Model	NDCG@10 (D1)	NEDCG@10 (D1)	NDCG@10 (D2)	NEDCG@10 (D2)
GBDT+LambdaLoss	0.593	0.286	0.342	0.340
GBDT+LambdaRank	0.543	0.212	—	—
RankSVM	0.465	0.077	0.368	0.366
GBDT Regression	0.404	-0.056	0.239	0.236

In multi-assay tasks (Dataset 1), LambdaLoss exhibited the highest scores on NDCG@10 and NEDCG@10, substantially outperforming both standard GBDT regression (which fell below random-level performance, NEDCG<0) and classical RankSVM. This indicates that LambdaLoss’s explicit pairwise ΔNDCG weighting enables discrimination at the top of the ranked list despite heterogeneity in activity scales across assays.

For single-assay settings (Dataset 2), all methods performed above random. Here, regression was competitive at very small $gain_i = 2^{y_i} - 1$1 (NDCG@1), but LambdaLoss matched or exceeded regression by $gain_i = 2^{y_i} - 1$2.

A plausible implication is that LambdaLoss confers specific advantages in scenarios where label ranges are not directly comparable between queries, while in uniform settings, its improvement relative to regression narrows.

5. Role of Evaluation Metrics: NEDCG vs NDCG

Evaluation of ranking performance relied on the traditional NDCG@K and the proposed Normalized Enrichment Discounted Cumulative Gain (NEDCG).

• NDCG@K measures ranked retrieval versus the ideal but is always $gain_i = 2^{y_i} - 1$3 and can obscure poor performance, as it does not indicate when a model is worse than random.
• NEDCG@K is defined such that $gain_i = 2^{y_i} - 1$4 corresponds to random-level ranking and $gain_i = 2^{y_i} - 1$5 to perfect, making it sensitive to negative enrichment and truly random performance.

The adoption of NEDCG exposed that regression-based methods may yield negative enrichment (NEDCG<0) in multi-assay settings, a fact that NDCG’s limited range cannot reveal. This suggests NEDCG is more informative in complex, heterogeneous virtual screening.

6. Hyperparameters and Practical Implementation

LightGBM’s implementation of LambdaLoss (NDCGLoss2) was employed in all experiments. Key hyperparameters included:

• num_leaves: $gain_i = 2^{y_i} - 1$6
• min_data_in_leaf: $gain_i = 2^{y_i} - 1$7
• feature_fraction: 0.7, bagging_fraction: 1.0, bagging_freq: 0
• learning_rate: 0.1 (tuning), 0.05 (final)
• lambdarank_truncation_level: 30 (Dataset 1), 200 (Dataset 2)
• label_gain step width: 1.0 (D1), 0.01 (D2)

Tuning used early stopping on NDCG@10 for complicated assays and NDCG@10% for large simple assays. Enrichment was robust to the number of leaves and data per leaf, but specific levels for truncation and label gain were explored for each scenario.

7. Significance and Context in Learning-to-Rank

LambdaLoss advances learning-to-rank by direct optimization of a metric (NDCG) closely aligned with top-K retrieval. Its key distinction is weighting each pair’s loss by the precise impact on the NDCG objective, focusing the model’s capacity where it is most consequential for enrichment, especially under label and scale heterogeneity across queries or assays (Furui et al., 2022).

This method addresses limitations of both unweighted pairwise losses (such as RankSVM) and standard regression, which may underperform in multi-context ranking by failing to standardize or prioritize top-list performance. LambdaLoss retains efficient computation and integration with tree boosting, making it suitable for large-scale, real-world compound screening and retrieval systems with complex assay structures.

A plausible implication is that LambdaLoss may be beneficial in any ranking domain where enrichment, early retrieval, and heterogeneous query labels play a central role.