Bilateral Softmax Loss (BSL)

• Bilateral Softmax Loss (BSL) is a robust loss function that symmetrically applies distributionally robust optimization and variance penalization to both positive and negative samples.
• It modifies the conventional softmax loss by using separate log-sum-exp aggregations with temperature parameters, leading to improved fairness and tolerance to label noise.
• BSL is implemented as a minimal drop-in replacement in deep learning frameworks and has been empirically validated in recommendation systems and feature learning benchmarks.

Bilateral Softmax Loss (BSL) is a principled loss function that builds on the conventional sampled softmax loss by symmetrically applying its robustifying structure to both positive and negative samples. BSL provides superior accuracy, robustness to label noise, and fairness by explicitly incorporating distributionally robust optimization (DRO) and variance penalization for both positive and negative data, with notable applications in recommendation systems and deep feature learning for classification. Its implementation requires minimal modification of existing code using softmax-based losses, making it attractive for practical adoption across a variety of architectures and data scenarios (Wu et al., 2023, Zhou et al., 2022).

1. Formal Definition and Mathematical Formulation

Let $f(u,i)\in\mathbb{R}$ denote a model score for user $u$ and item $i$. Define distributions $P^+_u$ and $P^-_u$ over positive and negative items (typically uniform on observed positives and sampled negatives).

The sampled softmax loss (SL) is:

$$L_{SL}(u) = - \mathbb{E}_{i\sim P^+_u}\left[ f(u,i) \right] + \tau \cdot \log \mathbb{E}_{j\sim P^-_u}\left[ \exp(f(u,j)/\tau) \right]$$

or, in normalized fashion:

$$L_{SL}(u) = - \mathbb{E}_{i\sim P^+_u}\left[ \log \frac{\exp(f(u,i)/\tau)}{ \exp(f(u,i)/\tau) + N^- \mathbb{E}_{j\sim P^-_u}\left[\exp(f(u,j)/\tau)\right]} \right]$$

Bilateral Softmax Loss (BSL) augments SL by applying the log-expectation-exp (log-sum-exp) structure to the positive term as well, with separate temperatures $\tau_1$ (positive) and $\tau_2$ (negative):

$$L_{BSL}(u) = -\tau_1 \cdot \log \mathbb{E}_{i\sim P^+_u}\left[ \exp(f(u,i)/\tau_1) \right] + \tau_2 \cdot \log \mathbb{E}_{j\sim P^-_u}\left[ \exp(f(u,j)/\tau_2) \right]$$

For sampled positives $u$0 and negatives $u$1:

$u$2

Standard SL is recovered as a special case when $u$3 and $u$4 (Wu et al., 2023).

2. Theoretical Underpinnings: DRO and Variance Penalization

SL can be interpreted as a worst-case optimization (DRO), providing robustness to distribution shifts and noise in negatives:

$u$5

The log-sum-exp negative term exactly realizes this DRO objective with $u$6 as the Lagrange multiplier.

BSL extends this DRO principle to positives, introducing a symmetric worst-case (max-over-KL-ball) reweighting around $u$7. This guards against both mislabeled negatives and positives, yielding robustness to both spurious negative and positive signals (Wu et al., 2023).

A second-order Taylor expansion for the log-sum-exp term reveals that both SL and BSL introduce implicit variance penalties:

$u$8

SL thus penalizes the variance among negative scores, shrinking the prediction gap between popular/unpopular items. BSL introduces a matching variance penalty term for positives, regularizing overconfidence in positive predictions and increasing robustness to noisy positives (Wu et al., 2023).

3. BSL in Margin-Based and Feature-Learning Frameworks

Recent work unifies BSL with margin-based deep feature learning objectives by defining two key geometric quantities:

• Class margin ($u$9): Smallest angular separation between class prototypes.
• Sample margin ($i$0): Logit gap between the true class and the highest competing class for a sample.

A generalized margin softmax (GM-Softmax) objective incorporates affine transforms of cosine/logit angles. BSL is realized by augmenting GM-Softmax with two regularizers:

• Sample-margin regularization ($i$1): Drives each sample’s true-class logit above its hardest negative.
• Zero-centroid regularization ($i$2): Enforces uniformity in class centers under imbalance.

The closed-form BSL objective in this context is:

$i$3

with $i$4 as the per-sample GM-Softmax loss (Zhou et al., 2022).

BSL recovers or subsumes prior heuristic margin-based losses (SphereFace, CosFace, ArcFace) by the appropriate choice of transformations, providing a more principled and unifying framework for discriminative feature learning and robust prediction (Zhou et al., 2022).

4. Implementation and Practical Guidelines

BSL can be adopted by replacing the averaged positive logits in SL with a log-sum-exp aggregation, requiring only one additional line in standard deep learning frameworks. In PyTorch-style pseudocode:

$i$9 The negative term remains unchanged. Typical hyperparameters are $i$5 and $i$6, with negative sampling budgets of 64–512. This minimal modification suffices for immediate deployment in any SL-based system (Wu et al., 2023).

For feature learning and classification, BSL is implemented as GM-Softmax plus weighted regularization with recommended weights $i$7 and $i$8 (Zhou et al., 2022).

5. Empirical Performance and Fairness Benefits

Empirical evaluations across diverse datasets and model architectures substantiate the effectiveness of BSL:

Backbone/Dataset	SL vs. Pointwise Losses	BSL vs. SL	Robustness to Noise
MF, LightGCN, NGCF	SL outperforms BPR, BCE, MSE by >15%	BSL improves SL by 2–5%	BSL tolerates up to 40% noisy positives with 5–15% performance drop
Yelp2018, Amazon, Gowalla, MovieLens-1M	Consistent relative wins	Universally positive improvement	Stable under negative noise; maintains performance where pointwise losses degrade

Group-wise analyses demonstrate a reduction in score variance and improved NDCG for tail (unpopular) items, confirming enhanced fairness in recommendations (Wu et al., 2023). In feature learning benchmarks (MNIST, CIFAR-10/100, Market1501/DukeMTMC, various face verification datasets), BSL raises class margins, improves top-1 accuracy, and maintains large inter-class separations, particularly benefitting imbalanced data (Zhou et al., 2022).

6. Adoption, Extensions, and Related Losses

BSL generalizes and unifies a range of popular softmax and margin-based losses. In the class-imbalanced regime, zero-centroid regularization restores regular simplex optima and large class margins. For balanced data, sample-margin regularization alone suffices to maximize margin and representation quality.

By situating BSL within the DRO and variance penalization framework, and by aligning its implementation with standard softmax practices, the loss function is broadly compatible with a variety of recommendation and representation learning systems. BSL’s close connection to existing methods such as SphereFace, CosFace, and ArcFace suggests its utility as a drop-in replacement with superior theoretical guarantees and practical results on metrics such as Recall@20, NDCG@20, and accuracy (Wu et al., 2023, Zhou et al., 2022).