---
title: Dual Sigmoid Loss Function
url: https://www.emergentmind.com/topics/dual-sigmoid-based-loss-function
type: topic
---

# Dual Sigmoid Loss Function

A dual sigmoid-based loss function denotes any loss design employing two distinct sigmoid functions to scale, modulate, or reweight intra-class and inter-class terms—primarily in metric learning, representation learning, and robust classification. Its central paradigm is to independently modulate the optimization drive on “pulling” (same-class) and “pushing” (different-class) sample pairs, explicitly balancing within-class compactness and between-class separability under noise, class imbalance, or multi-modal data sources. Notable instantiations include the SFace “sigmoid-constrained hypersphere” loss—designed for robust face recognition—and the bias-and-temperature-reparametrized sigmoid contrastive loss used in SigLIP/SigLIP2 for multi-modal representation alignment [2205.12010], [2509.18552].

## 1. Mathematical Formulation

Dual sigmoid-based losses combine two independently parameterized sigmoid functions. In the case of SFace loss for face recognition, sample $i$ with embedding $x_i$ and class center $W_{y_i}$ (all $\ell_2$-normalized) produces intra- and inter-class angular terms:
\[
\theta_{y_i} = \arccos(W_{y_i}^\top x_i), \qquad \theta_j = \arccos(W_j^\top x_i),\quad j \neq y_i
\]
The per-sample loss is:
\[
L_\mathrm{SFace}(x_i, y_i) = L_\mathrm{intra}(\theta_{y_i}) + L_\mathrm{inter}(\{\theta_j\}_{j\neq y_i})
\]
where:
\[
\begin{aligned}
L_\mathrm{intra}(\theta_{y_i}) & = -[r_\mathrm{intra}(\theta_{y_i})]_b \cos\theta_{y_i} \\
L_\mathrm{inter}(\{\theta_j\}) & = \sum_{j\ne y_i} [r_\mathrm{inter}(\theta_j)]_b \cos\theta_j
\end{aligned}
\]
with $[\cdot]_b$ denoting “block-gradient” (stop-gradient). The scaling functions are sigmoids:
\[
\begin{aligned}
r_\mathrm{intra}(\theta)  &= \frac{s}{1 + \exp(-k[\theta - a])} \\
r_\mathrm{inter}(\theta)  &= \frac{s}{1 + \exp(+k[\theta - b])}
\end{aligned}
\]
where $s>0$ is a scale (commonly $64$), $k>0$ is the sharpness, and $a, b$ set the transition point for intra- and inter-class modulation, respectively [2205.12010].

In Sigmoid Contrastive Loss as introduced in SigLIP/SigLIP2 [2509.18552], two parameters ($t$ as inverse temperature and $b$ as bias, or $b_\mathrm{rel}=b/t$ as relative bias) control the separation of positive (matched) and negative (unmatched) pairs:
\[
\begin{aligned}
L(\theta,\phi; t,b_\mathrm{rel}) = &\sum_{i=1}^N \log\bigl(1+\exp[-t(\langle U_i, V_i \rangle-b_\mathrm{rel})]\bigr) \\
&+ \sum_{i\ne j} \log\bigl(1+\exp[t(\langle U_i, V_j \rangle - b_\mathrm{rel})]\bigr)
\end{aligned}
\]

## 2. Motivation and Geometric Intuition

Traditional loss strategies, such as Center Loss or softmax variants, enforce fixed rates for intra-class collapse and inter-class separation, often leading to overfitting or sensitivity to outliers and label noise. Dual sigmoid-based losses address these issues by:

- Applying moderate rather than absolute optimization pressure to examples already close to the class center or already well separated, mitigating overfitting especially on noisy or low-quality data [2205.12010].
- Allowing the training signal to dynamically diminish as optimization achieves compactness or separation, leading to stability and resistance to “freezing” in intra-class variance minimization as seen in Center Loss [2009.08796].

Geometric analysis reveals that, for hypersphere-based dual sigmoid loss, gradient magnitudes fall smoothly to zero in regions where further pulling/pushing provides little benefit, constraining model updates to the tangent spaces of the hypersphere and promoting both compactness and robustness.

## 3. Hyperparameterization and Gradient Properties

Each sigmoid scaling function is parameterized as follows:

| Parameter | Role                       | Typical Range          |
|-----------|----------------------------|------------------------|
| $s$       | Maximum scale (“speed”)    | $\sim64$               |
| $k$       | Sharpness (“sigmoid slope”)| $\sim80$               |
| $a$       | Intra-class center         | $[0.8, 0.84]\ (\mathrm{rad})$ |
| $b$       | Inter-class center         | $\sim1.28\ (\mathrm{rad})$|

Tuning $a$ upward (intra-class) makes strong pulling “turn off” for larger angular errors—beneficial under high label noise. $b$ is relatively stable.

Gradient computation proceeds via
\[
\frac{\partial L_{\mathrm{SFace}}}{\partial x_i} = - r_{\mathrm{intra}}(\theta_{y_i})\frac{\partial \cos\theta_{y_i}}{\partial x_i} + \sum_{j\neq y_i} r_{\mathrm{inter}}(\theta_j)\frac{\partial \cos\theta_j}{\partial x_i}
\]
ensuring that re-scaling applies only to component-wise cosine gradients, not to the parameters of the sigmoid re-scale functions themselves.

## 4. Comparative Performance and Ablation Analysis

Empirical ablation indicates that sigmoid re-scaling of both gradient terms outperforms both constant and piecewise-hard-threshold alternatives in deep face recognition:
- Constant scaling: 90.05% five-set mean accuracy
- Piecewise (step function): 94.64%
- Full sigmoid (SFace): 94.80%

The dual sigmoid approach maintains a larger gap in intra-class angular distribution between clean and noisy subsets (e.g., $\Delta\approx 7.6^\circ$ for SFace vs. $4.8^\circ$ for ArcFace), evidencing greater noise robustness [2205.12010].

On synthetic “random-sphere” data, employing explicit $t$ and $b_\mathrm{rel}$ parameters leads to faster convergence and larger separation margins, thereby improving retrieval robustness in contrastive representation learning [2509.18552].

## 5. Extension to Multi-Modal and Contrastive Frameworks

In contrastive settings (e.g., SigLIP), the temperature ($t$) and bias ($b_\mathrm{rel}$) parameters implicitly act as global sigmoidal thresholds, shaping the distributions of intra- and inter-modal similarity scores. The concept of $(m, b_\mathrm{rel})$–Constellations provides a rigorous combinatorial-geometric underpinning: global minimizers correspond to configurations where all positive pairs are separated from negatives by a margin $m$ shifted by $b_\mathrm{rel}$.

In practice:
- Large $t$ enforces sharp decision boundaries
- Trainable $b_\mathrm{rel}$ aligns positive and negative distributions, crucial for automatic separation across modalities and for closing the modality gap [2509.18552].
- Freezing $b_\mathrm{rel}$ enables explicit control of the induced margin; adapter-like effects can be achieved without additional network parameters.

## 6. Robustness to Dataset Noise and Implementation Guidelines

Dual sigmoid-based losses excel in environments where class labels are imperfect or some samples are systematically noisy. Practical recommendations include tuning the intra-class sigmoid center $a$ rightward (higher angle) as noise increases, and maintaining high $k$ for sharp control. Inter-class center $b$ is stable. This moderation prevents the over-pulling of noisy or uncertain instances, and prevents over-pushing once inter-class separability is sufficient.

The block-gradient variant is essential: it avoids back-propagating through the scaling functions themselves, maintaining the intended per-sample re-weighting.

Implementation typically requires:
- $\ell_2$ normalization of both features and class weights
- Per-sample computation of angular distances and cosine similarities
- Independent sigmoid re-scales with block-gradient semantics
- Batchwise aggregation and update

## 7. Theoretical Characterization and Geometric Limits

Under dual sigmoid-based regimes with free temperature and bias (contrastive), global minima are characterized by strict separation of positive and negative similarities—quantified by the $(m, b_\mathrm{rel})$–Constellation formalism. Combinatorial geometric arguments (via spherical codes) relate the embedding dimension $d$, achievable margin $m$, and dataset cardinality $N$, prescribing fundamental limits on deployable capacity and separation. Such dual-sigmoid structures explain empirically observed modality gaps and inform architectural or regularization choices [2509.18552].

---

The dual sigmoid-based loss function paradigm is now a core methodological tool in robust metric learning and representation alignment, facilitating nuanced control over intra-class compactness and inter-class separation, with demonstrated benefits for noise-robustness, convergence, and multi-modal retrieval accuracy [2205.12010], [2509.18552].

Source: https://www.emergentmind.com/topics/dual-sigmoid-based-loss-function