---
title: Sub-center ArcFace Loss
url: https://www.emergentmind.com/topics/sub-center-arcface-loss
type: topic
---

# Sub-center ArcFace Loss

Sub-center ArcFace Loss is a generalization of the ArcFace (Additive Angular Margin Loss) used primarily in deep face recognition and related embedding learning tasks. It extends ArcFace by representing each class with $K$ sub-centers (prototypes) instead of a single center, permitting more flexible modeling of intra-class variation and conferring substantial robustness to label noise and outliers in large-scale, unconstrained datasets.

## 1. Mathematical Definition

Given sample $x_i \in \mathbb{R}^d$ (ℓ₂-normalized, scaled by a constant $s>0$, typically $s=64$) with ground-truth class $y_i \in \{1,\ldots,N\}$, and learnable sub-centers $W \in \mathbb{R}^{d \times N \times K}$, where $W_{j,k}$ is the $k$-th sub-center for class $j$, the Sub-center ArcFace loss is defined as follows [1801.07698, 2010.05350]:

- All $W_{j,k}$ are ℓ₂-normalized, $\|W_{j,k}\|=1$.
- Cosine similarity between $x_i$ and $W_{j,k}$:
  \[
  S_{j,k}(x_i) = W_{j,k}^\top x_i
  \]
- For each class, pool over sub-centers:
  \[
  S_j(x_i) = \max_{1 \leq k \leq K} S_{j,k}(x_i)
  \]
- Compute the positive angle:
  \[
  \theta_{y_i} = \arccos S_{y_i} (x_i)
  \]
- Define the target logit with angular margin $m>0$:
  \[
  \text{logit}_{j}(x_i) = 
  \begin{cases}
    s \cdot \cos(\theta_{y_i} + m), &\text{if } j=y_i \\
    s \cdot S_j (x_i), &\text{otherwise}
  \end{cases}
  \]
- Compute cross-entropy loss:
  \[
  L_i = -\log\frac{ \exp[s \cdot \cos(\theta_{y_i} + m)]}{ \exp[s \cdot \cos(\theta_{y_i} + m)] + \sum_{j \neq y_i} \exp[s \cdot S_j(x_i)] }
  \]

For mini-batch training, this loss is averaged over all $i$. The formulation easily generalizes standard ArcFace (set $K=1$).

## 2. Geometric and Statistical Intuition

In standard ArcFace ($K=1$), each class is restricted to a single prototype on the hypersphere, compelling all class samples—regardless of pose, lighting, or noise—to cluster about a single vector. This renders ArcFace susceptible to degradation in the presence of outliers and noisy labels, which can distort the class prototype.

With $K > 1$ sub-centers, each class forms up to $K$ distinct “modes” on the unit hypersphere. Clean, frontal, or canonical examples self-organize around the main (dominant) sub-center, while atypical, hard, or mislabeled instances attach to secondary (non-dominant) sub-centers. Each sample is assigned to its closest sub-center via $\max_k S_{j,k}$, isolating anomalies from the dominant mode and preserving intra-class compactness within meaningful sub-clusters [1801.07698].

This mechanism naturally encourages self-organizing cluster separation: the dominant sub-center accrues clean data, non-dominant sub-centers attract residual variation (e.g., pose, occlusion), and the angular margin $m$ maintains local angular discriminability. In effect, most network gradient updates from clean samples reinforce the dominant mode, while gradients from outliers are sequestered to non-dominant modes, mitigating distortion of embeddings.

## 3. Algorithmic Implementation

A typical training pipeline for Sub-center ArcFace comprises the following steps [1801.07698, 2010.05350]:

1. Produce feature embeddings: $z_i = f(\mathrm{image}_i; \theta)$; normalize and scale $x_i = z_i / \| z_i \| \cdot s$.
2. ℓ₂-normalize all sub-centers: $\widehat{W}_{j,k} = W_{j,k}/\|W_{j,k}\|$.
3. For each input $x_i$, compute $S_{j,k}^{(i)} = \widehat{W}_{j,k}^\top x_i / s$ for all $j,k$.
4. For each class $j$, pool $S_j^{(i)} = \max_k S_{j,k}^{(i)}$.
5. Compute positive logit for class $y_i$ using angular margin: $s \cdot \cos(\arccos S_{y_i}^{(i)} + m)$; negatives receive $s \cdot S_j^{(i)}$.
6. Apply softmax and cross-entropy to compute loss for $x_i$.
7. Back-propagate loss, update $\theta$ and $W$ via SGD/Adam, and re-normalize $W_{j,k}$.

For scalable training (millions of classes), a center-parallel strategy can distribute $W$ across GPUs. In frameworks such as PyTorch or TensorFlow, replace the final weight tensor $W\in\mathbb{R}^{d\times N}$ with $W\in\mathbb{R}^{d\times N\times K}$, inserting a max-over-$K$ prior to logit computation.

## 4. Selection and Effects of Sub-center Count $K$

| $K$ Value | Typical Regime           | Effect on Learning                  |
|:---------:|:------------------------|:------------------------------------|
| 1         | Low-noise, small-scale  | Reduces to ArcFace (single center)  |
| 2–5       | Medium-to-high noise    | Enhances robustness, preserves compactness |
| ≥10       | Large data, low utility | Sub-centers sparsely used, performance degrades |

Experiments advocate $K=3$ as a practical default on noisy data (label noise $\gtrsim 50\%$) [1801.07698, 2010.05350]. For “Web” datasets or massive class count, $K=3$–5 yields best trade-off between noise isolation and discriminative power. $K \geq 10$ is discouraged due to weakened intra-class margins and sparse assignment, while for clean, small-scale regimes $K=1$ or $2$ is sufficient. Cross-validation for $K\in\{2,3,5\}$ on a representative validation subset is recommended.

## 5. Empirical and Comparative Results

Sub-center ArcFace consistently outperforms standard ArcFace (K=1) in both face and landmark recognition under noisy or imbalanced conditions. Reported improvements, using verification TPR@FPR=$1\text{e}{-4}$ on IJB-C and retrieval GAP for landmarks, are tabulated below.

| Task/Dataset                 | Baseline (K=1) | Sub-center ArcFace | Post-Drop/Filtering |
|:-----------------------------|:--------------:|:------------------:|:-------------------:|
| MS1MV0 (noisy), face (IJB-C) | 90.27%         | 93.72% (+3.45)     | 95.92% (+5.65)      |
| MS1MV3 (clean), face         | 96.50%         | —                  | —                   |
| Celeb500K (50% noise)        | 92.15%         | 96.91%             | —                   |
| Google Landmark, val GAP     | ∼0.84          | ∼0.85              | —                   |
| Google Landmark, dinamic m   | —              | 0.8671             | —                   |

These results demonstrate that sub-center ArcFace nearly recovers or exceeds the performance of manually cleaned training, with ≈3–6% boost on standard benchmarks and robust generalization under label noise [1801.07698, 2010.05350]. On the highly imbalanced GLDv2 dataset, sub-center ArcFace with $K=3$ and dynamic margin yielded a +0.026 validation GAP over constant-margin ArcFace.

## 6. Extensions: Dynamic Margin Schedules

To address extreme class imbalance (e.g., long-tail distributions in landmark recognition), a “dynamic margin” strategy modulates the angular margin $m$ per class according to sample count $n$, via $m(n) = a n^{-\lambda} + b$ (clipped to $[m_\text{min}, m_\text{max}]$). Hyperparameters $(a, b, \lambda)$ are set so $m(n_\text{min})=m_\text{max}$, $m(n_\text{max})=m_\text{min}$, with recommended $\lambda = 1/4$ and $m \in [0.05, 0.50]$ for large-scale, imbalanced datasets [2010.05350]. This approach improves generalization, notably in tail classes.

## 7. Practical Considerations and Best Practices

- **Initialization**: Sub-centers can be initialized using standard Xavier/He initialization; no extra cluster collapse or orthogonality regularization is required, as diversity emerges during learning [2010.05350].
- **Embedding normalization and scaling**: Set $s=64$, $m=0.5$ by default, in line with ArcFace settings [1801.07698].
- **Training schedule**: Progressive fine-tuning over increasing image resolutions and class subsets is beneficial for complex tasks [2010.05350].
- **Noise filtering**: After initial training, discard samples with angular distance $>75^\circ$ to nearest sub-center. Optionally, drop non-dominant sub-centers and retrain on purified data [1801.07698].
- **Hardware scaling**: For million-class regimes, distribute sub-centers with center-parallel sharding [1801.07698].
- **Data augmentation**: In landmark recognition, excessive augmentation can harm retrieval accuracy [2010.05350].
- **Memory trade-offs**: Higher $K$ increases memory demands and may require careful balancing against batch size and class coverage.

Sub-center ArcFace can be seamlessly integrated into existing deep face recognition and descriptor learning pipelines, yielding substantial noise robustness and easy adaptation for class-imbalanced applications [1801.07698, 2010.05350].

Source: https://www.emergentmind.com/topics/sub-center-arcface-loss