Contrastive-center Loss for Classification

• Contrastive-center Loss is an auxiliary supervision objective that promotes intra-class compactness and inter-class separability by using a ratio of distances in feature space.
• It augments standard softmax loss by penalizing the ratio between the distance to the correct class center and aggregated distances to other centers, streamlining training without sample mining.
• Empirical results on datasets like MNIST, CIFAR-10, and LFW indicate measurable gains over traditional losses, enhancing both classification accuracy and feature discrimination.

Contrastive-center loss is an auxiliary supervision objective designed for deep neural networks, most notably in image classification and face recognition tasks. It introduces “class centers” in feature space and enforces desirable intra-class compactness and inter-class separability by directly penalizing the ratio between the distance of a feature to its correct class center and the sum of its distances to all other class centers. This approach augments the standard softmax loss and enhances the discriminative quality of learned features, operating in a manner distinct from both classical center loss and contrastive loss approaches (Qi et al., 2017).

1. Formal Definition and Mathematical Structure

Let $m$ denote the mini-batch size, $k$ the number of classes, and $d$ the feature dimension. For each input sample $x_i \in \mathbb{R}^d$ with label $y_i \in \{1,\dots,k\}$ and class centers $c_j \in \mathbb{R}^d$, the loss terms are defined as follows:

• $A_i = \Vert x_i - c_{y_i} \Vert^2$ (squared distance to its corresponding true class center)
• $$B_i = \sum_{j \neq y_i} \Vert x_i - c_j \Vert^2 + \delta$$ (sum of squared distances to all other class centers, stabilized by constant $\delta > 0$)

The contrastive-center loss is defined as:

$L_{ctc} = \frac{1}{2} \sum_{i=1}^m \frac{A_i}{B_i}$

In practice, a joint loss is deployed:

$L = L_{softmax} + \lambda L_{ctc}$

where $L_{softmax}$ is the standard cross-entropy and $\lambda \geq 0$ scales the contribution of the auxiliary loss.

2. Mechanism: Intra-Class Compactness and Inter-Class Separability

The ratio $A_i / B_i$ couples the goals of intra-class compactness and inter-class separability in a single term:

• Intra-class compactness: The numerator, $A_i$, increases if an embedding $x_i$ deviates from its true class center $c_{y_i}$. Minimization directly contracts class clusters in feature space.
• Inter-class separability: The denominator, $B_i$, aggregates the squared distances to all non-corresponding class centers. If $x_i$ approaches an incorrect center, $B_i$ diminishes, inflating the loss and imposing a repulsive penalty that enhances class separation.
• Joint effect: Minimizing this ratio for all samples simultaneously tightens clusters and maximizes inter-cluster gaps. The geometric interpretation is that each loss term is the sample’s intra-center distance "normalized" by its inter-cluster distance, providing scale-invariant enforcement.

3. Optimization Dynamics: Gradients and Center Updates

Given the composite structure, gradients with respect to both network parameters (via $x_i$) and the learned centers $c_n$ are essential for training:

• Sample gradients:

$$\frac{\partial L}{\partial x_i} = \frac{x_i - c_{y_i}}{B_i} - \frac{A_i}{B_i^2} \sum_{j \neq y_i} (x_i - c_j)$$

The update pulls $x_i$ towards its center and pushes away from incorrect centers proportionally to $A_i$.

• Center gradients:

$$\frac{\partial L}{\partial c_n} = \sum_{i: y_i = n} \left[ -\frac{x_i - c_n}{B_i} \right] + \sum_{i: y_i \neq n} \left[ \frac{A_i (x_i - c_n)}{B_i^2} \right]$$

Each center is adjusted by a batch-accumulated term and updated after each batch via a distinct learning rate $\alpha$:

$$c_n \gets c_n - \alpha \frac{\partial L}{\partial c_n}$$

This optimization scheme decouples center motion from main network learning rates, stabilizing training.

4. Hyperparameterization and Training Practices

Three principal hyperparameters govern practical application:

• $\delta$ (denominator shift): Prevents division by zero; default $\delta=1$ is robust across settings.
• $\lambda$ (contrastive-center loss weight): Adjusts the relative influence of $L_{ctc}$; typical values are $\lambda \approx 0.1$ for generic classification (MNIST, CIFAR-10) and $\lambda \approx 1.0$ for verification (LFW, CASIA-WebFace). Higher $\lambda$ can cause over-separation, hurting primary classification metrics.
• $\alpha$ (center learning rate): Typically in the range $[0.1, 0.5]$, chosen to be smaller than main network learning rate to avoid center instability.

Tuning proceeds by setting $\delta=1$, initializing a moderate $\lambda$, and adaptively adjusting $\lambda$ and $\alpha$ based on validation performance and observed convergence behavior.

5. Training Algorithm: Stepwise Pseudocode and Workflow

The contrastive-center loss augments a standard deep learning training loop without necessitating sample mining. The typical training iteration per mini-batch is:

for each mini-batch {x_i, y_i}_{i=1…m}: # Forward pass f_i = CNN_forward(x_i) # features logits = W·f_i + b # classifier L_s = cross_entropy(logits, y_i) # primary loss # Compute contrastive-center loss for i in range(m): A_i = ||f_i – c[y_i]||^2 B_i = δ + sum_{j≠y_i} ||f_i – c[j]||^2 L_ctc += 0.5 * (A_i / B_i) L_ctc *= λ # Backward pass L_total = L_s + L_ctc optimizer_net.step() # Update class centers c_j with small learning rate α

Centers are initialized to zero or randomly, and update rules are carried out per expressions in section 3.

6. Empirical Performance and Quantitative Evidence

Across a series of benchmarks, contrastive-center loss demonstrates consistent improvements over both vanilla softmax classification and the original center loss:

Dataset/Task	Softmax	Center Loss	Contrastive-center Loss
MNIST (LeNets++)	98.80%	98.94%	99.17%
CIFAR-10 (ResNet)	91.25%	92.10%	92.45%
LFW (CASIA-WebFace)	97.47%	98.55%	98.68%

Visualization on low-dimensional MNIST features shows an order-of-magnitude increase in average inter-center distance ($\approx 50$ for contrastive-center vs. $10$–$15$ for center loss), confirming enhanced cluster separation (Qi et al., 2017).

7. Practical Considerations and Applicability

• Computational cost: Calculating $B_i$ is $O(kd)$ per sample, which may be prohibitive for extremely large $k$. In such settings, negative center sub-sampling is feasible.
• Robustness: The method requires no sample mining, unlike traditional contrastive or triplet loss, streamlining training implementation.
• Stability: Slow center updates are crucial; overly fast updates cause oscillation in center positions.
• Synergy: Contrastive-center loss can be combined seamlessly with any backbone architecture and head, including CNN, ResNet, or any classifier based on softmax or margin variants.
• Interpretation: The numerator-denominator form provides a clear geometric mechanism for enforcing compactness and separability without explicit reliance on sampling strategies.

The contrastive-center loss presents a straightforward yet effective auxiliary objective, yielding superior discriminative features and measurable gains in both image classification and face verification contexts, as substantiated by comprehensive comparative experiments (Qi et al., 2017).