Virtual-Target-based Representation Regularization

• VRR is a representation learning framework that uses learnable virtual target codes to enhance discrimination and regularize network features.
• It integrates margin-based triplet and correlation-consistency losses to enforce tight intra-class clustering and clear inter-class separation.
• Empirical evaluations on fine-grained, imbalanced, and retrieval benchmarks show that VRR significantly improves classification and feature retrieval performance.

Virtual-Target-based Representation Regularization (VRR), also referred to as Learnable Target Coding (LTC), is a representation learning framework designed to improve the discriminative power and geometric structure of deep neural representations. Unlike traditional approaches that employ fixed target codes (such as one-hot encoding or Hadamard codes), VRR introduces a set of learnable virtual target codes that are simultaneously optimized with network parameters. Through a combination of margin-based triplet and correlation-consistency loss functions, VRR leverages these learnable codes as anchors in a high-dimensional code space, imposing geometric constraints and inter-class decorrelation that enhance both classification and retrieval performance (Liu et al., 2023).

1. Learnable Code Construction

VRR defines a learnable code matrix

$$W = [w_1; w_2; \dots; w_C] \in \mathbb{R}^{C \times L}$$

where each class $k$ is associated with a “pre-code” vector $w_k \in \mathbb{R}^L$. At each forward pass, codes are binarized via elementwise sign: $t_k = \mathrm{sgn}(w_k) \in \{\!-\!1, +1\}^L$ producing the virtual targets $T = \{t_1, \ldots, t_C\}$.

The code-length $L$ is a hyperparameter, commonly set to values such as 512. Codes are initialized randomly (Gaussian or uniform), and optimized jointly with the neural network. Back-propagation through the non-differentiable sign operator is achieved via a straight-through estimator: $$\frac{\partial\,\mathrm{sgn}(w)}{\partial w}\approx\mathrm{clip}\!\bigl(\nabla,-1,1\bigr), \quad \nabla = \frac{\partial L}{\partial \mathrm{sgn}(w)}$$ ensuring gradient updates propagate to $W$.

2. Loss Functions and Objective

Several loss components regulate the behavior of both the representation encoder and the target code matrix:

• Cross-Entropy Loss: Standard classification loss employing one-hot supervision:

$$L_{\mathrm{CE}} = -\frac{1}{N} \sum_{i=1}^{N} \log p_{i, y_i}$$

where $p_{i, y_i}$ is the softmax output.

• Mean-Squared Coding Loss: Enforces the semantic encoder output $v_i = \Phi_s(z_i)$ ($z_i$ from backbone feature extractor) to be close to its class-specific target code:

$$L_{\mathrm{MSE}} = \frac{1}{N L} \sum_{i=1}^N \|v_i - t_{y_i}\|_2^2$$

• Margin-Based Triplet Loss: In code space, for every sample and negative class,

$$L_{\mathrm{triplet}} = \frac{1}{N(C-1)} \sum_{i=1}^N \sum_{k\neq y_i} \max\left(v_i^\top t_k - v_i^\top t_{y_i} + m,\, 0 \right)$$

where $m$ is a margin hyperparameter.

• Correlation-Consistency Loss: Promotes near-orthogonality among all code pairs:

$$L_{\mathrm{corr}} = \frac{1}{C(C-1)} \sum_{k=1}^C \sum_{j\neq k} |\;t_k^\top t_j\;|$$

The total training objective is a weighted sum: $$L = L_{\mathrm{CE}} + \gamma L_{\mathrm{MSE}} + \lambda L_{\mathrm{triplet}} + \beta L_{\mathrm{corr}}$$ where $\gamma,\,\lambda,\,\beta$ control the contribution of each regularization.

3. Algorithmic Pipeline

VRR extends standard deep classification architectures by integrating a “semantic encoder” $\Phi_s$ mapping backbone features into code space and a learnable code matrix. Training proceeds as follows (see table for summary):

Step	Operation	Output
Feature Extraction	$z_i = \Phi_f(x_i; \theta_f)$	latent representation
Classification Output	$p_i = \Phi_c(z_i; \theta_c)$	softmax probabilities
Semantic Mapping	$v_i = \Phi_s(z_i; \theta_s)$	code-spaced semantic vector
Code Binarization	$t_k = \mathrm{sgn}(w_k)$ for all $k$	virtual target codes
Loss Computation	Compute $$L_{\mathrm{CE}}, L_{\mathrm{MSE}}, L_{\mathrm{triplet}}, L_{\mathrm{corr}}$$	all loss terms
Backpropagation	Update $\theta_f, \theta_c, \theta_s, W$	parameter optimization via SGD/Adam

The codebook and model parameters are updated jointly, exploiting the straight-through estimator for sign binarizations.

4. Geometric Interpretation and Representation Effects

The set of binarized class codes $T = \{t_k\}$ defines a set of anchor points—“virtual targets”—in the $L$-dimensional code (Hamming) space. The margin-based triplet loss drives each sample’s semantic code $v_i$ toward its class anchor $t_{y_i}$ while enforcing a minimum margin to all other class anchors. This regularizes representations by forming geometric “cones” centered on each class code, effectively enlarging inter-class margins.

The correlation-consistency loss ensures these class anchor codes spread as far apart as possible, favoring near-orthogonality, which geometrically sharpens class cluster separation. As a result, the combination of losses sculpts the representation manifold such that intra-class samples are tightly clustered and inter-class samples are maximally separated in the code space. Empirical t-SNE visualizations confirm that VRR yields clusters with tighter intra-class compactness and broader inter-class dispersion compared to both one-hot and fixed code schemes.

5. Experimental Results and Empirical Evaluation

VRR has been evaluated on a variety of challenging visual recognition and retrieval benchmarks:

• Fine-Grained Classification: On CUB-200-2011, Stanford Cars-196, and FGVC Aircraft-100 datasets using ResNet-18/34/50 backbones (pretrained on ImageNet), VRR exhibits consistent improvements in top-1 accuracy over one-hot encoding. For ResNet-50 on CUB, accuracy rises from 85.46% to 86.90%; on Cars, from 92.89% to 94.27%; on Aircraft, from 90.97% to 92.77%, using typical parameters $L=512, \gamma=1, \lambda=0.01, \beta=0.1, m=L$.
• Imbalanced Classification: On CIFAR-100-LT (100x imbalance), ImageNet-LT, and iNaturalist-18 datasets with ResNet-32/50, VRR demonstrates marked gains. For ImageNet-LT, the combination of VRR and deferred re-weighting (DRW) achieves 48.07% top-1 accuracy versus 43.69% for CE+DRW.
• Deep Metric Learning (Retrieval): On CUB and Cars datasets leveraging margin, multi-similarity, or circle loss, VRR increases recall@1, e.g., in Margin512 settings for CUB, rising from 65.2% to 66.4%.

These improvements underline VRR’s utility as an auxiliary regularization that synergistically exploits learnable code geometry for enhanced representation learning across both balanced and long-tailed recognition tasks (Liu et al., 2023).

6. Relation to Prior Coding Approaches

VRR distinguishes itself from conventional target coding techniques by learning the target anchor codes, rather than fixing them a priori. Fixed one-hot and Hadamard codes are less flexible in modelling inter-class correlation structures and lack mechanisms for adapting target geometry to dataset idiosyncrasies. VRR’s optimization of both anchor code positions and their orthogonality permits explicit control over the geometry of the class manifold and inter-class relationships, capturing latent dependencies in the data. A plausible implication is that this approach may generalize to other settings in which target structure should reflect data-driven statistical or semantic correlations.

7. Practical Implementation and Hyperparameter Choices

The practical realization of VRR involves augmenting the architecture with a small semantic encoder and code matrix, with overhead dependent on the code length $L$ and number of classes $C$. Typical settings involve code lengths of $L=512$, with loss term weights $\gamma=1$, $\lambda=0.01$, $\beta=0.1$, and triplet margin $m=L$. Training employs standard batch processing, and the simultaneous optimization of codebook and encoder parameters leverages established optimizers such as SGD or Adam. The source code is publicly available at https://github.com/AkonLau/LTC (Liu et al., 2023).