Contrastive Learning through Auxiliary Branch (CLAB)

• The paper introduces CLAB, a method that leverages an auxiliary contrastive branch during training to significantly enhance video object detection robustness.
• It integrates intermediate backbone features with a 128-dimensional projection and uses an InfoNCE loss, achieving up to +2.0 mAP improvement on ImageNet VID.
• Dynamic loss weighting gradually decays the contrastive loss's influence, ensuring robust representations under conditions like motion blur, occlusion, and deformation.

Searching arXiv for the CLAB paper and closely related auxiliary-branch contrastive learning work. Contrastive Learning through Auxiliary Branch (CLAB) denotes a training-time architectural strategy in which a primary model is augmented with an auxiliary branch optimized by a contrastive objective, with the auxiliary components removed or de-emphasized for inference. In its named form for video object detection, CLAB refers specifically to the method introduced in "Contrastive Learning through Auxiliary Branch for Video Object Detection" (Rakotoarivony, 28 Aug 2025), where a video detector is equipped with a contrastive auxiliary branch to improve backbone feature robustness under motion blur, occlusion, and deformable shapes, while incurring zero extra computational cost at test time. More broadly, CLAB belongs to a larger family of auxiliary-branch contrastive methods in which branch-specific objectives regularize shared representations, including scale-specific auxiliary decoders for vessel segmentation (Sadikine et al., 2024), a contrastive learning branch for long-tailed recognition (Chen et al., 2023), auxiliary contrastive supervision at intermediate layers (Zhang et al., 2022), and related dual-branch or sub-branch formulations in remote sensing and point-cloud representation learning (Ji et al., 24 Mar 2026, Wu et al., 30 May 2025).

1. Concept and scope

In the video object detection formulation, CLAB is introduced to address a recurrent limitation of prior video detectors: performance improvements have often depended on feature aggregation and complex post-processing techniques, increasing inference-time computation (Rakotoarivony, 28 Aug 2025). The defining intervention is therefore not a new inference pipeline, but a training-time regularizer attached to the detector backbone.

The method is built on a representative video object detection pipeline, specifically SELSA + Temporal ROI Align (TROI), itself based on Faster R-CNN with ResNet-101 or ResNeXt-101 backbones (Rakotoarivony, 28 Aug 2025). During training, an auxiliary branch receives intermediate backbone features and produces 128-dimensional encoded vectors used in a contrastive objective. The standard detection heads remain unchanged. At test time, the auxiliary branch is removed, and only the base detector is used (Rakotoarivony, 28 Aug 2025).

This design places CLAB within a technically distinct subclass of contrastive learning methods. Unlike self-supervised two-branch frameworks that contrast two augmented views throughout training and inference logic, CLAB uses contrastive learning as an auxiliary constraint on a supervised detector. Unlike auxiliary heads in conventional multi-task learning, its auxiliary objective is not another supervised prediction task but an instance-discrimination criterion over video frames (Rakotoarivony, 28 Aug 2025). This suggests that CLAB is best understood as a training-time representational regularizer rather than a persistent multi-head architecture.

2. Core architecture in video object detection

The CLAB framework introduces a Contrastive Auxiliary Branch (CAB) into the backbone during training (Rakotoarivony, 28 Aug 2025). For each minibatch, $N$ videos are sampled, and from each video $T$ frames are selected, including a target frame and support frames. The backbone extracts intermediate feature maps for each frame; these are sent both to the normal detection pathway and to the CAB (Rakotoarivony, 28 Aug 2025).

The CAB has a simple projection architecture:

Component	Operation
Feature adaptor	$3 \times 3$ convolution + ReLU
Spatial reduction	Adaptive average pooling
Projection MLP	FC $512 \rightarrow 256$ + ReLU, then $256 \rightarrow 128$ + ReLU

The CAB output is a 128-dimensional projection per frame (Rakotoarivony, 28 Aug 2025). The stated rationale for using an intermediate feature rather than a final one is that it encourages global representations and helps gradient flow, mitigating vanishing gradients (Rakotoarivony, 28 Aug 2025).

The detector branch continues to optimize the standard object detection losses, while the CAB contributes an auxiliary contrastive loss. Because the auxiliary branch operates only during training and is removed at inference, the method preserves the runtime profile of the original detector (Rakotoarivony, 28 Aug 2025). This training/inference asymmetry is one of the most characteristic properties of CLAB.

A plausible implication is that CLAB is especially attractive in deployment settings where latency budgets preclude temporal ensembling, memory banks at test time, or post-processing stages, but offline training complexity remains acceptable.

3. Contrastive objective and dynamic loss weighting

The contrastive supervision in CLAB uses an InfoNCE loss over CAB outputs (Rakotoarivony, 28 Aug 2025). Positive pairs are frames from the same video; negatives are frames from different videos in the minibatch. If $\mathbf{z}_i$ denotes the encoded feature of the $i$-th frame, $\mathbf{z}_{i'}$ a positive from the same video, and $\tau$ the temperature parameter, the auxiliary loss is

$$\mathcal{L}_{\mathrm{aux}} = -\frac{1}{N}\sum_{i=1}^N \log \frac{ \exp \left( \mathrm{sim}(\mathbf{z}_i, \mathbf{z}_{i'})/\tau \right) }{ \sum_{k=1}^{N \times T} \mathbf{1}_{[k \neq i]} \exp\left( \mathrm{sim}(\mathbf{z}_i, \mathbf{z}_k)/\tau \right) }$$

where $T$0 is cosine similarity (Rakotoarivony, 28 Aug 2025). The objective pulls together features of frames from the same video and pushes apart features from different videos. In the paper’s interpretation, this makes the detector’s features more robust to degradation and pose variation (Rakotoarivony, 28 Aug 2025).

A central refinement is the dynamic loss weighting (DLW) strategy. Rather than keeping the auxiliary contrastive loss active at a fixed magnitude, CLAB weights it highly early in training and decays the weight to zero as training proceeds (Rakotoarivony, 28 Aug 2025). The schedule is

$T$1

and the total loss is

$T$2

with $T$3 the bounding-box regression loss and $T$4 the classification loss (Rakotoarivony, 28 Aug 2025). The practical parameters reported are an initial $T$5, decaying to zero at $T$6 steps in a 7-epoch training schedule (Rakotoarivony, 28 Aug 2025).

This scheduling principle has close conceptual parallels in other auxiliary-branch contrastive systems. DB-LTR uses a dynamic loss weighting scheme in which the total objective is $T$7, and $T$8 decreases over time so training gradually shifts from the imbalanced branch to the contrastive learning branch (Chen et al., 2023). Although the application domain differs, both methods use temporal loss rebalancing to prevent the auxiliary branch from dominating late optimization.

4. Empirical results in ImageNet VID

CLAB is evaluated on the ImageNet VID dataset, which in the reported setup contains 30 categories, 3,862 training videos, and 555 validation videos (Rakotoarivony, 28 Aug 2025). Images are resized with short side 600 px; training uses $T$9 videos per batch and $3 \times 3$0 frames each; inference uses 30 support frames; and training data combines VID with matching 30 classes from ImageNet DET using the “FGFA” split (Rakotoarivony, 28 Aug 2025).

The principal results are reported on ImageNet VID validation:

Backbone	TROI	CLAB
ResNet-101	82.0	84.0
ResNeXt-101	84.3	85.2

With ResNet-101, CLAB reaches 84.0% mAP; with ResNeXt-101, 85.2% mAP, achieving state-of-the-art performance for CNN-based models without additional post-processing methods (Rakotoarivony, 28 Aug 2025). The comparison table in the paper also lists FGFA, MANet, STSN, SELSA, MEGA, HVRNet, and BoxMask, with CLAB outperforming all listed CNN-based methods in the reported setting (Rakotoarivony, 28 Aug 2025).

The ablation study isolates the effect of the auxiliary branch and the dynamic weighting strategy:

Method	CAB	DLW	mAP (%)
TROI baseline			82.0
CLAB	✓		83.4
CLAB	✓	✓	84.0

Thus, adding CAB yields a $3 \times 3$1 mAP improvement over the TROI baseline, and adding DLW on top yields a further $3 \times 3$2, for a net gain of $3 \times 3$3 mAP with no extra inference cost (Rakotoarivony, 28 Aug 2025).

Hyperparameter ablations further report that $3 \times 3$4 is best among the tested temperatures, producing 84.0% mAP, whereas 0.05 gives 83.1, 0.5 gives 83.5, and 1 gives 83.6 (Rakotoarivony, 28 Aug 2025). For the auxiliary loss weight, 0.005 is best at 84.0%; 0.001 gives 83.1, 0.01 gives 83.1, and 0.05 gives 82.7 (Rakotoarivony, 28 Aug 2025).

Qualitative results in the paper indicate greater robustness under heavy blur, rare poses, occlusions, and clutter, where CLAB gives correct predictions in cases where the baseline yields misclassification or low scores (Rakotoarivony, 28 Aug 2025).

5. Relation to broader auxiliary-branch contrastive learning

Although the acronym CLAB is specific to video object detection in (Rakotoarivony, 28 Aug 2025), the underlying pattern appears across several research areas. The common structure is a shared backbone or encoder, one or more auxiliary branches, a contrastive or contrastive-like loss, and a representational benefit that transfers to the primary task.

In liver vessel segmentation, a standard 3D UNet is extended to multi-task learning with scale-specific auxiliary decoders $3 \times 3$5, $3 \times 3$6, each trained on a scale-specific mask derived from clustering vessel branches by estimated radius (Sadikine et al., 2024). Projection heads attached to auxiliary decoder features produce normalized embeddings $3 \times 3$7, and the contrastive loss encourages intra-scale compactness and inter-scale separation (Sadikine et al., 2024). The total training loss is

$3 \times 3$8

where $3 \times 3$9 controls contrastive loss strength (Sadikine et al., 2024). On 3D-IRCADb, the version with $512 \rightarrow 256$0 improves DSC, Jaccard, and clDSC over the $512 \rightarrow 256$1 variant, with clDSC increasing from 46.90 to 48.13 (Sadikine et al., 2024). This is closely analogous to CLAB’s ablation logic: an auxiliary branch exists first as a multi-task component, and the contrastive term is then shown to add measurable value.

In long-tailed recognition, DB-LTR comprises an imbalanced learning branch and a Contrastive Learning Branch (CoLB) sharing the same backbone (Chen et al., 2023). CoLB focuses on tail and medium-shot classes, uses prototype-based metric learning, and optimizes both intra-branch and inter-branch contrastive losses (Chen et al., 2023). Its purpose is to improve adaptation to tail classes and assist the main branch in learning a well-represented feature space and discriminative decision boundary (Chen et al., 2023). Here the auxiliary branch is not discarded at inference in the same way as CLAB, but the shared-backbone regularization principle is directly related.

In "Contrastive Deep Supervision" (Zhang et al., 2022), auxiliary projection heads are attached to intermediate layers, and those intermediate layers are supervised by augmentation-based contrastive learning rather than task cross-entropy. The authors argue that standard deep supervision conflicts with the tendency of shallow layers to learn low-level features, whereas contrastive supervision is better aligned with intermediate-layer function (Zhang et al., 2022). The auxiliary heads can be discarded at test time, yielding no extra inference cost (Zhang et al., 2022). Among currently available analogues, this is one of the closest conceptual relatives of CLAB: both methods use contrastive auxiliary pathways purely during training to regularize a primary supervised model.

Point-cloud self-supervised learning offers a structurally different but still relevant variant. PoCCA divides each main branch into global and local sub-branches and introduces information exchange between branches before the final loss computation (Wu et al., 30 May 2025). Rather than a single auxiliary head attached to a supervised detector, PoCCA is a cross-branch fusion framework for unsupervised representation learning. Still, its use of sub-branches to enrich contrastive interactions places it in the same architectural lineage.

Few-shot remote sensing provides two more examples. GenCo uses an auxiliary generator within a MoCo-v2-like pretraining framework, generating feature-space positives $512 \rightarrow 256$2 that participate in the contrastive loss and are later reused in fine-tuning to enrich scarce labeled data (Wu et al., 2023). DCN introduces two auxiliary supervised contrastive branches, CCL and DCL, respectively targeting inter-class discriminability and intra-class invariance through context and detail feature domains (Ji et al., 24 Mar 2026). These works broaden the auxiliary-branch contrastive paradigm beyond detection and segmentation.

6. Mechanisms, interpretation, and recurring design principles

Across the cited literature, several recurrent design principles define auxiliary-branch contrastive learning.

First, the auxiliary pathway typically acts on an intermediate or specialized representation rather than the final task head. CLAB feeds intermediate backbone features to CAB (Rakotoarivony, 28 Aug 2025). Contrastive Deep Supervision attaches projection heads to intermediate layers (Zhang et al., 2022). The liver vessel method attaches projection heads to auxiliary decoder features (Sadikine et al., 2024). This repeated choice suggests that auxiliary contrastive learning is often intended to shape representational geometry before final task specialization.

Second, auxiliary contrastive objectives usually encode a domain-specific notion of invariance or discrimination. In CLAB, positives are frames from the same video, reflecting temporal continuity under degradation (Rakotoarivony, 28 Aug 2025). In vessel segmentation, positives share the same anatomical scale and negatives come from other scales (Sadikine et al., 2024). In DB-LTR, inter-branch contrastive loss pushes tail features away from head features (Chen et al., 2023). In DCN, context-guided and detail-guided branches target inter-class separability and intra-class consistency, respectively (Ji et al., 24 Mar 2026). Thus, the auxiliary branch is not merely extra capacity; it operationalizes a carefully chosen relational prior.

Third, many such methods separate training-time regularization from inference-time deployment. CLAB removes CAB at test time (Rakotoarivony, 28 Aug 2025). Contrastive Deep Supervision discards intermediate projection heads (Zhang et al., 2022). C2AL, though not a contrastive branch in the same representation-learning sense, also keeps only the primary head at serving time while auxiliary heads are used only in training (Cokbas et al., 2 Oct 2025). This suggests a broader systems-level pattern: auxiliary branches are often used to improve optimization and learned representation while preserving serving efficiency.

Fourth, loss scheduling or loss balancing is frequently important. CLAB explicitly decays auxiliary influence to zero (Rakotoarivony, 28 Aug 2025). DB-LTR uses a time-varying weighting between the imbalanced branch and CoLB (Chen et al., 2023). The liver vessel method controls auxiliary task and contrastive contributions with $512 \rightarrow 256$3 and $512 \rightarrow 256$4 (Sadikine et al., 2024). A plausible implication is that auxiliary contrastive learning tends to be most effective when treated as an early or mid-training shaping force rather than a permanently dominant objective.

7. Significance, limitations, and possible extensions

The significance of CLAB lies in demonstrating that a simple training-time-only contrastive auxiliary branch can improve a mature video detection baseline without adding post-processing or inference overhead (Rakotoarivony, 28 Aug 2025). The empirical result—84.0% mAP with ResNet-101 and 85.2% with ResNeXt-101 on ImageNet VID validation—shows that representational regularization alone can close or exceed gaps previously addressed by more elaborate temporal or memory-based systems (Rakotoarivony, 28 Aug 2025).

At the same time, the available evidence is domain-specific. The named CLAB method is validated on ImageNet VID within a Faster R-CNN / SELSA + TROI pipeline (Rakotoarivony, 28 Aug 2025). It is therefore established as a method for video object detection, not yet as a universal recipe across all detection architectures. Statements about transfer to other detection frameworks would require direct evidence not present in the source data. What can be said is that the paper characterizes the method as straightforward to implement and potentially pluggable into modern backbone-based detectors (Rakotoarivony, 28 Aug 2025).

A common misconception would be to equate CLAB with generic two-view self-supervised contrastive learning. In fact, CLAB is supervised-task-centric: the main detector remains the primary model, the auxiliary contrastive pathway is an added regularizer, and the contrastive pairs are defined by video identity within the training batch rather than by a standalone pretraining protocol (Rakotoarivony, 28 Aug 2025). Another misconception would be to view auxiliary branches as mere deep supervision. In several related papers, including Contrastive Deep Supervision (Zhang et al., 2022) and the scale-specific vessel framework (Sadikine et al., 2024), the point is precisely that contrastive supervision can impose a more appropriate inductive bias than duplicating task loss at intermediate layers.

Future directions are not explicitly enumerated in the source material, but the body of related work suggests several plausible extensions. One is the adaptation of CLAB-style training-time auxiliary contrastive branches to tasks already relying on temporal support, such as tracking-oriented detection or video instance segmentation. Another is the combination of CLAB-style auxiliary regularization with scale-specific or cohort-specific branches, as seen in medical segmentation (Sadikine et al., 2024) and recommendation systems (Cokbas et al., 2 Oct 2025). These possibilities remain interpretive rather than established claims.

In the current literature, CLAB is best situated as a concrete instance of a broader methodological trend: the use of auxiliary branches to impose contrastive structure on shared representations in ways that are aligned with domain-specific nuisances, while preserving the computational characteristics of the deployed model (Rakotoarivony, 28 Aug 2025, Sadikine et al., 2024, Zhang et al., 2022).