Cross-Modal Manifold CutMix for Video SSL

• The technique fuses feature maps from different video modalities using CutMix-style splicing, preserving temporal integrity and leveraging complementary cues.
• It integrates into the STC-mix pipeline with alternating encoder training, addressing the shortcomings of traditional MixUp and CutMix in video contexts.
• Empirical evaluations demonstrate enhanced action recognition and video retrieval performance in low-data regimes, validating its robust self-supervised learning capacity.

Cross-Modal Manifold CutMix (CMMC) is a data augmentation technique designed for self-supervised video representation learning, particularly effective in domains with limited labeled or unlabeled data. CMMC operates by performing CutMix-style feature map splicing across distinct video modalities (such as RGB and optical flow), preserving temporal structure while leveraging complementary modality-specific inductive biases. It constitutes the central mechanism in the STC-mix (“Space–Time–Channel Mix”) training pipeline, enhancing regularization and representation quality for downstream tasks. CMMC addresses the limitations of direct extensions of image-domain MixUp and CutMix to video, which often disrupt spatio-temporal coherence or yield sub-optimal regularization effects (Das et al., 2021).

1. Motivation and Context

Self-supervised contrastive learning frameworks, such as MoCo and SimCLR-3D, excel in video representation learning when massive volumes of unlabeled sequences are available. In low-data regimes typical of applications like patient monitoring or smart home surveillance, common augmentations (cropping, color-jitter) fail to sufficiently regularize, leading to overfitting. Image-SSL augmentations (MixUp, CutMix, Manifold-MixUp) do not transfer effectively: input-space MixUp on video preserves little temporal structure, while video CutMix destroys motion continuity. CMMC is motivated by the need for augmentations that (1) exploit multi-modal data and (2) preserve critical spatio-temporal information (Das et al., 2021).

2. Mathematical Formulation

Given two modalities $a$ and $b$ with hidden representations $$F^a \in \mathbb{R}^{c_a\times t_a\times h_a\times w_a}$$ and $$F^b \in \mathbb{R}^{c_b\times t_b\times h_b\times w_b}$$, CMMC produces a mixed feature by:

$$F^{\text{mix}} = M \odot F^{a} + (1 - M)\odot F^{b},$$

where $$M \in \{0,1\}^{c_{\min} \times t_{\min} \times h_{\min} \times w_{\min}}$$ is a binary mask sampled by selecting bounding box coordinates. Typically, the temporal and channel axes are preserved ($b_{t1},b_{t2}=(0,t_\min)$; $b_{c1},b_{c2}=(0,c_\min)$), with spatial sub-blocks randomly cut. The effective mixing coefficient is

$$\lambda = \frac{\sum_{i,j,k,\ell}M_{i,j,k,\ell}}{\text{Vol}(M)} = 1 - \frac{\text{Vol of cut region}}{\text{Total vol}},$$

and the soft labels are defined as

$$y^{\mathrm{mix}} = \lambda y^{a} + (1-\lambda)y^{b}.$$

This strategy ensures that mixed representations simultaneously reflect features from different modalities while maintaining semantic continuity.

3. STC-mix Pipeline and Training Procedure

The STC-mix pipeline integrates two principal mixing stages:

• Stage 0 (Input MixUp): For each modality, standard MixUp is performed in input space, with mixing coefficient $b$0. Encoders are pre-trained for 300 epochs using InfoNCE and MixUp.
• Stages 1–4 (CMMC Alternating Training): Four alternating stages are executed, each fixing one modality’s encoder and training the other via CMMC:
  • Randomly select two layer indices $b$1.
  • For a batch, compute feature maps at layer $b$2 for modality $b$3 and layer $b$4 for modality $b$5.
  • Apply CMMC between these layers to generate mixed features and virtual soft labels.
  • Continue feedforward through unfrozen modality's encoder.
  • Apply cross-modal contrastive loss with a momentum encoder on the fixed stream.
  • Alternate roles of $b$6 and $b$7 to complete two full cycles.

Pseudo-code for a CMMC stage is provided as Algorithm 1 in the reference (Das et al., 2021).

4. Contrastive Learning Objective

For mixed queries $b$8 and corresponding positives $b$9 from a momentum encoder, the $$F^a \in \mathbb{R}^{c_a\times t_a\times h_a\times w_a}$$0-way InfoNCE loss is used:

$$F^a \in \mathbb{R}^{c_a\times t_a\times h_a\times w_a}$$1

where $$F^a \in \mathbb{R}^{c_a\times t_a\times h_a\times w_a}$$2 is the temperature, and embeddings are L2-normalized. Negatives are sampled from a MoCo-style queue updated by momentum ($$F^a \in \mathbb{R}^{c_a\times t_a\times h_a\times w_a}$$3). Queue size is 2048 for UCF and 16,384 for Kinetics-400.

5. Architectural and Modal Setup

CMMC is instantiated with independent encoders for each modality. For RGB and flow streams, S3D is used; for skeleton sequences, ST-GCN (with channels scaled by $$F^a \in \mathbb{R}^{c_a\times t_a\times h_a\times w_a}$$4). In two-stream settings, $$F^a \in \mathbb{R}^{c_a\times t_a\times h_a\times w_a}$$5 and $$F^a \in \mathbb{R}^{c_a\times t_a\times h_a\times w_a}$$6 correspond (for instance) to RGB and flow. For skeletons, the modalities can be Joints and Motion cues. CMMC is applied to pairs of intermediate feature blocks, with the precise block indices selected randomly per batch. CMMC interleaves cut regions along space, time, and channel axes within the feature tesseracts.

Key architectural and data flow details are summarized as follows:

Stream Type	Backbone	Modalities
RGB + Flow	S3D	RGB, TV-L1 flow
Skeleton	ST-GCN	Joint, Motion

Hyperparameters include MixUp $$F^a \in \mathbb{R}^{c_a\times t_a\times h_a\times w_a}$$7, CutMix $$F^a \in \mathbb{R}^{c_a\times t_a\times h_a\times w_a}$$8, Adam optimizer for S3D (lr=$$F^a \in \mathbb{R}^{c_a\times t_a\times h_a\times w_a}$$9, wd=$$F^b \in \mathbb{R}^{c_b\times t_b\times h_b\times w_b}$$0, batch=32/GPU), four CMMC stages of 100 epochs (700 epochs total if counting all), and SGD for ST-GCN (lr=0.1 $$F^b \in \mathbb{R}^{c_b\times t_b\times h_b\times w_b}$$1 0.01). For skeletons, pre-training is run for 150 epochs per channel, followed by two 150-epoch CMMC stages.

6. Empirical Results and Ablation Findings

CMMC demonstrates strong performance in small-data regimes across action recognition and video retrieval on UCF101, HMDB51, and NTU60:

• Video-only mixing: Input-space MixUp increases UCF linear-probe top-1 from 38.2% to 49.6%; direct spatio-temporal CutMix (VideoMix) degrades accuracy.
• Cross-modal ablations: Progressive enhancements—spatial, then spatio-temporal, then full (with channels)—yield UCF linear-probe rises from 52.8% (spatial) to 55.8% (full) for RGB, and from 68.6% to 72.4% for Flow.
• Action Recognition: STC-mix (two-stream) achieves 74.0% linear-probe on UCF101 (vs. 72.1% CoCLR) and 38.1% on HMDB (vs. 36.3%); fine-tuning yields 87.5% (UCF)/59.1% (HMDB).
• Video Retrieval: R@1 improves from 55.9% to 58.1% (UCF) and from 26.1% to 27.1% (HMDB).
• Skeleton SSL: CMMC on Joints/Motion lifts NTU-60 CS accuracy from 70.1% to 72.5%, CV from 77.2% to 79.1%. Two-stream setups approach fully supervised benchmarks (75.2%/83.1%).
• Kinetics-400 Transfer: Integrated with BYOL, STC-mix† on S3D achieves 96.9%/75.4% (UCF/HMDB), surpassing prior results.

These results underline the superior regularization and manifold shaping provided by cross-modal mixing, particularly when randomizing mix layer and cut region.

7. Practical Considerations and Limitations

CMMC necessitates the simultaneous availability of at least two modalities per sample—potentially a constraint for domains lacking multi-modal data. Alignment and reshaping of feature maps are required when modality outputs differ in dimensionality (e.g., RGB vs. sparse skeleton, resolved by reshaping joints to 2D patches as described in the Appendix). Effective stochasticity—randomizing both cut regions and layers—is critical; fixed cuts reduce gain. In severe low-data settings, preliminary input-space MixUp is essential for stability. CMMC introduces modest computational overhead (single batch-wise splice), as no additional forward passes over frozen encoders are needed.

8. Significance and Impact

Cross-Modal Manifold CutMix, as deployed in the STC-mix pipeline, constitutes a robust augmentation paradigm for self-supervised video representation learning in small-data or multi-modal domains. By enabling feature-space mixing across modalities, CMMC (1) maintains spatio-temporal integrity, (2) aligns and exploits complementary signals, and (3) empirically yields improved or state-of-the-art linear probe and fine-tuned performance across standard video understanding benchmarks, especially when data is too sparse for conventional SSL to be effective (Das et al., 2021).