Cross-Modal Feature Alignment Module

• Cross-Modal Feature Alignment modules are techniques that use contrastive learning to align feature representations from different modalities and domains.
• They leverage both cross-modal and cross-domain contrastive losses with engineered positive and negative pair sampling, supported by memory banks for stable optimization.
• Empirical evaluations on video action recognition benchmarks show that CFA modules outperform traditional methods by providing robust, transferable representations.

Cross-Modal Feature Alignment (CFA) modules are a class of techniques designed to regularize, align, or reconcile learned feature representations across different data modalities (e.g., RGB and optical flow in video action recognition, or disparate video domains), with the express purpose of enabling robust and transferable representations for downstream tasks such as domain adaptive action recognition. In the context of video-based domain adaptation, CFA modules leverage contrastive learning and carefully crafted sampling strategies to encourage semantically related representations to cluster while maintaining separation between unrelated instances. The following sections elaborate on the methodological foundations, theoretical underpinnings, practical strategies, empirical evaluations, and technical implications of CFA modules as introduced in leading works on video domain adaptation (Kim et al., 2021).

1. Theoretical Foundations: Contrastive Regularization Across Modalities and Domains

At the core of CFA modules is a contrastive learning paradigm that seeks to regularize representation spaces jointly by:

• Cross-modal alignment: Encouraging embeddings originating from different modalities (e.g., RGB and optical flow, denoted as $F^a$ and $F^m$) but the same video instance to be close in feature space.
• Cross-domain alignment: Bridging the domain gap between source (labeled) and target (unlabeled or pseudo-labeled) domains, such that features from these domains representing the same semantic concept (e.g., action class) exhibit strong similarity.

This is formalized by two types of contrastive objectives:

• Cross-modal contrastive loss: For within-domain pairs, "positive" pairs consist of the features from different modalities of the same instance, while "negative" pairs are sampled from different videos.
• Cross-domain contrastive loss: For cross-domain pairs, positive samples are pairs from source and target domains with the same label (including pseudo labels generated with high prediction confidence for target samples), while negatives are disparate videos.

The similarity between two feature vectors from modalities $k$ and $l$ (where $k,l \in \{a, m\}$) is measured after projection, following the SimCLR approach: $$\varphi^s(F_{s,i}^k, F_{s,j}^l) = \exp\left( \frac{h(F_{s,i}^k)^\top h(F_{s,j}^l)}{\tau} \right)$$ where $h(\cdot)$ is a projection head, and $\tau$ is a temperature parameter controlling the sharpness of the alignment.

The contrastive loss for a clip, simplified, is: $$\mathcal{L}_{\text{mo}}^s = -\log \left( \frac{\text{Sum over positives}}{\text{Sum over positives} + \text{Sum over negatives}} \right)$$ Analogous losses are computed for the target domain.

2. Sampling and Memory Strategies for Positive and Negative Pairs

Central to the effectiveness of CFA modules is the design of positive and negative sample selection:

• Positive Pairs: Composed by cross-modality features from the same video, where temporal windows may differ (i.e., random selection from possibly overlapping but not identical frames), enforcing flexibility in temporal alignment.
• Negative Pairs: Features from all other videos—regardless of action label—ensuring that the model does not collapse representations.
• Cross-domain positive pairs: Target videos surpassing a confidence threshold (e.g., $T=0.8$) are pseudo-labeled; positive pairs are then constructed across domains given label agreement.
• Memory Bank: To approximate features over the entire dataset and allow for efficient negative sampling, features are stored in a momentum-updated memory bank: $M^a_s \leftarrow \delta M^a_s + (1-\delta) F^a_s$ for update rate $\delta$ (e.g., $0.5$). This is critical for stable optimization and diversity of negatives.

This sampling protocol supports both intra-modal and cross-domain variability, facilitating the emergence of robust, modality- and domain-invariant representations.

3. Embedding Space Regularization: Joint Modal and Domain Unification

The CFA module’s loss objectives induce:

• Intra-domain, cross-modal regularization: Features corresponding to the same video but different modalities are embedded together, producing representations that inherently encode both appearance and motion.
• Cross-domain, class-conditioned regularization: Features tied to the same semantic class, even when distributed differently due to domain shift, are pulled together, directly combating domain drift.
• Disentanglement across classes/instances: Features from different video instances, irrespective of modality or domain origin, are systematically separated, which empirically enhances class discriminability.

Theoretical and empirical (e.g., t-SNE) analyses in the literature indicate that these losses not only blend modalities within each domain but also foster a compact yet discriminative representation that is less susceptible to domain-induced confounders.

4. Empirical Evaluation and Ablation: Benchmark Results

CFA modules have been evaluated on several action recognition domain adaptation benchmarks:

• UCF ↔ HMDB:
  • Source-only two-stream (RGB + flow): 82.8% (UCF→HMDB), 90.7% (HMDB→UCF)
  • Adding cross-modal contrastive loss: 84.7% (UCF→HMDB), an increment of 1.9 pp.
  • Joint cross-modal + cross-domain contrastive objectives: 84.7% (UCF→HMDB), 92.8% (HMDB→UCF); this outperforms SOTA methods such as TA³N, TCoN, SAVA, MM-SADA.
• EPIC-Kitchens:
  • Source-only baseline: 45.5% mean accuracy.
  • Full CFA (contrastive) framework: 51.0%.
  • Ablations: Both cross-modal and cross-domain modules independently provide improvement; random, independent temporal sampling further enhances alignment.

These results decisively demonstrate that contrastive CFA modules with carefully engineered sampling outperform purely adversarial or non-contrastive alignment schemes.

5. Practical Implementation Considerations and Extensions

For practical deployment:

• Projection Head: A neural network (most often a small MLP) is used for $h(\cdot)$; its depth and dimensionality impact final performance and computational burden.
• Batch Size and Memory Bank: The ability to approximate negative distributions hinges on sufficiently large batch sizes or the inclusion of a continually updated memory bank, especially in GPU-constrained environments.
• Pseudo-Label Reliability: Cross-domain alignment effectiveness relies on the accuracy of pseudo-label predictions; confident threshold selection directly impacts positive pair quality.
• Temporal Windowing: By decoupling positive pair temporal windows, the approach is robust to intra-clip asynchrony—a common issue in optical flow and RGB stream extraction.

These implementation details are critical for faithfully replicating the CFA pipeline and optimizing for real-world video domain adaptation scenarios.

6. Comparative Positioning and Architectural Implications

The CFA paradigm, as introduced for video action recognition, moves away from conventional adversarial or purely marginal alignment strategies. By leveraging instance- and class-centric semantic structure via contrastive learning, it offers robust, modality-generalizable, and domain-invariant feature spaces, critical for high-performance transfer learning. Notably, the empirical superiority of this approach is particularly pronounced under class imbalance and fine-grained setting, as seen in challenging datasets like EPIC-Kitchens.

Architecturally, the joint contrastive framework can be readily composed atop modern two-stream networks and is compatible with memory and efficiency constraints through the explicit design of projection heads and memory banks.

---

In summary, Cross-Modal Feature Alignment modules, implemented via contrastive loss with robust and flexible sampling strategies, represent an effective and scalable method for regularizing multi-modal and cross-domain feature spaces in video-based action recognition and related tasks. Their design directly addresses modality and domain gaps, providing strong empirical and theoretical advantages over earlier adversarial or single-modality alignment methods (Kim et al., 2021).