Unsupervised Multi-Modal Pseudo-Labeling

• Unsupervised multi-modal pseudo-labeling is a method that leverages complementary data from modalities like vision, audio, and LiDAR to create robust labels.
• The approach employs ensemble-based aggregation, cross-modal consistency, and hybrid label mixing to reduce noise and enhance semantic fidelity.
• Experimental benchmarks reveal that these techniques outperform single-modality methods, providing improved accuracy and stable representations.

Unsupervised multi-modal pseudo-labeling refers to a set of techniques that assign data-driven labels to unlabeled samples by leveraging information from multiple complementary modalities, without recourse to human annotation. These methods integrate signals from heterogeneous sources—such as vision, audio, text, motion cues, or event streams—to construct more robust, semantically meaningful pseudo-labels that guide downstream self-training, clustering, or representation learning regimes. The principal motivation is to unlock high-performance learning in domains where exhaustive manual annotation is prohibitive or infeasible, while systematically addressing the pitfalls of single-modality confirmation bias and label noise.

1. Key Principles of Multi-Modal Pseudo-Labeling

The central design pattern in multi-modal pseudo-labeling is the cross-modal exploitation of independent or weakly correlated cues to enhance label reliability and granularity. These methods typically instantiate one or more of the following principles:

• Complementarity: Each modality—e.g., appearance (RGB), motion (optical flow), audio—offers distinct, partial information. Aggregating predictions across modalities provides error correction and sharper semantic delineation (“complementary views help obtain more reliable pseudo-labels on unlabeled video” (Xiong et al., 2021)).
• Cross-modal consistency: For a given sample, pseudo-labels are constructed such that predictions from different modalities either serve as targets for each other (cross-supervision) or are aggregated (ensemble) for increased confidence (Xiong et al., 2021, Asano et al., 2020, Jing et al., 2024).
• Noise mitigation: Hybridization across modalities breaks the feedback loops of single-modality self-training, reducing confirmation bias and label collapse stemming from modal artifacts or spurious correlations (Jing et al., 2024).
• Shared representation or fusion space: Many methods train one “shared” backbone across modalities—achieving unified embeddings and, frequently, zero or minimal extra inference cost (Xiong et al., 2021, Ghilotti et al., 8 Jan 2026).

2. Algorithmic Frameworks

A wide variety of unsupervised and semi-supervised tasks have adopted multi-modal pseudo-labeling. Key frameworks include:

2.1 Ensemble-Based Cross-View Pseudo-Labeling

• Multiview Pseudo-Labeling (MvPL): Each unlabeled sample is encoded in M distinct views—e.g., RGB (appearance), optical flow (motion), and temporal gradients. A single 3D CNN backbone (e.g., ResNet-50 Slow) processes all modalities, creating per-view predictions $q^m_i$. Pseudo-labels are then aggregated via weighted averaging:

$$s_i = \frac{1}{\sum_{m=1}^M w_m} \sum_{m=1}^M w_m q^m_i,\quad \hat{s}_i = \arg\max s_i$$

A confidence threshold τ filters out low-confidence predictions, and models are trained to enforce consistency between weakly/strongly augmented views using cross-entropy on these pseudo-labels (Xiong et al., 2021).

2.2 Multiple Pseudo-Label Sources and Noisy Label Mixing

• Hybrid Pseudo-Labeling for Event Segmentation (HPL-ESS): Two independent pseudo-label sources are constructed: (i) direct self-training via a mean-teacher student/teacher network on native event data; (ii) pseudo-labels inferred from segmentation on reconstructed images obtained from event-to-image translation. These labels are mixed:

$$\hat V^l = (1-\alpha)V^l_{\mathrm{img}} + \alpha V^l_{\mathrm{evt}}$$

The mixing mitigates reconstruction noise and confirmation bias. A Soft Prototypical Alignment module aligns feature spaces across modalities to further regularize and synchronize representations (Jing et al., 2024).

2.3 Cross-Modal Clustering and Alignment

• SeLaVi Framework: Joint clustering is performed on audio–visual data using modality-specific deep encoders, with assignments enforced to be mutually predictive via symmetric cross-entropy losses. A permutation alignment step matches clusters between modalities, generating a unified set of pseudo-labels that are well-aligned with human semantic categories (Asano et al., 2020).
• UniLiPs for LiDAR: Temporal geometric consistency across LiDAR sweeps (SLAM) provides strong priors for separating static/dynamic points. Semantic cues lifted from vision foundation models (e.g., Segment Anything, OneFormer, BLIP/CLIP) are fused via an iterative, geometry-grounded update rule that propagates labels and segments moving objects, producing dense, coherent 3D pseudo-labels and bounding boxes (Ghilotti et al., 8 Jan 2026).

3. Mathematical Formalism

Pseudo-label generation generally proceeds by defining cross-modal objectives, regularizations, and update rules. Representative forms include:

• Weighted cross-view ensembling (MvPL): $s_i$ as above; labels kept only if $\max s_i \geq \tau$.
• Clustering with Sinkhorn-Knopp: For each modality, calculate soft assignments $Q^A, Q^V$, enforce cross-modal self-prediction, and align clusters via permutation minimization:

$$\mathcal{L}_{\text{cluster}} = -\frac{1}{HB} \sum_{h,i,k} \left[ Q^A_{ih,k}\log P^V_{ih,k} + Q^V_{ih,k}\log P^A_{ih,k} \right]$$

$$\min_{\pi \in S_K} \sum_{k=1}^{K} \left[1 - \cos(C^A_{h,k},\,C^V_{h,\pi(k)})\right]$$

(Asano et al., 2020)

• Hybrid noisy label mixing (HPL-ESS): Linear blending of two sources under student–teacher with iterative feature/prototype alignment.
• Geometric label propagation (UniLiPs): Static probability for each map point is updated by combining range credibility and semantic consistency in fixed-point iterations.

4. Experimental Benchmarks and Comparative Performance

Extensive experiments across modalities and domains demonstrate that multi-modal pseudo-labeling consistently outperforms single-modality and naive baselines:

Method	Domain	Unsupervised Input	Pseudo-label Mechanism	Benchmark Results
MvPL (Xiong et al., 2021)	Video	Appearance, Motion, Gradients	Ensemble cross-view aggregation	UCF101: 80.5% (vs 48.5% supervised)
HPL-ESS (Jing et al., 2024)	Event Seg.	Events, Reconstructions	Noisy label mixing + SPA	DSEC-Semantic: 89.92% acc, 55.19% mIoU (outperforming SOTA)
UniLiPs (Ghilotti et al., 8 Jan 2026)	3D LiDAR	LiDAR, RGB, Language	SLAM+vision fusion, geometry-consistent	KITTI LiDAR: 64.9% mIoU (point), 31.0% mAP (3D det.)
SeLaVi (Asano et al., 2020)	Video	Audio, Video	Cross-modal clustering/alignment	VGG-Sound: NMI=56.7, UCF101: 87.7% downstream

Ablations repeatedly indicate that hybrid/multimodal supervision reduces error propagation (confirmation bias), increases stability of clusters, and yields pseudo-labels with greater semantic purity and downstream utility. Multi-view and cross-modal methods outperform strong single-modality FixMatch, mean-teacher, and clustering baselines.

5. Practical Considerations and Limitations

Significant strengths of multi-modal pseudo-labeling include:

• Zero or minimal inference-time cost increase if a shared backbone and view-invariant model is used (Xiong et al., 2021).
• Mitigation of noise: hybridization and cross-supervision unbinds systematic errors present in mono-modal pipelines (Jing et al., 2024).
• Flexibility: new modalities (audio, text, radar) can be inserted into established frameworks with straightforward extension (Xiong et al., 2021, Ghilotti et al., 8 Jan 2026).

However, these approaches typically impose substantial computational requirements during training (e.g., 64 GPU jobs for large-scale clustering (Asano et al., 2020)), require pre-processing for some modalities (pre-computing optical flow, event-to-image reconstructions), and are sensitive to calibration or synchronization errors between modalities (as observed in LiDAR-vision fusion (Ghilotti et al., 8 Jan 2026) and AV sync in SeLaVi (Asano et al., 2020)). The assumption of strong cross-modal semantic correlation is necessary for cluster alignment or ensembling, and its violation (e.g., in silent videos, occluded scenes) degrades label quality.

6. Open Directions and Future Extensions

Contemporary research identifies multiple directions for advancing unsupervised multi-modal pseudo-labeling:

• Fully unsupervised bootstrapping: While many frameworks rely on a small labeled seed set (for initialization or prototype computation), iterative clustering or cross-view self-training can be extended to operate entirely on unlabeled collections by leveraging cluster refinement and cross-modal agreement.
• Incorporation of further modalities: Text, language, and sensor data (event cameras, radar) can be unified under current frameworks via shared embedding/fusion architectures (Xiong et al., 2021, Ghilotti et al., 8 Jan 2026).
• Dynamic and hierarchical assignment: Learning the number of clusters or pseudo-labels adaptively, and hierarchical clustering, could accommodate data distributions not well-modeled by balanced partitions (Asano et al., 2020).
• Learned fusion and graph update rules: Replacing fixed iterative rules with learned message-passing or GNNs holds promise for more robust multimodal fusion in geometry-based domains (Ghilotti et al., 8 Jan 2026).
• Online or real-time pseudo-labeling: Real-time variants would enable deployment in dynamic or time-sensitive environments, a plausible implication for extension even where current methods remain offline or operate on precomputed representations.

The breadth and empirical success of current approaches demonstrate that unsupervised multi-modal pseudo-labeling is a powerful paradigm for scalable annotation and robust representation learning across video, event streams, 3D perception, and beyond (Xiong et al., 2021, Jing et al., 2024, Ghilotti et al., 8 Jan 2026, Asano et al., 2020).