Multimodal Pseudo-Labeling Methods

• Multimodal pseudo-labeling is a method that leverages complementary information across modalities to create supervisory signals when human annotation is scarce.
• It employs techniques such as cross-modal alignment, modality translation, and view-based arbitration to refine pseudo-labels and enhance model training.
• Applications span instance segmentation, histology image analysis, video-language pre-training, and more, demonstrating significant performance gains over traditional methods.

Searching arXiv for the cited papers and related multimodal pseudo-labeling work. Multimodal pseudo-labeling denotes a family of weakly supervised, semi-supervised, and self-training methods that construct supervisory targets by exploiting complementary information across modalities, views, or cross-modal embeddings rather than relying exclusively on human annotation. In the cited literature, the method appears in open-vocabulary instance segmentation, histology image segmentation, multimodal classification, video-language pre-training, semi-supervised video recognition, audio-visual source localization, single-positive multi-label learning, dataset construction for document-to-image generation, and hidden emotion understanding from video (Huynh et al., 2021, Boschet et al., 2024, Zyblewski et al., 2024, Lialin et al., 2023, Xiong et al., 2021, Guo et al., 2024, Xing et al., 2023, Kim et al., 24 Aug 2025, Wang et al., 8 Feb 2026). Across these settings, pseudo-labels are not merely guessed labels; they are inferred from alignment, translation, clustering, agreement, ranking, or vision-language similarity, and are then used to train a student model, a classifier, or a joint multimodal system.

1. Definition and problem scope

The central problem addressed by multimodal pseudo-labeling is supervision scarcity under cross-modal or multi-view structure. In open-vocabulary instance segmentation, the objective is to segment novel classes without mask annotations; in histology, the objective is to generate pseudo-labels for an unlabeled modality $U$ using annotations from a labeled modality $L$; in multimodal classification, the objective is to propagate labels more reliably than unimodal or fusion-only baselines; and in video-language learning, the objective is to pre-train on abundant unlabeled videos without any parallel video-text data (Huynh et al., 2021, Boschet et al., 2024, Zyblewski et al., 2024, Lialin et al., 2023).

The same principle extends to view-defined multimodality. In multiview pseudo-labeling for video, appearance and motion information are treated as complementary views, specifically RGB frames, optical flow, and temporal gradients. In semi-supervised audio-visual source localization, audio and visual streams are jointly exploited, and two models learn from each other through cross pseudo-labels. In single-positive multi-label learning, frozen vision-LLMs provide pseudo-labels for labels that are unannotated rather than confirmed negative. In document-image generation, image-caption-summary alignment is used to select a pseudo-labeled image from multiple candidate images, and in hidden emotion understanding, a large vision-LLM produces pseudo-labels and reasoning text that become weak supervision for downstream multimodal models (Xiong et al., 2021, Guo et al., 2024, Xing et al., 2023, Kim et al., 24 Aug 2025, Wang et al., 8 Feb 2026).

Setting	Pseudo-label source	Representative paper
Open-vocabulary segmentation	Caption-word to region-mask alignment	(Huynh et al., 2021)
Histology segmentation	Unpaired modality translation $L \leftrightarrow U$	(Boschet et al., 2024)
Multimodal classification	Per-modality clustering and cross-propagation	(Zyblewski et al., 2024)
Video-language pre-training	Image-captioner pseudo-captions for video clips	(Lialin et al., 2023)
Semi-supervised video learning	Aggregated predictions from multiple views	(Xiong et al., 2021)
Audio-visual localization	Cross-refined soft pseudo-labels	(Guo et al., 2024)
SPML	Vision-language similarity-based pseudo-labels	(Xing et al., 2023)

2. Core mechanisms for generating pseudo-labels

A first major mechanism is cross-modal alignment. The robust cross-modal pseudo-labeling framework for open-vocabulary instance segmentation aligns word semantics in captions with visual features of object masks in images. It extracts a set of object nouns $\mathcal{O}_c$ from the caption, relies on WordNet to select “object”-type words, embeds regions into a joint semantic space with an embedding head $h_{\text{Emb}}$, and uses a class-agnostic mask head $h_{\text{Mask}}$ to generate pseudo masks for aligned regions. The resulting pseudo masks allow novel classes in captions to self-train a student model, even though no human mask annotations are available for those classes (Huynh et al., 2021).

A second mechanism is modality translation. In histology image segmentation, an unpaired image translation model, SynDiff, learns $L \to U$ and $U \to L$. This enables two pseudo-labeling strategies. In Tutorship (L → U), annotations $Y_L$ are paired with style-transferred images $X_{L \to U}$, a segmentation model is trained on $L$0, and then applied to real $L$1 to obtain $L$2. In Adaptation (U → L), $L$3 is translated to $L$4, a pre-trained $L$5 segmentation model predicts labels there, and the resulting masks become $L$6 for $L$7. The method does not require paired data and does not require annotation in the target domain (Boschet et al., 2024).

A third mechanism is separate per-modality inference followed by cross-modal arbitration. CMCSL extracts deep features for each modality, clusters each modality independently using pre-labeled seeds as fixed centroids, propagates labels within each cluster, and then exchanges class-membership information between modalities using Euclidean distance to resolve disagreements. The method explicitly avoids operating only in a unified or fused feature space; instead, it uses each modality’s own geometry and then performs cross-propagation between them (Zyblewski et al., 2024).

A fourth mechanism is multimodal caption or ranking transfer. In video-language pre-training without aligned data, unlabeled videos are split into 8-second clips, the center frame of each clip is passed through BLIP, and the generated caption becomes a pseudo-label for that video segment. In MMCIG dataset construction, the gold summary independently ranks images and captions, and a pseudo-label is assigned to an image only when both the image and its corresponding caption are ranked first in their respective rankings. This ranking-based construction is followed by removal of documents that contain direct image references within texts (Lialin et al., 2023, Kim et al., 24 Aug 2025).

A fifth mechanism is agreement across views or models. MvPL ensembles predictions from appearance and motion views to obtain a more reliable pseudo-label for unlabeled video, while XPL trains two models in parallel and each model learns from the pseudo-labels generated by the other. In SPML, VLPL uses a frozen vision-LLM, such as CLIP, to suggest strong positive and optionally negative pseudo-labels, though the reported best performance is obtained with only strong positive pseudo-labels (Xiong et al., 2021, Guo et al., 2024, Xing et al., 2023).

3. Representative mathematical formulations

In cross-modal pseudo-mask generation, each caption object $L$8 is aligned to the candidate bounding box

$L$9

where $L \leftrightarrow U$0 is the BERT embedding for $L \leftrightarrow U$1, $L \leftrightarrow U$2 is the region feature, and $L \leftrightarrow U$3 is the proposal set. The corresponding pseudo mask is

$L \leftrightarrow U$4

This formulation makes the pseudo-label a region-level consequence of visual-semantic compatibility rather than a direct caption token assignment (Huynh et al., 2021).

In CMCSL, the labeled seeds serve as fixed centroids for a single iteration of k-means. For modality $L \leftrightarrow U$5, each sample is assigned by

$L \leftrightarrow U$6

and each unlabeled sample in $L \leftrightarrow U$7 receives the centroid label $L \leftrightarrow U$8. When visual and textual pseudo-labels disagree, the label from the modality with smaller Euclidean distance to its centroid is selected. This turns pseudo-labeling into a two-stage procedure: within-modality propagation followed by cross-modality conflict resolution (Zyblewski et al., 2024).

In MvPL, the preferred aggregated-supervision strategy averages predictions from all views,

$L \leftrightarrow U$9

with equal weights $\mathcal{O}_c$0 in practice. The unsupervised loss then applies only when the confidence threshold is met: $\mathcal{O}_c$1 This structure makes pseudo-labeling a function of cross-view agreement before strong augmentation is enforced (Xiong et al., 2021).

In XPL, the unlabeled objective is a cross pseudo-labeling loss,

$\mathcal{O}_c$2

and the pseudo-labels are stabilized by exponential moving average: $\mathcal{O}_c$3 The framework thereby replaces static hard pseudo-labels with temporally smoothed soft targets (Guo et al., 2024).

In VLPL, a frozen vision-LLM computes similarity between image embedding $\mathcal{O}_c$4 and label embeddings $\mathcal{O}_c$5: $\mathcal{O}_c$6 Any label with $\mathcal{O}_c$7 is treated as a positive pseudo-label. The downstream model is trained with an Entropy-Maximization loss that uses confirmed positives, pseudo positives, and entropy on unknown labels (Xing et al., 2023).

4. Reliability estimation, denoising, and bias control

A recurring theme is that multimodal pseudo-labels are informative but noisy. XPM explicitly models this by estimating a per-pixel noise level through a neural module $\mathcal{O}_c$8, assuming Gaussian noise,

$\mathcal{O}_c$9

and deriving a reliability score

$h_{\text{Emb}}$0

This reliability score downweights unreliable masks in the cross-modal loss, and the paper states that not all pseudo masks are reliable due to errors in detection, ambiguous captions, or weak object-caption grounding (Huynh et al., 2021).

XPL addresses three specific weaknesses of vanilla hard pseudo-labels—bias accumulation, noise sensitivity, and instability. It introduces soft pseudo-labels with sharpening, pseudo-label exponential moving average, and a curriculum data selection module. Reliability is estimated through inter-model agreement measured by the Pearson correlation coefficient

$h_{\text{Emb}}$1

and only highly reliable samples are used first, with less reliable samples introduced gradually. The paper reports that this curriculum data selection adaptively selects pseudo-labels with high quality during training to mitigate potential bias (Guo et al., 2024).

In SPML, noise control takes a different form. VLPL reports that best performance is achieved using only strong positive pseudo-labels, while leaving the rest as unknown, and label smoothing is used for robustness. A later continuation, AEVLP, introduces the Generalized Pseudo-Label Robust Loss and Dynamic Augmented Multi-focus Pseudo-labeling. GPR Loss is explicitly described as a general loss that can adapt to multiple pseudo-labeling schemes and mitigate both label noise and pseudo-label uncertainty, while DAMP regenerates pseudo-labels at each epoch through global and local crops, class-correlation modeling with GCN, and threshold-based positive and negative mining (Xing et al., 2023, Tran et al., 28 Aug 2025).

In translation-based pseudo-labeling, the principal failure mode is translation quality. The histology study states that if the image translation model fails to realistically map features between modalities, pseudo-labels may have lower quality, and the optimal strategy is not fixed: tutorship is stronger for TEM $h_{\text{Emb}}$2 SEM, whereas adaptation marginally outperforms tutorship for TEM $h_{\text{Emb}}$3 BF. This directly counters the assumption that a single pseudo-labeling path is uniformly best across domain shifts (Boschet et al., 2024).

5. Architectures, optimization patterns, and modality roles

The architectural organization of multimodal pseudo-labeling typically separates pseudo-label generation from downstream learning, even when both are trained jointly. In XPM, the teacher model is based on Mask R-CNN with an embedding head $h_{\text{Emb}}$4 and a class-agnostic mask head $h_{\text{Emb}}$5, while the student jointly learns embedding, mask, and noise modules. In histology segmentation, SynDiff performs unpaired image translation and nnU-Net carries out segmentation with 5-fold cross-validation. In CMCSL, feature extraction is modality-specific, using ResNet-18 for visuals and MiniLM-L6-v2 for text, followed by fixed-centroid clustering and downstream classifiers such as Gaussian Naïve Bayes, Logistic Regression, and CART (Huynh et al., 2021, Boschet et al., 2024, Zyblewski et al., 2024).

In video-language settings, pseudo-labeling is often combined with frozen backbones and lightweight trainable interfaces. The video captioning method uses a TimeSformer visual encoder and an OPT LLM, with adapters inserted into select OPT layers and a separable cross-attention mechanism. Only adapters and fusion components are trained, while the core OPT and TimeSformer are frozen. In MvPL, a single 3D CNN is shared across appearance and motion inputs, so the model capitalizes on multiple views during training but incurs no additional computation overhead at inference time, where only RGB is used (Lialin et al., 2023, Xiong et al., 2021).

Vision-language pseudo-labeling further emphasizes asymmetry between a strong frozen teacher and a trainable task model. VLPL uses frozen CLIP vision and text encoders only at training time to generate pseudo-labels, while the task backbone can be a CNN or transformer such as ResNet, ConvNeXt, or ViT. In the hidden emotion setting, Gemini 2.5 Pro generates pseudo-labels and reasoning text through Chain-of-Thought and Reflection prompting; YOLO 11x detects and crops portraits, DINOv2-Base extracts visual features, OpenPose yields 137-dimensional key-point sequences augmented with inter-frame offsets, an MLP models key-point spatial structure, Transformers encode the image and key-point sequences, and BERT encodes the reasoning text. Each modality is first pre-trained in isolation, then fine-tuned jointly, with pseudo-labeled samples merged into the training set (Xing et al., 2023, Wang et al., 8 Feb 2026).

Dataset construction can itself be the end product of multimodal pseudo-labeling rather than only an intermediate step. In MMCIG, factual consistency filtering uses BERTScore Art, AlignScore, and SummaCscore; image relevance is ranked by CLIP similarity, caption relevance by BERTScore, and the multimodal pseudo-label is kept only when both rankings identify the same image. This formulation treats pseudo-labeling as precision-oriented corpus curation for downstream multimodal generation (Kim et al., 24 Aug 2025).

6. Empirical behavior, limitations, and common misconceptions

The empirical record reported in the cited works is consistently favorable, but it is highly setting-dependent. In open-vocabulary instance segmentation, XPM improves target mAP in the generalized setting from 17.1% to 21.6% on MS-COCO and from 16.9% to 22.7% on Open Images & Conceptual Captions, corresponding to gains of 4.5 points and 5.1 points. The paper further states that both cross-modal loss and mask loss are vital, and that noise estimation for pseudo masks outperforms class-score or pixel-score thresholding for reliability (Huynh et al., 2021).

In histology segmentation, the tutoring path reaches a mean Dice score of $h_{\text{Emb}}$6 on the SEM dataset. When the source and target are similar, such as TEM $h_{\text{Emb}}$7 TEM-MACAQUE, direct transfer is nearly as effective as pseudo-labeling strategies; when the domain shift is moderate or large, such as TEM $h_{\text{Emb}}$8 SEM or TEM $h_{\text{Emb}}$9 BF, direct transfer fails and pseudo-labeling significantly outperforms it. Both approaches are reported as suitable as a starting point for manual refinement and are expected to speed up annotation effort by 25–50% (Boschet et al., 2024).

In multimodal classification, CMCSL was evaluated on 20 datasets derived from MM-IMDb. For binary datasets in the visual modality, average balanced accuracy is reported as 0.636 for PRE, 0.680 for UNI, and 0.732 for CMCSL. The gains are especially pronounced with 1–5 labeled samples per class, and the weaker modality often benefits most from cross-propagation. The paper explicitly contrasts this with early and late fusion, arguing that local confidence-based cross-propagation avoids degradation when modalities are unequally informative (Zyblewski et al., 2024).

In video and audio-visual learning, the contrast with naive pseudo-labeling is similarly strong. The video captioning work reports that BLIP pseudo-captions are better than HowTo100M ASR captions in 65% of manually inspected cases, that ASR captions refer to video content in only 45% of cases while BLIP captions do so in 88% of cases, and that pre-training on both images and videos yields 54.0 CIDEr-D on MSR-VTT versus 49.7 or 49.6 for single-modality variants. MvPL reports that on UCF101 split-1 with 10% labels, RGB only yields 48.5%, RGB + Flow 76.5%, RGB + temporal gradients 74.0%, and all three 79.1%; on Kinetics-400 with 10% labeled data, supervised training yields 39.2% while MvPL reaches 58.2%. XPL raises CIoU from 82.36 to 89.28 on Flickr-144k and from 40.68 to 48.40 on Vggss-144k, outperforming both vanilla hard pseudo-labeling and Attention10k-SSL (Lialin et al., 2023, Xiong et al., 2021, Guo et al., 2024).

A common misconception is that multimodal pseudo-labeling is a single algorithm or that it always requires paired data. The cited literature shows otherwise: some methods rely on cross-modal alignment, some on unpaired translation, some on clustering and arbitration, some on ranking intersection, and some on agreement across models or views. Another misconception is that harder pseudo-labels are necessarily better. XPL identifies hard pseudo-labels as a source of bias accumulation, noise sensitivity, and instability, while VLPL finds that strong positive pseudo-labels without pseudo-negative labels give the best results in SPML. A further misconception is that multimodal pseudo-labeling is only a training trick for classifiers; the MMCIG study shows that it can also be used to construct a higher-precision dataset, and the hidden emotion study shows that pseudo-labels and reasoning texts can serve as weak supervision that lifts accuracy from under 0.6 in prior work to 69.23% on iMiGUE (Xing et al., 2023, Kim et al., 24 Aug 2025, Wang et al., 8 Feb 2026).

Taken together, the literature presents multimodal pseudo-labeling as a general supervision-transfer strategy: modalities are not simply fused, but used to expose latent supervision that is absent from the labeled set. This suggests that the central design question is not whether to pseudo-label, but which cross-modal relation—alignment, translation, ranking, clustering, or agreement—best matches the structure and failure modes of the target problem.