Semi-Supervised Multimodal Domain Generalization

• The paper introduces SSMDG, a novel framework that simultaneously tackles multimodality, domain shift, and limited annotations to learn a unified predictor.
• It employs a multi-encoder architecture with consensus-driven consistency and disagreement-aware regularization to refine pseudo-labels and ensure robustness in missing-modality scenarios.
• Empirical evaluations on HAC and EPIC-Kitchens show significant accuracy improvements over baselines, demonstrating the method’s effectiveness under standard and missing-modality conditions.

Semi-Supervised Multimodal Domain Generalization (SSMDG) is a problem setting introduced to study how to learn robust multimodal models from multi-source data with few labeled samples while generalizing out-of-the-box to an unseen target domain. In the formulation given in "Towards Multimodal Domain Generalization with Few Labels," SSMDG combines three constraints that are usually treated separately: multimodality, domain shift, and limited annotation. The setting assumes multiple source domains with small labeled sets and large unlabeled sets, and seeks a single predictor that exploits both while remaining robust under standard evaluation and missing-modality scenarios (Li et al., 26 Feb 2026).

1. Formal problem formulation

Let $M$ be the number of modalities, with examples such as video $v$ and audio $a$, so that $M=2$. The input space is $$\mathcal{X}=\mathcal{X}_1\times \dots \times \mathcal{X}_M$$ and the label set is $\mathcal{Y}$. A multimodal sample is $X=(x_1,\dots,x_M)$ with $x_m\in\mathcal{X}_m$.

There are $K$ source domains $S_1,\dots,S_K$, each given by a joint distribution $v$0. The assumed domain shift is

$v$1

while the labeling function is shared:

$v$2

In each source domain $v$3, the observed data consist of a small labeled set

$v$4

and a large unlabeled set

$v$5

The learning objective is to train a single predictor $v$6 using both

$v$7

so that it generalizes to an unseen target domain $v$8 with unknown $v$9 but the same $a$0. Performance is measured by Top-1 accuracy on $a$1, without adaptation (Li et al., 26 Feb 2026).

This formulation is motivated by the observation that existing approaches do not cover the full setting effectively: multimodal domain generalization methods cannot exploit unlabeled data, semi-supervised multimodal learning methods ignore domain shifts, and semi-supervised domain generalization methods are confined to single-modality inputs. A plausible implication is that SSMDG should be viewed as an intersection problem rather than a straightforward extension of any one of those lines of work.

2. Architectural setup and training objective

The framework processes each mini-batch of labeled and unlabeled samples at train time. For modality $a$2, it uses an encoder $a$3, a unimodal classifier $a$4, and a late-fusion classifier $a$5 acting on concatenated features.

On labeled data $a$6, standard fusion and unimodal cross-entropy are applied:

$a$7

The full objective combines supervised loss with three semi-supervised regularizers:

$a$8

with

$a$9

The network and optimization configuration are specified as follows. The video encoder $M=2$0 is SlowFast pre-trained on Kinetics-400 with output dimension $M=2$1. The audio encoder $M=2$2 is ResNet-18 pre-trained on VGGSound with $M=2$3. For three-modal experiments, an optional flow encoder $M=2$4 is SlowFast slow-pathway with $M=2$5. Fusion is simple concatenation $M=2$6. The translators are 2-layer MLPs with hidden size 2048 and ReLU. The optimizer is AdamW with learning rate $M=2$7, weight decay $M=2$8, and batch size 32. The confidence threshold is $M=2$9, the GCE exponent is $$\mathcal{X}=\mathcal{X}_1\times \dots \times \mathcal{X}_M$$0, and the EMA momentum is $$\mathcal{X}=\mathcal{X}_1\times \dots \times \mathcal{X}_M$$1 (Li et al., 26 Feb 2026).

Weak augmentations are random flip and translation for video, gain and pitch for audio, and flips and translations for flow. Strong augmentations are RandAugment plus Cutout for video, SpecAugment plus noise for audio, and Cutout plus noise for flow.

3. Consensus-Driven Consistency Regularization and Disagreement-Aware Regularization

The first two components govern pseudo-label generation and the use of unlabeled samples.

For Consensus-Driven Consistency Regularization (CDCR), each unlabeled sample $$\mathcal{X}=\mathcal{X}_1\times \dots \times \mathcal{X}_M$$2 is weakly augmented per modality, producing features

$$\mathcal{X}=\mathcal{X}_1\times \dots \times \mathcal{X}_M$$3

and predictions

$$\mathcal{X}=\mathcal{X}_1\times \dots \times \mathcal{X}_M$$4

The pseudo-label is defined as

$$\mathcal{X}=\mathcal{X}_1\times \dots \times \mathcal{X}_M$$5

A sample enters the consensus set $$\mathcal{X}=\mathcal{X}_1\times \dots \times \mathcal{X}_M$$6 if three conditions hold: $$\mathcal{X}=\mathcal{X}_1\times \dots \times \mathcal{X}_M$$7 $$\mathcal{X}=\mathcal{X}_1\times \dots \times \mathcal{X}_M$$8, $$\mathcal{X}=\mathcal{X}_1\times \dots \times \mathcal{X}_M$$9 $\mathcal{Y}$0 for at least one $\mathcal{Y}$1, and $\mathcal{Y}$2 that unimodal confidence $\mathcal{Y}$3 as well.

For $\mathcal{Y}$4, consistency is enforced on strongly augmented views $\mathcal{Y}$5 using

$\mathcal{Y}$6

The CDCR loss is

$\mathcal{Y}$7

Disagreement-Aware Regularization (DAR) addresses high-confidence unlabeled samples that fail full consensus. These samples are collected in

$\mathcal{Y}$8

DAR uses the same weak-view pseudo-label $\mathcal{Y}$9 but replaces cross-entropy with the Generalized Cross-Entropy (GCE) loss:

$X=(x_1,\dots,x_M)$0

DAR then enforces GCE consistency across both weak and strong views and all unimodal and fused heads:

$X=(x_1,\dots,x_M)$1

The division between consensus and non-consensus samples is central to the method. This suggests that the framework does not treat pseudo-label reliability as binary in a purely confidence-based sense; instead, it couples confidence with cross-head agreement and then uses a noise-robust objective when agreement is incomplete (Li et al., 26 Feb 2026).

4. Cross-Modal Prototype Alignment and missing-modality robustness

Cross-Modal Prototype Alignment (CMPA) is designed to enforce domain- and modality-invariant representations while promoting robustness under missing modalities via cross-modal translation.

The method maintains per-domain, per-class, per-modality prototypes $X=(x_1,\dots,x_M)$2, updated by exponential moving average using labeled features of class $X=(x_1,\dots,x_M)$3 in domain $X=(x_1,\dots,x_M)$4:

$X=(x_1,\dots,x_M)$5

It also defines cross-domain averaged prototypes

$X=(x_1,\dots,x_M)$6

To handle missing modalities and enforce cross-modal invariance, the framework learns translators $X=(x_1,\dots,x_M)$7 and $X=(x_1,\dots,x_M)$8. Given any feature $X=(x_1,\dots,x_M)$9, the other modality is synthesized as

$x_m\in\mathcal{X}_m$0

For all pseudo-labeled samples in $x_m\in\mathcal{X}_m$1 with pseudo-label $x_m\in\mathcal{X}_m$2 and domain $x_m\in\mathcal{X}_m$3, both original and translated features are aligned to intra-domain and cross-domain prototypes using

$x_m\in\mathcal{X}_m$4

The total CMPA loss is

$x_m\in\mathcal{X}_m$5

The missing-modality mechanism is tied directly to the translation modules rather than to a purely masking-based treatment. In the reported experiments, under 5% labels with missing video, a zero-fill baseline achieved 52.45% and translation achieved 55.03%, a gain of 2.58 points; similar gains were reported on EPIC-Kitchens (Li et al., 26 Feb 2026). A plausible implication is that cross-modal translation is functioning as both a regularizer during training and a reconstruction pathway at test time.

5. Benchmarks, protocols, and quantitative results

The experimental setup uses leave-one-domain-out evaluation on two datasets. EPIC-Kitchens includes domains $x_m\in\mathcal{X}_m$6 and 8 action classes such as put, take, and pour. HAC consists of Human, Animal, and Cartoon domains with 7 classes such as sleep, watch TV, and open door. Label budgets are either fixed $x_m\in\mathcal{X}_m$7-shot per class, with 5 or 10 labels in each source domain, or proportional, with 5% or 10% labeled per class. The protocol holds out one domain for test and trains on both labeled and unlabeled data from the other two domains. Metrics are Top-1 accuracy on the unseen domain and robustness under missing video or audio at test time (Li et al., 26 Feb 2026).

The principal quantitative results are summarized below.

Setting	Prior baseline	Proposed method
HAC, 5 labels per class	CGMatch^M 55.45%	60.77%
EPIC-Kitchens, 5 labels per class	CGMatch^M 36.04%	39.94%
HAC, 10 labels per class	CGMatch^M 59.33%	65.00%
HAC three-modal, 5% labels	NIED-LR^M 62.86%	66.44%
HAC three-modal, 10% labels	NIED-LR^M 63.82%	66.91%

For 5 labels per class, the mean over three splits on HAC improves from 55.45% with the strongest prior baseline, CGMatch^M, to 60.77%, a gain of 5.32 percentage points. On EPIC-Kitchens, the best prior result is 36.04% and the reported SSMDG result is 39.94%, a gain of 3.90 percentage points. For 10 labels per class on HAC, CGMatch^M rises from 59.33% to 65.00%, a gain of 5.67 percentage points. In the three-modal HAC setting with 5% and 10% labels, NIED-LR^M obtains 62.86% and 63.82%, whereas the proposed method reports 66.44% and 66.91%, respectively (Li et al., 26 Feb 2026).

These results are reported both in standard evaluation and in missing-modality scenarios. Within the scope of the reported benchmarks, the framework consistently outperforms strong baselines in both standard and missing-modality settings.

6. Ablations, pseudo-label behavior, and stated limitations

The ablation study isolates the contribution of each module under 5-shot supervision. The supervised-only baseline reports 45.39% on HAC and 31.91% on EPIC-Kitchens. Adding CDCR alone gives 54.76% on HAC and 36.48% on EPIC-Kitchens. Adding DAR alone gives 54.89% on HAC and 36.41% on EPIC-Kitchens. Combining CDCR and DAR yields 58.64% on HAC and 37.85% on EPIC-Kitchens. Using any two of $x_m\in\mathcal{X}_m$8 gives approximately 58% on HAC, and using all three gives 60.77% on HAC and 39.94% on EPIC-Kitchens (Li et al., 26 Feb 2026).

The pseudo-label analysis on HAC reports that CDCR yields the highest per-sample accuracy on pseudo-labels and maintains greater than 70% utilization, while DAR and CMPA further boost utilization by stabilizing feature space. This suggests that pseudo-label quality and feature-space regularity are treated as coupled phenomena rather than independent design goals.

The paper also states several limitations and future directions. Reliance on a fixed high confidence threshold $x_m\in\mathcal{X}_m$9 may under-utilize moderately confident samples, and adaptive or curriculum thresholds could help. The domain shift model assumes only $K$0 changes, so more severe shifts such as open-set classes or $K$1 drift are not addressed. Translators learned for two modalities may not scale to many-modal settings, motivating exploration of shared latent spaces or generative models for missing data. The current prototype alignment treats all classes equally, so class imbalance or fine-grained inter-class relations might require more sophisticated metric learning. Future work may integrate meta-learning for rapid domain adaptation, study theoretical generalization bounds under SSMDG, and extend the framework to foundation models and larger real-world multimodal collections (Li et al., 26 Feb 2026).