MMG-Ego4D: Multimodal Egocentric Action Benchmark

• MMG-Ego4D is a benchmark for multimodal generalization in egocentric action recognition that rigorously tests models under missing and disjoint sensor conditions.
• It leverages re-annotated Ego4D data to enable both standard supervised training and few-shot learning using video, audio, and IMU data fused via transformers.
• Its innovative design, including modality dropout and cross-modal contrastive alignment, enhances robustness for privacy-preserving and resource-efficient deployments.

MMG-Ego4D is a benchmark and research framework for multimodal generalization (MMG) in egocentric action recognition, providing a rigorous platform for studying models’ ability to generalize when data from key modalities such as video, audio, or inertial measurement unit (IMU) sensors are partially missing or entirely disjoint between training and inference. Derived from the large-scale Ego4D dataset and extensively re-annotated, MMG-Ego4D enables evaluation and methodological innovation across both standard supervised and challenging few-shot learning scenarios, incorporating real-world considerations of missing-sensor robustness, privacy, and sensor cost efficiency (Gong et al., 2023).

1. Multimodal Generalization Problem Formulation

Multimodal Generalization (MMG) addresses the following classification setting: Given an egocentric data point $x = \{x^m : m \in \mathcal{M}\}$ composed of up to $M$ modalities (here $M=\{\text{video}, \text{audio}, \text{IMU}\}$), the goal is to learn a classifier

$$f: \{x^m : m \in \mathcal M_{\mathrm{train}}\} \rightarrow y \in \{1, \ldots, C\}$$

where $\mathcal M_{\mathrm{train}} \subseteq \{$video, audio, IMU$\}$ are modalities observed during training. MMG formalizes two robust generalization scenarios:

• Missing-Modality Generalization: At test time, $$\mathcal M_{\mathrm{test}} \subset \mathcal M_{\mathrm{train}}$$, requiring the classifier to maintain accuracy when one or more modalities are unavailable (e.g., deactivated video for privacy).
• Cross-Modal Zero-Shot Generalization: Test-time modalities $\mathcal M_{\mathrm{test}}$ are disjoint from those seen during training ($$\mathcal M_{\mathrm{test}} \cap \mathcal M_{\mathrm{train}} = \emptyset$$). For example, models trained solely on audio and IMU are evaluated on video alone.

This bifurcation supports study of both partial sensor failure and fully non-overlapping sensor regimes, relevant for on-device versus cloud inference, privacy compliance, and cost-constrained deployments.

2. MMG-Ego4D Dataset Construction

The MMG-Ego4D dataset is curated from the Ego4D Moments track and comprises approximately 202 hours of synchronously sampled video (at 4 FPS), audio, and IMU (accelerometer plus gyroscope at 200 Hz). Data construction involves:

• Clip Standardization: Each sample is a 5-second clip, temporally subsampled as above.
• Label Consolidation: Human annotators use WordNet hierarchies to reduce label ambiguity, collapsing multi-label or hierarchical clips to a unique activity class and discarding ambiguous samples.
• Class Splits: 79 total activity classes, partitioned into 65 "base" (for supervised train/test) and 14 "novel" (for few-shot evaluation) ensuring that clips from the same raw video are not split across train/test.
• Few-Shot Protocols: Define standard 5-way 5-shot evaluation episodes on the novel classes (10,000 random episodes).

Data Modality and Resource Statistics

Modality	Raw Memory (KB/s)	Backbone (Type, FLOPs, Params)
Video	593.92	MViT-B, 70.5 GF, 36.5 M
Audio	62.76	AST, ~42.1 GF, 87.0 M
IMU	9.44	ViT-style transformer, ~1.65 GF, 15.6 M

This multimodal configuration enables comprehensive study of sensor fusion, selective ablation, and sensor-rich to sensor-poor transfer.

3. MMG Model Architecture and Training Strategies

3.1. Unimodal Feature Extractors

• Video: MViT-B ($16 \times 4$), pre-trained on Kinetics-400.
• Audio: Audio Spectrogram Transformer (AST), pre-trained on AudioSet.
• IMU: ViT-style transformer, trained from scratch on Ego4D IMU.

3.2. Transformer-Based Fusion

A 2-layer transformer (12 heads, $d=768$) fuses per-modality feature tokens, each augmented by a learnable modality embedding. The fusion module concatenates $$[\mathbf x^{video}_{\rm out}+\mathbf e^{video};\, \mathbf x^{audio}_{\rm out}+\mathbf e^{audio};\, \mathbf x^{IMU}_{\rm out}+\mathbf e^{IMU}]$$ and outputs a sequence of fused tokens. Mean-pooling generates the final latent $\mathbf z_{\rm fuse}$.

3.3. Modality Dropout

To induce missing-modality robustness, each modality is randomly dropped (input zeroed) during training with probability $p=0.6$.

3.4. Cross-Modal Contrastive Alignment

For unified embedding, a Noise Contrastive Estimation (NCE) loss is applied: $$\mathcal L_{\rm align} = \sum_{m\in\{\text{audio}, \text{IMU}\}} -\log \frac{ \exp(\mathbf z_{video}^\top \mathbf z_m/\tau) }{ \exp(\mathbf z_{video}^\top \mathbf z_m/\tau) + \sum_{(\mathbf z')\in \mathcal N} \exp(\mathbf z_{video}^\top \mathbf z'/\tau) }$$ where $\tau = 0.07$, aligning representations across modalities for each clip.

3.5. Cross-Modal Prototypical Loss in Few-Shot

For episodic few-shot settings with possible query-support modality mismatch, class prototypes are computed as

$$c^k_m = \frac{1}{|\mathcal Z^k_m|} \sum_{\mathbf z^k_m\in\mathcal Z^k_m} \mathbf z^k_m$$

and the prototypical loss for a query of modality $n$ is

$\mathcal L_{\rm proto} = -\log P_{y_{\rm true}}$

with the probability $P_k$ defined via cross-modal Euclidean distances.

4. Experimental Protocols and Quantitative Results

Training Pipelines

• Supervised (regular/missing-modality): Unimodal pre-train with $\mathcal L_{CE}$; multimodal train with $\mathcal L_{CE} + \mathcal L_{\rm align}$.
• Supervised zero-shot cross-modal: Unsupervised multimodal pre-train with $\mathcal L_{\rm align}$ on 167 h of unlabeled data; multimodal supervised train on allowed modalities.
• Few-shot (5-way 5-shot): Unimodal $\mathcal L_{CE}$, then multimodal $\mathcal L_{CE} + \mathcal L_{\rm align}$, then meta-training with $\mathcal L_{\rm proto}$.

Key Hyperparameters: Learning rate $1\,\text{e}^{-4}$, AdamW optimizer, supervised batch size $32$, few-shot $10{,}000$ episodes, modality dropout $p=0.6$, contrastive temperature $\tau=0.07$.

Quantitative Benchmarks

Unimodal 5-way 5-shot (Accuracy):

Modality	FLOPs	Acc (%)
Video	70.5 G	58.89
Audio	42.1 G	31.06
IMU	1.65 G	40.07

Multimodal Few-Shot Scenarios (5-way 5-shot):

Scenario (Support → Query)	Accuracy (%)
V+A+I → V+A+I (regular)	63.00
V+A → A (missing-modality)	61.76
V+I → I (missing-modality)	62.79
V+A+I → A (missing-modality)	50.77
V+A+I → I (missing-modality)	47.48
A+I → V (cross-modal zero-shot)	50.80
V → A (cross-modal zero-shot)	44.01
V → I (cross-modal zero-shot)	46.56

All above with a single model weight set.

Supervised Full-Modal and MMG (Top-1 Accuracy):

Train	Test	Top-1 (%)
V+A+I	V+A+I	55.66
V+A+I	A+I	37.07
A+I	V	25.03
V	A+I	43.43

Ablation Studies

• Fusion: Transformer > MLP by 3–4% on MMG.
• Contrastive alignment ($\mathcal L_{\rm align}$): Adds +22.7% (supervised zero-shot) / +17.5% (few-shot zero-shot).
• Prototypical loss ($\mathcal L_{\rm proto}$): Improves few-shot by 0.6–0.8% across tasks.

5. Analysis, Insights, and Limitations

Multimodal transformer fusion integrated with contrastive cross-modal alignment is essential; naïve unimodal or MLP-based concatenation approaches show significant failure modes in both missing-modality and disjoint-modality regimes. Video consistently provides the strongest unimodal performance. Absence of video at inference results in $>$30% supervised top-1 degradation, underscoring its informativeness and the challenge of sensor-lean settings. Cross-modal zero-shot remains feasible (25–50% top-1), yet lags full-modality settings, indicating that deeper advances in unsupervised cross-modal alignment and transfer are needed.

Key application domains include privacy-preserving wearables (e.g., sensor-based disabling of video streams), on-device resource-frugal adaptation (IMU, audio) with cloud-based high-bandwidth inference (video), robust AR/VR assistants, and robotics. Principal open challenges include scaling MMG beyond three modalities (e.g., adding gaze, depth), enhancing unsupervised multimodal pre-training, advancing few-shot cross-modal meta-learning, and fortifying adversarial robustness under missing or corrupted sensors.

6. Public Resources and Benchmark Availability

All data splits, annotation guidelines, evaluation code, and model implementations are available at https://github.com/facebookresearch/MMG_Ego4D, facilitating benchmarking and reproducibility for the research community (Gong et al., 2023).