DRF: Distribution-based Feature Recovery & Fusion

• DRF is a robust multimodal framework that recovers missing modality features via statistical distribution modeling and cross-modal mapping.
• It uses FIFO queue-based distribution estimation and both deterministic and diffusion-based recovery to ensure each modality contributes quality data.
• Empirical results demonstrate that DRF improves accuracy and resilience under modality corruption in tasks like sentiment analysis and emotion recognition.

Distribution-based Feature Recovery and Fusion (DRF) refers to a family of techniques for robust multimodal learning—particularly in contexts where certain modalities may be missing or corrupted—by leveraging statistical distribution modeling of modality-specific feature representations. DRF is designed to achieve stable and accurate fusion of modalities (such as image and text, or audio, vision, text) by quantifying data quality, recovering missing modalities via cross-modal mapping or generative modeling, and integrating multiple signals with distribution-sensitive weighting. This framework has recently advanced state-of-the-art performance in multimodal sentiment analysis and incomplete multimodal emotion recognition (Wu et al., 24 Nov 2025, Jin et al., 23 May 2025).

1. Conceptual Foundations of DRF

DRF operates on the principle that robust multimodal fusion requires both the ability to recover plausible representations when certain modalities are absent or unreliable, and to fuse multiple inputs in a way that respects their empirical distributions and quality. In practice, this is achieved through two core strategies:

• Distribution modeling of unimodal features (e.g., via feature queues or generative models), which enables detection and quantification of “in-distribution” versus “out-of-distribution” (OOD) features.
• Distribution-conditioned recovery and adaptive fusion, where missing or low-quality modality representations are reconstructed from other modalities, and the eventual fusion is weighted to reflect the quality of each input.

In DRF, empirical feature distributions play a central role both in recalibrating the impact of noisy data and in guiding reconstruction of missing content (Wu et al., 24 Nov 2025, Jin et al., 23 May 2025).

2. Queue-Based Distribution Modeling and Modality Quality Estimation

DRF as introduced in "Robust Multimodal Sentiment Analysis with Distribution-Based Feature Recovery and Fusion" (Wu et al., 24 Nov 2025) maintains a fixed-size FIFO queue $Q_m$ for each modality $m$ (e.g., $v$ for vision, $t$ for text), where

$Q_m = \{ f_j^m \mid j \in q_m \}, \quad |q_m|=L.$

This supports real-time estimation of empirical mean $\mu_m$ and standard deviation $\sigma_m$:

$$\mu_m = \frac{1}{L} \sum_{j \in q_m} f_j^m, \quad \sigma_m = \sqrt{\frac{1}{L}\sum_{j \in q_m}\|f_j^m - \mu_m\|_2^2}$$

New features are enqueued if their estimated Gaussian likelihood $p(f_i^m \mid \mu_m, \sigma_m)$ exceeds the running mean over the queue. The modality quality score $\alpha_i^m = p(f_i^m \mid \mu_m, \sigma_m)$, directly quantifies how in-distribution a feature is, and thus modulates its contribution to downstream fusion. Features generated via inter-modal recovery mappings (i.e., $f_i^{v\to t}, f_i^{t\to v}$) are also assessed via the same distributional scoring.

3. Recovery Mechanisms for Missing Modalities

Two major DRF recovery methodologies are established:

3.1. Deterministic Cross-Modal Mapping (Wu et al., 24 Nov 2025)

Missing features are reconstructed using learnable converters $C_{v \to t}$ and $C_{t \to v}$ (2-layer MLPs), each trained with both sample-based loss

$$L_{v\to t}^s = \lambda_i^v \lambda_i^t \|C_{v\to t}(f_i^v) - f_i^t\|_2$$

and distribution-based alignment loss

$$L_{v\to t}^d = \| \mu_{v\to t} - \mu_t \|_2 + | \sigma_{v\to t} - \sigma_t |$$

where $\mu_{v\to t}, \sigma_{v\to t}$ are empirical moments over mapped features.

3.2. Diffusion-Based Conditional Generation (Jin et al., 23 May 2025)

RoHyDR’s DRF leverages Denoising Diffusion Probabilistic Models (DDPMs) to sample missing modality features conditioned on the remaining modalities’ features. The forward process applies progressive Gaussian noise to $x_m$; the reverse process, parameterized by a transformer-based $\epsilon_\theta$, attempts to denoise back to a plausible $x_m$ given observed modalities. A unimodal reconstructor MLP further refines these outputs, minimizing reconstruction error with respect to present/target features.

4. Distribution-Aware Fusion Process

Fusion in DRF is performed by constructing several “views” for each sample—combinations of original and recovered modalities. For the image-text case (Wu et al., 24 Nov 2025), three pairs are constructed:

• $(f_i^v, f_i^t)$
• $(f_i^v, f_i^{v\to t})$
• $(f_i^{t\to v}, f_i^t)$

Each concatenated pair is projected with a fusion MLP $F_{v+t}$, and a modality-presence indicator $\lambda_i^m \in \{0,1\}$ is used. The aggregate fused representation is a weighted sum:

$$\begin{align*} M_i = &~ \lambda_i^v\lambda_i^t(\alpha_i^v\alpha_i^t)F_{v+t}([f_i^v,f_i^t]) \ &+ \lambda_i^v(\alpha_i^v\alpha_i^{v\to t})F_{v+t}([f_i^v, f_i^{v\to t}]) \ &+ \lambda_i^t(\alpha_i^{t\to v}\alpha_i^t)F_{v+t}([f_i^{t\to v},f_i^t]) \end{align*}$$

These weights ensure that unreliable features contribute minimally to the final embedding.

In RoHyDR (Jin et al., 23 May 2025), DRF fuses recovered and available modality features with a multimodal transformer, followed by reconstruction and discrimination via adversarial learning to match the distribution of fully observed cases.

5. Loss Functions and Constraints

DRF jointly optimizes:

• Classification loss $L_{cls}$: Cross-entropy against ground truth, applied to the fused representation.
• Recovery loss $L_{rec}$: Sum of sample-based and distribution-alignment losses over all cross-modal mappings.
• Distribution constraint $L_{dis}$: Encourages each feature to cluster tightly around its own modality mean and remain distant from opposing modality means:

$$L_{dis} = \lambda_i^v e^{(\|f_i^v-\mu_v\|_2 - \|f_i^v-\mu_t\|_2)} + \lambda_i^t e^{(\|f_i^t-\mu_t\|_2 - \|f_i^t-\mu_v\|_2)}$$

The aggregate loss is $L = L_{cls} + L_{rec} + L_{dis}$ (Wu et al., 24 Nov 2025).

RoHyDR (Jin et al., 23 May 2025) develops an expanded DRF-inspired regime with three-stage optimization: unimodal recovery (diffusion and refinement losses), fusion recovery (adversarial and feature alignment losses), and supervised classification.

6. Empirical Protocols, Benchmarks, and Results

The effectiveness of DRF is validated through comprehensive experiments:

Dataset	Task	Main DRF Strategy	Empirical Finding
MVSA-S	Image-Text Sentiment	Queue/model fusion	+3.8% ACC over prior SOTA under fixed disruptions
TumEmo	Image-Text Emotion	Queue/model fusion	+3.5% (image) / +2.6% (text) ACC under disruptions
MOSI/MOSEI	Audio-Text-Video	Diffusion + adversary	Top ACC/F1 across all missing-modality and availability

Under random disruptions ($dr=1.0$), prior SOTA methods’ accuracy declines by as much as 18%, while DRF’s accuracy drops only 2.5–6.5% (Wu et al., 24 Nov 2025). In ablation studies, removing distribution-guided recovery, Gaussian-weighted fusion, or “three-view” expansion each causes substantial performance loss, underscoring the necessity of each DRF component.

RoHyDR (Jin et al., 23 May 2025) further demonstrates that omitting the diffusion mechanism or adversarial fusion sharply reduces incomplete multimodal emotion recognition performance (e.g., >4% drop in ACC$_2$).

7. Qualitative Insights and Model Behavior

t-SNE analyses in (Wu et al., 24 Nov 2025) show that at low disruption rates, both sample-based and distribution-based mappings produce well-aligned recovered features, but at maximal corruption (all samples missing one modality), only the distribution-guided recovery preserves correct clustering structure.

A plausible implication is that DRF not only addresses the incomplete modality problem but inherently regularizes the feature space, allowing for robust downstream classification even in the presence of severe real-world data corruption or dropout. The multiplicity of “views” and the explicit distributional regularization are particularly beneficial in stabilizing cross-modal representations.

---

References:

• "Robust Multimodal Sentiment Analysis with Distribution-Based Feature Recovery and Fusion" (Wu et al., 24 Nov 2025).
• "RoHyDR: Robust Hybrid Diffusion Recovery for Incomplete Multimodal Emotion Recognition" (Jin et al., 23 May 2025).