DaQ-MSA: Diffusion Augmentations for MSA

• The paper introduces DaQ-MSA, a novel framework that uses diffusion-based augmentations and a quality scoring module to tackle data scarcity and annotation noise in sentiment analysis.
• DaQ-MSA employs specialized methods, FateZero for video style transfer and Seed-VC for voice conversion, to generate semantically consistent and diverse augmentations.
• Empirical results on benchmarks like CH-SIMS, CMU-MOSI, and MUStARD demonstrate significant performance improvements, validating enhanced data efficiency and sentiment analysis accuracy.

DaQ-MSA (Denoising and Qualifying Diffusion Augmentations for Multimodal Sentiment Analysis) is a framework that addresses data scarcity and annotation noise in multimodal sentiment analysis (MSA) by generating and qualifying high-fidelity, semantics-preserving augmentations of video and audio modalities. Leveraging off-the-shelf diffusion models and an explicit quality-aware scoring mechanism, DaQ-MSA improves the data efficiency and generalization capacity of Multimodal LLMs (MLLMs) on sentiment benchmarks, operating without additional manual annotation or external supervision (Liang et al., 11 Jan 2026).

1. Motivation and Context

Multimodal sentiment analysis seeks to infer human affect by processing the joint interaction of text, visual, and acoustic signals. Existing sentiment datasets—such as MOSI, MOSEI, CH-SIMS, and MUStARD—are limited by their small size, inconsistent annotation quality, subjective sentiment labels, and frequent cross-modal temporal misalignments. These constraints make it difficult for MLLMs to generalize, particularly since model capacity alone does not compensate for inadequate or misaligned training samples. Traditional data augmentation strategies (random cropping, color jitter, pitch shifting) only affect low-level signal statistics and fail to deliver the semantic diversity necessary for robust learning.

Diffusion models have recently shown promise in augmenting vision, video, and speech data via semantic-preserving transformations, yet the na\"ive inclusion of all generated samples can introduce artifacts (lip-sync errors, prosodic distortions, sentiment misalignment) that degrade model performance. DaQ-MSA addresses this by coupling diffusion-based augmentation with an explicit quality-aware scoring and reweighting paradigm, thereby filtering out unreliable synthetic data and enhancing the learning process.

2. Diffusion Augmentation Methodology

DaQ-MSA utilizes two specialized diffusion models:

• FateZero is employed for video style transfer. This method inverts the latent space of a source video using DDIM-based inversion, producing latent vectors $z_T$ that are progressively denoised. During synthesis, self-attention maps from the inversion and editing stages are fused using a thresholded mask derived from cross-attention responses, enabling the creation of semantically preserved, stylistically varied videos. Temporal coherence between frames is maintained through an attention mechanism that jointly considers each frame and temporally warped neighbors.
• Seed-VC is used for zero-shot voice conversion. Here, a flow-matching diffusion model learns a vector field that approximates deterministic trajectories (flows) in audio space under timbre perturbations. At inference, the model solves a deterministic ODE to convert the timbre of input speech to that of a randomly selected target speaker, while preserving the semantic content.

Both methods operate according to the standard diffusion formalism: a forward process incrementally adds noise to the input, and a learned reverse process denoises it conditioned on the desired attributes (e.g., prompt for video style, speaker embedding for audio).

Augmented samples $(V_i^{aug}, A_i^{aug})$ are paired with their original sentiment-bearing transcripts $T_i$, expanding the training distribution with cross-modally consistent yet presentation-diverse data, without additional annotation.

3. Quality Scoring and Semantic Consistency

Diffusion-generated augmentations exhibit variable fidelity. To ensure only high-quality samples inform model updates, DaQ-MSA introduces a decoupled quality-aware (QA) scoring module. This module explicitly measures the cross-modal semantic consistency and reliability of augmented samples. It operates as follows:

• Extracts feature embeddings from each modality: SigLIP for video, Whisper for audio, BERT (plus a learned mapping) for text, and an embedding of the style or speaker prompt.
• Concatenates these features into a unified representation, which is passed through a two-layer MLP with GELU activation to produce a scalar quality score $Q(x)=s$, interpreted as the reliability of the sample.
• The QA module is trained to discriminate between positive (high-fidelity) and three types of negative (flawed) samples: (i) feature mixing (modality swapping with opposite sentiment), (ii) random masking (partial feature occlusion), and (iii) label flipping.

The weighted sum of binary cross-entropy losses over these cases supervises the module, teaching it to recognize and penalize semantically misaligned or noisy synthetic data.

4. Adaptive Training Reweighting

Each augmented sample $i$ receives a QA module-derived score $s_i$ that is mapped via a tunable power law to a training weight $$w_i = w_{\min} + s_i^\gamma \,(w_{\max} - w_{\min})$$. Real (human) samples always receive full weight ($w_i=1$). During downstream MLLM fine-tuning, the per-sample cross-entropy loss $\bar{\ell}_i$ is weighted by $w_i$, resulting in an overall objective:

$$\mathcal{L}_{\text{task}} = \frac{1}{B}\sum_{i=1}^B w_i\,\bar\ell_i$$

This approach systematically reduces the negative impact of low-fidelity augmentations and intensifies the influence of reliable, semantically consistent synthetic data, leading to more stable and effective gradient flows during learning.

5. Benchmark Evaluation and Empirical Results

DaQ-MSA's efficacy is demonstrated on three benchmarks:

Dataset	Language	Task/Classes	# Clips	Main Metrics
CH-SIMS	Chinese	5-class sentiment	2,281	Acc5, Acc2, F1, MAE, Corr
CMU-MOSI	English	7-class sentiment	2,199	Acc7, Acc2, F1, MAE, Corr
MUStARD	English	Sarcasm (binary)	690	Weighted Accuracy, F1

On CH-SIMS, DaQ-MSA achieves 61.49% Acc5, 90.15% Acc2, 0.240 MAE, and 0.832 Corr, improving Acc5 by +9.39 and Acc2 by +5.05 points over the best baseline (HumanOmni). On CMU-MOSI, it achieves 55.33% Acc7, 92.37% Acc2, 0.498 MAE, and 0.907 Corr, with Acc7 up by +2.53 and a 4.8% reduction in MAE. On MUStARD, DaQ-MSA attains 70.59% weighted accuracy and F1, +3.59 points above the strongest prior model.

Ablation studies on CH-SIMS show that diffusion augmentations alone boost performance (60.39% Acc5 vs. 52.08% without augmentation), while full DaQ-MSA with quality reweighting further improves binary accuracy (90.15% Acc2). The value of the QA module is particularly pronounced with limited annotations: with only 10% labeled data, DaQ-MSA retains 82.12% Acc2, outperforming unweighted mixing by 3.35 points.

Modality-focused ablations confirm that video-based style transfer most benefits text+video sentiment analysis, while voice conversion yields moderate gains in text+audio settings.

6. Analysis of Method Properties and Limitations

DaQ-MSA's high-quality diffusion augmentations enhance affective style diversity without sacrificing semantic content, critical for fine-grained sentiment discrimination. Noisy synthetic samples—especially those with lip-sync or prosodic artifacts—can mislead model predictions if not properly mitigated. The QA-driven weighting approach effectively filters such augmentations, focusing model updates on data with reliable sentiment cues.

The framework incurs non-trivial computational overhead due to the need for 50 DDIM steps per video and 1,000 training epochs for voice conversion. It is consequently not suited for real-time, online augmentation. The QA module is limited to down-weighting poorly aligned samples rather than directly repairing them, and occasional artifact leakage (e.g., lip-sync mismatches, style overfitting) can still occur.

Cross-lingual efficacy is established on both Chinese and English datasets; however, the framework's adaptability to low-resource languages or culturally complex sentiment remains an open research question.

7. Directions for Future Research

Prospective research avenues include optimizing diffusion inference through distillation or acceleration to reduce computational demands, integrating explicit alignment losses into the generative models to lower artifact rates, and broadening the QA scoring paradigm to handle additional modalities (such as physiological signals) or more nuanced affective taxonomies. This suggests the potential for DaQ-MSA’s annotation-free augmentation and qualification paradigm to generalize to broader multimodal understanding challenges, contingent upon future empirical validation in diverse domains.

(Liang et al., 11 Jan 2026)