MDD-Net: Multimodal Depression Detection

• MDD-Net is a class of multimodal frameworks that integrate acoustic, visual, and textual signals to detect and assess depression.
• The architectures employ varied fusion strategies like mutual-transformer, early fusion, and staged optimization for tasks ranging from binary classification to continuous regression.
• Empirical results demonstrate improved screening accuracy and robust fusion, though challenges remain with label noise and low-resource training environments.

Multimodal Depression Detection Network (MDD-Net) denotes a class of neural systems that infer depression-related status from multiple information streams rather than from a single modality. In the literature, the label has been applied to several non-identical architectures spanning social-media vlogs, clinical interviews, smartphone-mediated assessments, and multimodal social-media posts. These systems variously combine acoustic, visual, textual, image, and hand-crafted behavioral signals, and they target binary depression detection, multi-class severity estimation, or continuous score regression rather than one uniform prediction problem (Haque et al., 11 Aug 2025, Lam et al., 2024, Gimeno-Gómez et al., 2024, Zhao et al., 25 Nov 2025, Li et al., 2024, Shalu et al., 2020, Diep et al., 2022).

1. Terminological scope and problem formulations

The term MDD-Net does not identify a single canonical architecture. In one formulation, it is an audio-visual mutual-transformer model for depression detection in social-media vlogs (Haque et al., 11 Aug 2025). In another, it is a text-audio network for DAIC-WOZ built from a pre-trained Transformer, a deep 1D-CNN, and a feed-forward fusion module (Lam et al., 2024). Other works use the same label for a non-verbal video model built around a single Transformer over pretrained feature streams (Gimeno-Gómez et al., 2024), an audio-visual-text multimodal LLM with timestamp-level alignment (Zhao et al., 25 Nov 2025), or an audio-visual regression network for AVEC 2014 that predicts a continuous BDI-II score (Li et al., 2024). Related systems also incorporate one-shot prototype adaptation (Shalu et al., 2020) or explicitly combine deep and clinically informed hand-crafted features (Diep et al., 2022).

Variant	Modalities and target	Core mechanism
(Haque et al., 11 Aug 2025)	Acoustic + visual; binary depression decision	AFEM, VFEM, mutual transformer, detection layer
(Lam et al., 2024)	Text + audio; binary classification	6-layer Transformer text branch, 1D-CNN audio branch, FFN fusion
(Gimeno-Gómez et al., 2024)	Audio + FaceEmo + FaceLM + BodyLM + HandLM + Gaze + Blink; binary classification	Early fusion into a single Transformer encoder
(Zhao et al., 25 Nov 2025)	Audio + video + transcript; binary classification	Whisper-v3, vision encoder, timestamp-level alignment, Qwen2-Audio-7B-Instruct with LoRA
(Li et al., 2024)	Audio + video; BDI-II regression	Attention-CNN, Cov-Attention ResNet, weighted loss fusion
(Shalu et al., 2020)	Video + audio + text; four-way PHQ-8 category prediction	Late fusion MLP with one-shot/prototype adaptation
(Diep et al., 2022)	Audio + text + hand-crafted features; binary screening	Wav2Vec 2.0, RoBERTa, Bi-LSTM, multi-head attention, MLP

This heterogeneity is central to the concept. A plausible implication is that “MDD-Net” functions more as a recurring multimodal design label within depression-detection research than as a standardized benchmark architecture.

2. Architectural families

The 2025 mutual-transformer MDD-Net for social-media vlogs is among the most explicitly modular instantiations. It processes aligned acoustic and visual streams through four sequential modules: an Acoustic Feature Extraction Module (AFEM), a Visual Feature Extraction Module (VFEM), a Mutual Transformer Fusion Module, and a Depression Detection Layer (Haque et al., 11 Aug 2025). Its forward path is specified as

$$X_A^o \leftarrow \mathrm{AFEM}(X_A), \quad X_V^o \leftarrow \mathrm{VFEM}(X_V),$$

followed by two cross-modal mutual-correlation branches, a self-transformer branch over $\mathrm{concat}(X_A^o,X_V^o)$, average pooling, concatenation, and final classification (Haque et al., 11 Aug 2025). The output is a binary decision defined as depressed if $p>0.5$ and normal otherwise.

A distinct family uses parallel unimodal encoders followed by late fusion. In the DAIC-WOZ text-audio MDD-Net, a pre-trained 6-layer Transformer encoder produces a 512-dimensional textual feature vector, while a four-layer 1D-CNN over log-scaled Mel-frequency spectrograms produces a 64-dimensional audio feature vector; the joint representation is

$z_n=[a_n^t;a_n^a]\in\mathbb{R}^{576},$

which is passed to a small feed-forward network for final classification (Lam et al., 2024). The 2020 hybrid multimodal system similarly concatenates video, audio, and text embeddings and feeds them to a fusion MLP, but adds a clinician-facing one-shot or few-shot adaptation loop based on class prototypes in the text-embedding space (Shalu et al., 2020). The speech-screening model of Diep et al. likewise concatenates $h_{audio}$, $h_{text}$, and $h_{hand}$ before a two-layer MLP (Diep et al., 2022).

A third family pushes fusion earlier. The non-verbal video MDD-Net ingests multiple pretrained high-level streams—Audio, FaceEmo, FaceLM, BodyLM, HandLM, Gaze, and Blink—maps each stream into a common $d=256$ space, augments each token with modality and fractional positional embeddings, concatenates all tokens, prepends a global [CLS] token, and performs binary classification from the final hidden state of a standard Transformer encoder (Gimeno-Gómez et al., 2024). By contrast, the multimodal LLM formulation aligns audio and visual embeddings at the timestamp level, fuses them by element-wise addition, concatenates the fused sequence with transcript embeddings, and uses the final [CLS]-token output of Qwen2-Audio-7B-Instruct for prediction (Zhao et al., 25 Nov 2025).

The AVEC 2014 regression model occupies a separate point in the design space. It is a two-stream audio-visual system in which an Attention-CNN audio branch is pretrained to regress depression from speech, a Cov-Attention ResNet video branch models dynamic facial expressions, and fused embeddings are concatenated and regressed to a continuous BDI-II score (Li et al., 2024). Rather than explicit cross-modal attention, multimodal coupling is imposed through a weighted loss that uses the audio branch as a guiding signal during fine-tuning.

3. Modal representations and fusion operators

Acoustic representations vary substantially across MDD-Net variants. The mutual-transformer vlog model uses 25 low-level descriptors per frame, specifically “25 openSMILE GeMAPS acoustic descriptors,” and the AFEM constructs $Q_a$, $K_a$, and $\mathrm{concat}(X_A^o,X_V^o)$0 with a $\mathrm{concat}(X_A^o,X_V^o)$1 convolution before applying content and positional attention (Haque et al., 11 Aug 2025). The DAIC-WOZ text-audio system instead extracts log-scaled Mel-frequency spectrograms and applies four stacked 1D convolutions, each with 64 filters of width $\mathrm{concat}(X_A^o,X_V^o)$2 (Lam et al., 2024). The multimodal LLM system starts from raw waveform sampled at 16 kHz, computes 128-channel Mel-spectrograms with 10 ms hop, and uses a Whisper-v3 backbone to obtain $\mathrm{concat}(X_A^o,X_V^o)$3 (Zhao et al., 25 Nov 2025). Other formulations use PASE+ speech embeddings (Gimeno-Gómez et al., 2024), COVAREP features after CycleGAN denoising (Shalu et al., 2020), Wav2Vec 2.0 base followed by Bi-LSTM and multi-head attention (Diep et al., 2022), or STFT/log-Mel spectrograms (Li et al., 2024).

Visual representations are equally heterogeneous. The mutual-transformer MDD-Net uses “68 dlib facial landmarks” represented as a 136-dimensional vector per frame, then applies patch embedding, GroupNorm, SiLU, and repeated Transformer blocks (Haque et al., 11 Aug 2025). The non-verbal video model uses face emotion embeddings, face, body and hand landmarks, gaze, and blinking information, each passed through a modality-specific encoder and then fused as Transformer tokens (Gimeno-Gómez et al., 2024). The multimodal LLM system encodes “30 fps facial landmarks/AUs” with a strided 1D-conv, Transformer layers, and average pooling to match audio temporal resolution (Zhao et al., 25 Nov 2025). Other systems rely on OpenFace facial-action units, head-pose, and gaze (Shalu et al., 2020) or a 3D ResNet with explicit spatial-temporal attention over face-aligned 64-frame clips (Li et al., 2024).

Textual signals are absent in some MDD-Net variants and central in others. The DAIC-WOZ text branch retains only “7 clinically relevant topics” such as sleep and feelings of failure, uses topic-based data augmentation, and fine-tunes a pre-trained Transformer with 8 heads and hidden dimension 512 (Lam et al., 2024). The multimodal LLM architecture concatenates transcript embeddings with synchronized audio-visual embeddings before classification (Zhao et al., 25 Nov 2025). Other systems use Siamese Sentence-BERT based on RoBERTa (Shalu et al., 2020), RoBERTa base plus Bi-LSTM and attention (Diep et al., 2022), or user-level text embeddings from EmoBERTa, optionally paired with CLIP image embeddings and time2vec positional encoding in social-media depression detection (Bucur et al., 2023).

Fusion operators reflect different assumptions about cross-modal dependence. The mutual-transformer formulation explicitly models bidirectional correlations:

$\mathrm{concat}(X_A^o,X_V^o)$4

with a symmetric term for $\mathrm{concat}(X_A^o,X_V^o)$5, followed by residual, nonlinear, and layer-normalized mutual-correlation outputs (Haque et al., 11 Aug 2025). The text-audio DAIC-WOZ system uses pure concatenation at the embedding level (Lam et al., 2024). The LLM-based system adopts synchronized additive fusion,

$\mathrm{concat}(X_A^o,X_V^o)$6

preserving temporal alignment before conditioning on text (Zhao et al., 25 Nov 2025). The non-verbal video model performs early fusion by concatenating modality-conditioned token sequences and letting self-attention discover dependencies (Gimeno-Gómez et al., 2024). The AVEC 2014 regression model concatenates $\mathrm{concat}(X_A^o,X_V^o)$7 and $\mathrm{concat}(X_A^o,X_V^o)$8 and relies on weighted loss fusion rather than explicit attention (Li et al., 2024).

4. Objectives, optimization, and inference protocols

Loss design is a major axis of differentiation. The mutual-transformer MDD-Net uses a hybrid objective,

$\mathrm{concat}(X_A^o,X_V^o)$9

with label smoothing in the binary cross-entropy term, focal loss, and L2 regularization (Haque et al., 11 Aug 2025). This design is explicitly motivated as a response to class imbalance and noisy labels. The DAIC-WOZ text-audio MDD-Net uses binary cross-entropy with L2 regularization for branch training and fusion (Lam et al., 2024). The non-verbal video model uses standard binary cross-entropy (Gimeno-Gómez et al., 2024). The multimodal LLM formulation combines classification and alignment,

$p>0.5$0

where $p>0.5$1 is a contrastive timestamp-level alignment loss over utterance-pooled audio and visual vectors (Zhao et al., 25 Nov 2025). The AVEC 2014 regression model uses audio and video MAE terms and fuses them as

$p>0.5$2

during end-to-end training (Li et al., 2024).

Training protocols likewise differ by modality and scale. The mutual-transformer MDD-Net reports a “70% train / 10% val / 20% test (10-fold cross-validation)” split, Adam with $p>0.5$3, $p>0.5$4, $p>0.5$5, batch size 8, up to 200 epochs, and early stopping with patience 15 on validation loss (Haque et al., 11 Aug 2025). The DAIC-WOZ text-audio model trains the Transformer with Adam at $p>0.5$6, the CNN at $p>0.5$7, and the fusion network at $p>0.5$8, while topic-based augmentation expands the training data from 107 participants to 534 transcript/audio pairs for single-modality training and to a balanced 307-pair set for multimodal training (Lam et al., 2024). The non-verbal video model uses AdamW, initial learning rate $p>0.5$9, weight decay $z_n=[a_n^t;a_n^a]\in\mathbb{R}^{576},$0, cosine decay over 200 epochs, and batch size 8 (Gimeno-Gómez et al., 2024).

The multimodal LLM system formalizes training in three stages: Stage I visual MAE pretraining, Stage II utterance alignment, and Stage III multimodal instruction tuning. It uses AdamW with learning rates $z_n=[a_n^t;a_n^a]\in\mathbb{R}^{576},$1, $z_n=[a_n^t;a_n^a]\in\mathbb{R}^{576},$2, and $z_n=[a_n^t;a_n^a]\in\mathbb{R}^{576},$3 across those stages; PEFT through QLoRA 4-bit adapters; mixed precision BF16; TF32 on H200 GPUs; and timestamp-synchronized subdialogue sampling of 30–120 s (Zhao et al., 25 Nov 2025). Inference is multi-scale: 200 subdialogues per conversation at durations 30 s, 75 s, and 120 s, with majority vote across scales. A plausible implication is that recent MDD-Net variants increasingly shift from monolithic end-to-end training toward staged optimization and parameter-efficient adaptation.

Evaluation protocols are not uniform. Some papers report Accuracy, Precision, Recall, and F1 (Haque et al., 11 Aug 2025, Lam et al., 2024); others report MAE and RMSE because the target is a continuous BDI-II score (Li et al., 2024); still others emphasize AUC or four-class PHQ-8 severity accuracy (Shalu et al., 2020). This suggests that cross-paper comparisons require attention to label construction, target type, and dataset split rather than only to the headline metric.

5. Benchmarks and reported empirical results

D-Vlog is a prominent benchmark for audio-visual depression detection from social-media vlogs. The mutual-transformer MDD-Net evaluates on “961 vlogs from 816 individuals (322M/639F),” with “555 depressed vs. 406 normal,” where labels are obtained via “vlog-title keywords + manual transcript annotation” (Haque et al., 11 Aug 2025). On this dataset, it reports $z_n=[a_n^t;a_n^a]\in\mathbb{R}^{576},$4, $z_n=[a_n^t;a_n^a]\in\mathbb{R}^{576},$5, and $z_n=[a_n^t;a_n^a]\in\mathbb{R}^{576},$6, with improvements over Bi-LSTM ($z_n=[a_n^t;a_n^a]\in\mathbb{R}^{576},$7), TFN ($z_n=[a_n^t;a_n^a]\in\mathbb{R}^{576},$8), TAMFN ($z_n=[a_n^t;a_n^a]\in\mathbb{R}^{576},$9), and MDAVIF ($h_{audio}$0), including “+17.37%” over Bi-LSTM and “+2.11%” over MDAVIF (Haque et al., 11 Aug 2025). Unimodal ablations yield $h_{audio}$1 for AFEM only and $h_{audio}$2 for VFEM only, while mutual-transformer fusion exceeds Add, Concat, and Multiply by “+3.62% over best alternative” (Haque et al., 11 Aug 2025). The non-verbal Transformer model also reports strong D-Vlog performance, with “F1 = 0.78 ±0.01, p < .01 over previous SOTA (0.75),” and a monotonic ablation trend from AV to AV+EYES to AV+LM to AV+LM+EYES, namely $h_{audio}$3 (Gimeno-Gómez et al., 2024).

DAIC-WOZ anchors several clinically oriented MDD-Net variants. The text-audio DAIC-WOZ MDD-Net evaluates on the development set under Text-only, Audio-only, and Text+Audio settings and reports its best multimodal configuration, Trf+CNN-Augm, at $h_{audio}$4, $h_{audio}$5, and $h_{audio}$6, versus a prior SOTA of $h_{audio}$7 (Lam et al., 2024). In the same study, the best text-only model reaches $h_{audio}$8 and the best audio-only model $h_{audio}$9 (Lam et al., 2024). The multimodal LLM MDD-Net uses 189 DAIC-WoZ interviews with PHQ-8 thresholding at $h_{text}$0, split into Train 107, Dev 35, and Test 47, and reports Dev $h_{text}$1 for “MDD-Net (full),” Dev $h_{text}$2 for “MDD-Net (no MS),” and Test $h_{text}$3 on DAIC-WoZ, outperforming HiQuE at $h_{text}$4 and MultiDepNet at $h_{text}$5 on the test set (Zhao et al., 25 Nov 2025). Its ablation study reports $h_{text}$6 for text only, $h_{text}$7 for text plus audio, $h_{text}$8 after adding video, and $h_{text}$9 after multi-scale inference; removing the alignment stage lowers Dev $h_{hand}$0 from $h_{hand}$1 to $h_{hand}$2 (Zhao et al., 25 Nov 2025).

AVEC 2014 supports the regression-oriented audio-visual MDD-Net. Using training, development, and test sets of 150, 50, and 100 videos, respectively, the model reports “MAE = 7.23, RMSE = 9.36” on depression detection, versus audio-only “MAE = 7.64” and video-only “MAE = 7.58” (Li et al., 2024). Its ablation shows “Audio+Video (no $h_{hand}$3 guidance) MAE ≈ 7.42” and “Full MDD-Net ($h_{hand}$4) MAE = 7.23” (Li et al., 2024). A separate hybrid video-audio-text system on DAIC-WOZ/AVEC reports “Accuracy = 96.3% (±1.48%) and AUC = 0.9682” for four-way PHQ-8 categorization (Shalu et al., 2020).

Beyond video and interview corpora, multimodal depression detection extends to user-level social media and speech screening. A time-enriched multimodal transformer using EmoBERTa and CLIP reaches “0.931 F1 score on a popular multimodal Twitter dataset” and “0.902 F1 score on the only multimodal Reddit dataset” (Bucur et al., 2023). A deep-plus-hand-crafted speech-screening model on the DEPAC corpus improves depression “overall F1 score” from “0.58” with hand-crafted features alone to “0.63” when deep-learned audio and text features are added (Diep et al., 2022). These results indicate that the MDD-Net design space is not restricted to one dataset family or one modality combination.

6. Limitations, misconceptions, and research directions

A recurring limitation is label quality. The mutual-transformer D-Vlog model explicitly notes that D-Vlog labels are noisy because they are keyword-based, with attendant “mislabeling risk” (Haque et al., 11 Aug 2025). Social-media depression datasets used in multimodal transformers also rely on self-reported diagnoses and demographic skews, while the exact timing of self-reports may be unknown (Bucur et al., 2023). A plausible implication is that gains reported on user-generated-content benchmarks partly depend on how weak supervision is operationalized.

Another limitation is scale relative to model capacity. The mutual-transformer paper characterizes D-Vlog as “modest for deep transformer training” and warns of possible overfitting (Haque et al., 11 Aug 2025). The non-verbal video model emphasizes missing data, variable frame rates, lighting variation, camera motion, and background noise in “in-the-wild” vlogs (Gimeno-Gómez et al., 2024). The multimodal LLM paper, despite stronger results, still motivates timestamp alignment and PEFT partly to reduce the need for “extensive training data and computational resources” (Zhao et al., 25 Nov 2025). These observations collectively suggest that multimodal depression detection remains a low-resource regime relative to contemporary large-scale multimodal pretraining.

Several misconceptions are addressed by the literature itself. First, multimodality is not synonymous with text inclusion: some high-performing MDD-Net variants are explicitly non-verbal, using only acoustic and visual cues (Haque et al., 11 Aug 2025, Gimeno-Gómez et al., 2024), whereas others treat text as indispensable (Lam et al., 2024, Zhao et al., 25 Nov 2025, Diep et al., 2022). Second, not all MDD-Net systems perform the same task: binary classification, four-way PHQ-8 categorization, and BDI-II regression are all represented (Shalu et al., 2020, Li et al., 2024). Third, multiple works explicitly frame these systems as screening aids rather than medical diagnosis; the non-verbal video model states that it is “a screening tool, not a medical diagnosis,” and the social-media transformer likewise distinguishes screening from professional assessment (Gimeno-Gómez et al., 2024, Bucur et al., 2023).

Future directions recur across the corpus. Proposed extensions include adding textual modality to non-verbal systems, experimenting with early, middle, and late fusion hybrids, using gating mechanisms, pretraining on larger audio-visual emotion datasets, exploring factorized or tensor-based fusion, and incorporating temporal convolutional backbones (Haque et al., 11 Aug 2025). Other works propose explicit cross-modal attention for fusion, self-supervised pretraining on large emotion corpora, inclusion of physiological signals, and end-to-end pretraining on larger multimodal mental-health corpora (Li et al., 2024, Zhao et al., 25 Nov 2025). Taken together, these proposals indicate an ongoing shift from narrowly specified bimodal detectors toward temporally aligned, parameter-efficient, and extensible multimodal systems that can integrate additional synchronized streams without abandoning depression-specific inductive structure.