MMFformer: Multimodal Depression Detection

• The paper introduces a multimodal fusion transformer that combines facial video and speech audio to detect depression in social media vlogs.
• It employs a dual-branch architecture with residual video transformers and audio transformer encoders enhanced by three fusion strategies: IT, LT, and IA.
• Empirical results on D-Vlog and LMVD datasets show significant F1-score improvements (up to 13.92%) over previous depression detection methods.

MMFformer is a multimodal fusion transformer network for depression detection from social media video blogs that jointly models facial video and speech audio and learns their interdependencies over time. It is designed to retrieve depressive spatio-temporal high-level patterns from multimodal social media information by combining a transformer network with residual connections for spatial features in video, a transformer encoder for temporal dynamics in audio, and fusion modules that implement late and intermediate fusion strategies to identify intermodal correlations. On two large-scale depression detection datasets, D-Vlog and LMVD, it is reported to improve the F1-Score by 13.92% and 7.74%, respectively, over prior approaches (Haque et al., 8 Aug 2025).

1. Terminological scope and disambiguation

The label “MMFformer” is not unique across recent arXiv literature, and close variants denote distinct architectures in different domains. In the strict sense used here, MMFformer refers to the depression-detection model introduced in “MMFformer: Multimodal Fusion Transformer Network for Depression Detection” (Haque et al., 8 Aug 2025). A common misconception is to conflate it with similarly named models whose objectives, modalities, and architectural choices differ substantially.

Name	Domain	Distinguishing description
MMFformer (Haque et al., 8 Aug 2025)	Depression detection	Multimodal social-media video and audio fusion for binary depression classification
MMFormer (Zhang et al., 2023)	Remote sensing	HSI+LiDAR classification with convolutional tokenization and Multi-scale Multi-head Self-Attention
mmFormer (Zhang et al., 19 Sep 2025)	Medical imaging	Multimodal MRI classification with missing-modality compensation for liver fibrosis staging
MMformer (Xin et al., 18 Apr 2025)	Time-series forecasting	Meta-learning multivariate time-series forecasting with Adaptive Transferable Multi-head Attention
FMMformer (Nguyen et al., 2021)	Efficient sequence modeling	Transformer with decomposed near-field and far-field attention

This distinction is important because only the depression-detection MMFformer operates on audiovisual social-media data and evaluates on D-Vlog and LMVD (Haque et al., 8 Aug 2025). By contrast, the remote-sensing MMFormer uses Hyperspectral Image accompanied by LiDAR for remote sensing image classification and introduces convolutional layers plus a Multi-scale Multi-head Self-Attention module (Zhang et al., 2023). The medical-imaging mmFormer retains hybrid modality-specific encoders and a modality-correlated encoder while adding missing-modality compensation with zero-padding, modality masks, and a Delta Function (Zhang et al., 19 Sep 2025). The environmental MMformer combines self-attention and meta-learning for multivariate time-series forecasting (Xin et al., 18 Apr 2025). FMMformer is a separate efficient-transformer family inspired by the fast multipole method and decomposes attention into near-field and far-field components (Nguyen et al., 2021).

2. Problem formulation and motivation

MMFformer addresses automated depression detection from social media video blogs by treating the task as binary classification, depressed versus non-depressed, from audiovisual content (Haque et al., 8 Aug 2025). The motivating assumption is that depression cues are subtle and distributed across modalities and time: facial micro-expressions, head pose and eye gaze dynamics, prosody, speaking rate, and articulation vary over sequences of seconds to minutes. Single-modality models and static snapshot analysis therefore risk missing complementary and temporally evolving signals.

The model’s design is organized around three motivations stated in the source material. First, facial and vocal signals are complementary; depression may be expressed more in vocal prosody than facial expression for some individuals, and vice versa. Second, cross-modal interactions are informative; incongruent smiling with flat prosody is given as a representative case. Third, the signals are non-stationary, so temporal modeling must accommodate long-range dependencies. Transformers are used for this purpose because they capture temporal structure without relying on manually crafted features (Haque et al., 8 Aug 2025).

The datasets reinforce this problem framing. D-Vlog contains 961 vlogs, approximately 160 hours, from 816 individuals on YouTube, and LMVD contains 1,823 vlogs, approximately 214 hours, from 1,475 individuals across Bilibili, TikTok, Sina Weibo, and YouTube (Haque et al., 8 Aug 2025). These are large, in-the-wild corpora rather than tightly controlled laboratory recordings. A plausible implication is that the architecture is intended not merely for multimodal fusion in the abstract, but for fusion under heterogeneous recording conditions, socially natural behavior, and variable temporal extent.

3. Modality-specific representation learning

MMFformer uses two transformer branches, one for video and one for audio, with standard multi-head self-attention and pre-norm residual structure (Haque et al., 8 Aug 2025). The video branch is described as a transformer network with residual connections that captures spatial features from videos, while the audio branch is a transformer encoder that models temporal dynamics in speech.

For video, the input tensor is denoted by $\mathcal{X}_v$, temporally downsampled by a 1D convolutional block to a fixed temporal length $L$, and then tokenized with a linear patch embedding. The source gives the sequence of operations as

$$\widetilde{\mathcal{X}_v} = \mathcal{F}_{d}(\mathcal{X}_v),$$

$$\mathcal{X}_\mathcal{E} = \mathcal{F}_{emb}(\widetilde{\mathcal{X}_v}) = \mathcal{W}_{emb} \widetilde{\mathcal{X}_v},$$

with $$\mathcal{W}_{emb} \in \mathbb{R}^{\mathcal{C} \times \mathcal{D}}$$. A learnable $[\mathrm{CLS}]$ token is prepended and learnable positional encodings are added:

$$\mathcal{X}_v^{(0)} = \mathcal{T}_{cls} \oplus \mathcal{X}_\mathcal{E},$$

$$\mathcal{X}_v^{(1)} = \mathcal{X}_v^{(0)} + \mathcal{P}.$$

The paper states that a pre-trained Vision Transformer backbone is used, but the exact number of transformer blocks, heads, and the value of $D$ are not disclosed (Haque et al., 8 Aug 2025).

For audio, the input is mapped to a time-frequency representation $$\mathcal{X}_{f} \in \mathbb{R}^{\mathcal{F} \times \mathcal{T}}$$ and standardized to $L$0 through learnable projection using convolution, normalization, and pooling:

$L$1

Overlapping 2D convolutional patches of size $L$2 and stride $L$3 produce $L$4 patches, which are flattened to tokens. Positional embeddings are adapted from Audio Spectrogram Transformer by bilinearly interpolating a base positional grid to $L$5 and flattening it to $L$6. The embedded sequence includes both $L$7 and a distillation token $L$8, giving total length $L$9 (Haque et al., 8 Aug 2025).

Both branches use conventional multi-head self-attention:

$$\widetilde{\mathcal{X}_v} = \mathcal{F}_{d}(\mathcal{X}_v),$$0

$$\widetilde{\mathcal{X}_v} = \mathcal{F}_{d}(\mathcal{X}_v),$$1

$$\widetilde{\mathcal{X}_v} = \mathcal{F}_{d}(\mathcal{X}_v),$$2

Residual and feed-forward updates are given as

$$\widetilde{\mathcal{X}_v} = \mathcal{F}_{d}(\mathcal{X}_v),$$3

$$\widetilde{\mathcal{X}_v} = \mathcal{F}_{d}(\mathcal{X}_v),$$4

In the video branch, the block update is written

$$\widetilde{\mathcal{X}_v} = \mathcal{F}_{d}(\mathcal{X}_v),$$5

$$\widetilde{\mathcal{X}_v} = \mathcal{F}_{d}(\mathcal{X}_v),$$6

In the audio branch, the encoder layer is

$$\widetilde{\mathcal{X}_v} = \mathcal{F}_{d}(\mathcal{X}_v),$$7

$$\widetilde{\mathcal{X}_v} = \mathcal{F}_{d}(\mathcal{X}_v),$$8

4. Fusion mechanisms and learning objective

The defining property of MMFformer is its use of explicit intermodal fusion rather than simple post hoc concatenation (Haque et al., 8 Aug 2025). Three fusion strategies are implemented: Intermediate Transformer Fusion (IT), Late Transformer Fusion (LT), and Intermediate Attention Fusion (IA).

In Intermediate Transformer Fusion, each branch first applies Conv1D to obtain intermediate representations, after which transformer blocks exchange information via cross-attention before final pooling. In Late Transformer Fusion, cross-modal interaction occurs near the end of each branch; each branch queries its own features and attends to the other modality’s keys and values, and the outputs are pooled and concatenated prior to classification. Intermediate Attention Fusion uses attention-based interaction without direct feature mixing: dot-product attention highlights salient cross-modal features and produces attention vectors that refine each branch before pooling and concatenation (Haque et al., 8 Aug 2025).

The general cross-attention form given for video attending to audio is

$$\widetilde{\mathcal{X}_v} = \mathcal{F}_{d}(\mathcal{X}_v),$$9

The paper’s LT and IT instantiation is written, for acoustic queries to visual keys and values, as

$$\mathcal{X}_\mathcal{E} = \mathcal{F}_{emb}(\widetilde{\mathcal{X}_v}) = \mathcal{W}_{emb} \widetilde{\mathcal{X}_v},$$0

For IA, which omits value multiplication, the attention weights are

$$\mathcal{X}_\mathcal{E} = \mathcal{F}_{emb}(\widetilde{\mathcal{X}_v}) = \mathcal{W}_{emb} \widetilde{\mathcal{X}_v},$$1

and the visual attention vector is aggregated as

$$\mathcal{X}_\mathcal{E} = \mathcal{F}_{emb}(\widetilde{\mathcal{X}_v}) = \mathcal{W}_{emb} \widetilde{\mathcal{X}_v},$$2

The final classifier pools transformer outputs, typically by using the $$\mathcal{X}_\mathcal{E} = \mathcal{F}_{emb}(\widetilde{\mathcal{X}_v}) = \mathcal{W}_{emb} \widetilde{\mathcal{X}_v},$$3 token or global pooling, concatenates the modality representations, and applies a fully connected classifier with softmax activation for binary depression detection. The loss function is standard cross-entropy:

$$\mathcal{X}_\mathcal{E} = \mathcal{F}_{emb}(\widetilde{\mathcal{X}_v}) = \mathcal{W}_{emb} \widetilde{\mathcal{X}_v},$$4

The paper does not report class weighting, focal loss, probability-weighted late fusion, or logit-level linear fusion; it emphasizes learned fusion through cross-attention blocks instead (Haque et al., 8 Aug 2025).

5. Datasets, protocol, and empirical results

The empirical evaluation uses 10-fold cross-validation on D-Vlog and LMVD, and each experiment reports mean $$\mathcal{X}_\mathcal{E} = \mathcal{F}_{emb}(\widetilde{\mathcal{X}_v}) = \mathcal{W}_{emb} \widetilde{\mathcal{X}_v},$$5 standard deviation over 10 runs (Haque et al., 8 Aug 2025). D-Vlog provides acoustic features via OpenSMILE eGeMAPS and visual face landmarks via Dlib, sampled at 1 Hz. LMVD provides VGGish audio embeddings together with visual features including facial action units, landmarks, eye gaze, and head pose. Reported metrics include Accuracy, Precision, Recall, and F1-score for both weighted averages and unweighted averages. The source defines

$$\mathcal{X}_\mathcal{E} = \mathcal{F}_{emb}(\widetilde{\mathcal{X}_v}) = \mathcal{W}_{emb} \widetilde{\mathcal{X}_v},$$6

$$\mathcal{X}_\mathcal{E} = \mathcal{F}_{emb}(\widetilde{\mathcal{X}_v}) = \mathcal{W}_{emb} \widetilde{\mathcal{X}_v},$$7

Dataset	Best configuration	Key reported result
D-Vlog	IT fusion	Acc 0.8108, Pr 0.8924, Rc 0.9380, F1 0.9092; UAF1 0.9239
LMVD	LT fusion	Acc 0.8071, Pr 0.9013, Rc 0.9112, F1 0.9048; UAF1 0.9014

These results are contextualized by explicit comparisons with prior systems. On D-Vlog, prior F1 scores listed in the source include EMO-Mamba 0.7566, CAIINET 0.6655, STST 0.7500, Depressformer 0.5900, and XGBoost 0.7700, whereas MMFformer reaches 0.9092 (Haque et al., 8 Aug 2025). On LMVD, the strongest prior method identified in the source is LMTformer with Acc 0.8276 and F1 0.8274, while MMFformer achieves F1 0.9048. The paper characterizes these as relative F1 improvements of 13.92% on D-Vlog and 7.74% on LMVD.

The modality breakdown is also informative. On D-Vlog, audio-only yields WAF1 $$\mathcal{X}_\mathcal{E} = \mathcal{F}_{emb}(\widetilde{\mathcal{X}_v}) = \mathcal{W}_{emb} \widetilde{\mathcal{X}_v},$$8, video-only yields $$\mathcal{X}_\mathcal{E} = \mathcal{F}_{emb}(\widetilde{\mathcal{X}_v}) = \mathcal{W}_{emb} \widetilde{\mathcal{X}_v},$$9, and multimodal variants yield LT $$\mathcal{W}_{emb} \in \mathbb{R}^{\mathcal{C} \times \mathcal{D}}$$0, IT $$\mathcal{W}_{emb} \in \mathbb{R}^{\mathcal{C} \times \mathcal{D}}$$1, and IA $$\mathcal{W}_{emb} \in \mathbb{R}^{\mathcal{C} \times \mathcal{D}}$$2 (Haque et al., 8 Aug 2025). On LMVD, audio-only yields $$\mathcal{W}_{emb} \in \mathbb{R}^{\mathcal{C} \times \mathcal{D}}$$3, video-only yields $$\mathcal{W}_{emb} \in \mathbb{R}^{\mathcal{C} \times \mathcal{D}}$$4, and multimodal variants yield LT $$\mathcal{W}_{emb} \in \mathbb{R}^{\mathcal{C} \times \mathcal{D}}$$5, IT $$\mathcal{W}_{emb} \in \mathbb{R}^{\mathcal{C} \times \mathcal{D}}$$6, and IA $$\mathcal{W}_{emb} \in \mathbb{R}^{\mathcal{C} \times \mathcal{D}}$$7. These numbers support the paper’s claim that audio-only models consistently outperform video-only in these datasets, while fusion further improves performance by exploiting complementary facial information.

The paper also reports fusion-operation ablations using Add, Multiply, Concat, and Tensor Fusion. On D-Vlog, Concat performs best among these simple operations with WAF1 0.8833, Add is second-best with WAF1 0.8860 and the highest WAR 0.9112, and Tensor Fusion performs worst with WAF1 0.7357 (Haque et al., 8 Aug 2025). On LMVD, Add performs best with WAF1 0.8998, Concat is second-best with WAF1 0.8833, and Tensor Fusion underperforms with WAF1 0.8680. The proposed IT, LT, and IA transformer-based fusion designs surpass these simpler fusion baselines on both datasets.

Cross-corpus experiments evaluate generalization under dataset shift. Training on D-Vlog and testing on LMVD, IA attains WAF1 0.8715, IT gives WAA 0.7170, and LT yields a lower WAF1 0.8529 (Haque et al., 8 Aug 2025). Training on LMVD and testing on D-Vlog, IA reaches WAF1 0.8562 with WAR 0.9172, IT reaches WAF1 0.8561 with UAF1 0.8781, and LT is lowest with WAF1 0.8188. This suggests that the preferred fusion stage is dataset-dependent in-domain—IT is best for D-Vlog and LT for LMVD—whereas IA appears strongest under cross-corpus transfer.

6. Robustness, limitations, and broader significance

The training setup reported for MMFformer is relatively specific at the optimizer level and relatively sparse at the architectural level (Haque et al., 8 Aug 2025). Optimization uses Adam with learning rate $$\mathcal{W}_{emb} \in \mathbb{R}^{\mathcal{C} \times \mathcal{D}}$$8, weight decay $$\mathcal{W}_{emb} \in \mathbb{R}^{\mathcal{C} \times \mathcal{D}}$$9, and epsilon $[\mathrm{CLS}]$0; training uses batch size 16, early stopping with patience of 15 epochs, and a maximum of 225 epochs, on two NVIDIA RTX A6000 GPUs with 48 GB each. The video branch uses a pre-trained ViT and the audio branch uses AST-style positional embeddings. However, learning-rate warmup, dropout, gradient clipping, freezing policy, transformer depth, number of heads, and detailed augmentation policies are not reported.

Several limitations are stated directly. Statistical significance testing, confusion matrices, and per-class breakdowns are not reported. Audio sampling and windowing specifics are not described. The paper does not provide detailed hyperparameters for layer count or head count. These omissions constrain architectural comparability and exact reproduction, although public code is reported at https://github.com/rezwanh001/Large-Scale-Multimodal-Depression-Detection (Haque et al., 8 Aug 2025).

The paper also identifies substantive data and deployment limitations. Because the labels derive from public vlogs, volunteered disclosures, annotator judgment, or volunteer labels validated by clinical professionals, the approach inherits potential label noise and demographic bias (Haque et al., 8 Aug 2025). The source explicitly notes that using social media content for mental-health inference requires careful consent, privacy, and responsible deployment, and that the public nature of the datasets does not ensure informed consent for clinical inference. This places MMFformer within a technically successful but ethically sensitive research area.

Within the broader transformer literature, MMFformer is best understood as a domain-specific multimodal fusion architecture rather than a generic transformer family. The remote-sensing MMFormer focuses on HSI+LiDAR fusion and multiscale self-attention (Zhang et al., 2023); the medical mmFormer centers on hybrid modality-specific encoders, a modality-correlated encoder, and missing-modality compensation (Zhang et al., 19 Sep 2025); the environmental MMformer combines inverted variable-token processing, Adaptive Transferable Multi-head Attention, MC Dropout, and MAML-like training for multivariate time-series forecasting (Xin et al., 18 Apr 2025); and FMMformer targets efficient sequence modeling through decomposed near-field and far-field attention (Nguyen et al., 2021). Against that background, MMFformer’s distinctive contribution is the combination of a residual-learning transformer for video, a transformer encoder for audio, and three explicit cross-modal fusion strategies tuned to depression detection in naturalistic social-media data (Haque et al., 8 Aug 2025).

The future directions stated in the source are correspondingly application-specific: evaluation on raw, real-life data, addition of textual and physiological modalities, and the use of LLMs for cross-domain representation and multimodal reasoning (Haque et al., 8 Aug 2025). A plausible implication is that MMFformer’s current audiovisual formulation is not presented as the endpoint of multimodal depression modeling, but as a transformer-based baseline for richer multimodal clinical-inference pipelines.