---
title: 'MMFformer: Multimodal Depression Detection'
url: https://www.emergentmind.com/topics/mmfformer
type: topic
---

# MMFformer: Multimodal Depression Detection

MMFformer is a multimodal fusion transformer network for depression detection from social media video blogs that jointly models facial video and speech audio and learns their interdependencies over time. It is designed to retrieve depressive spatio-temporal high-level patterns from multimodal social media information by combining a transformer network with residual connections for spatial features in video, a transformer encoder for temporal dynamics in audio, and fusion modules that implement late and intermediate fusion strategies to identify intermodal correlations. On two large-scale depression detection datasets, D-Vlog and LMVD, it is reported to improve the F1-Score by 13.92% and 7.74%, respectively, over prior approaches [2508.06701].

## 1. Terminological scope and disambiguation

The label “MMFformer” is not unique across recent arXiv literature, and close variants denote distinct architectures in different domains. In the strict sense used here, MMFformer refers to the depression-detection model introduced in “MMFformer: Multimodal Fusion Transformer Network for Depression Detection” [2508.06701]. A common misconception is to conflate it with similarly named models whose objectives, modalities, and architectural choices differ substantially.

| Name | Domain | Distinguishing description |
|---|---|---|
| MMFformer [2508.06701] | Depression detection | Multimodal social-media video and audio fusion for binary depression classification |
| MMFormer [2303.13101] | Remote sensing | HSI+LiDAR classification with convolutional tokenization and Multi-scale Multi-head Self-Attention |
| mmFormer [2509.16436] | Medical imaging | Multimodal MRI classification with missing-modality compensation for liver fibrosis staging |
| MMformer [2504.14050] | Time-series forecasting | Meta-learning multivariate time-series forecasting with Adaptive Transferable Multi-head Attention |
| FMMformer [2108.02347] | Efficient sequence modeling | Transformer with decomposed near-field and far-field attention |

This distinction is important because only the depression-detection MMFformer operates on audiovisual social-media data and evaluates on D-Vlog and LMVD [2508.06701]. By contrast, the remote-sensing MMFormer uses Hyperspectral Image accompanied by LiDAR for remote sensing image classification and introduces convolutional layers plus a Multi-scale Multi-head Self-Attention module [2303.13101]. The medical-imaging mmFormer retains hybrid modality-specific encoders and a modality-correlated encoder while adding missing-modality compensation with zero-padding, modality masks, and a Delta Function [2509.16436]. The environmental MMformer combines self-attention and meta-learning for multivariate time-series forecasting [2504.14050]. FMMformer is a separate efficient-transformer family inspired by the fast multipole method and decomposes attention into near-field and far-field components [2108.02347].

## 2. Problem formulation and motivation

MMFformer addresses automated depression detection from social media video blogs by treating the task as binary classification, depressed versus non-depressed, from audiovisual content [2508.06701]. The motivating assumption is that depression cues are subtle and distributed across modalities and time: facial micro-expressions, head pose and eye gaze dynamics, prosody, speaking rate, and articulation vary over sequences of seconds to minutes. Single-modality models and static snapshot analysis therefore risk missing complementary and temporally evolving signals.

The model’s design is organized around three motivations stated in the source material. First, facial and vocal signals are complementary; depression may be expressed more in vocal prosody than facial expression for some individuals, and vice versa. Second, cross-modal interactions are informative; incongruent smiling with flat prosody is given as a representative case. Third, the signals are non-stationary, so temporal modeling must accommodate long-range dependencies. Transformers are used for this purpose because they capture temporal structure without relying on manually crafted features [2508.06701].

The datasets reinforce this problem framing. D-Vlog contains 961 vlogs, approximately 160 hours, from 816 individuals on YouTube, and LMVD contains 1,823 vlogs, approximately 214 hours, from 1,475 individuals across Bilibili, TikTok, Sina Weibo, and YouTube [2508.06701]. These are large, in-the-wild corpora rather than tightly controlled laboratory recordings. A plausible implication is that the architecture is intended not merely for multimodal fusion in the abstract, but for fusion under heterogeneous recording conditions, socially natural behavior, and variable temporal extent.

## 3. Modality-specific representation learning

MMFformer uses two transformer branches, one for video and one for audio, with standard multi-head self-attention and pre-norm residual structure [2508.06701]. The video branch is described as a transformer network with residual connections that captures spatial features from videos, while the audio branch is a transformer encoder that models temporal dynamics in speech.

For video, the input tensor is denoted by $\mathcal{X}_v$, temporally downsampled by a 1D convolutional block to a fixed temporal length $L$, and then tokenized with a linear patch embedding. The source gives the sequence of operations as
$$
\widetilde{\mathcal{X}_v} = \mathcal{F}_{d}(\mathcal{X}_v),
$$
$$
\mathcal{X}_\mathcal{E} = \mathcal{F}_{emb}(\widetilde{\mathcal{X}_v}) = \mathcal{W}_{emb} \widetilde{\mathcal{X}_v},
$$
with $\mathcal{W}_{emb} \in \mathbb{R}^{\mathcal{C} \times \mathcal{D}}$. A learnable $[\mathrm{CLS}]$ token is prepended and learnable positional encodings are added:
$$
\mathcal{X}_v^{(0)} = \mathcal{T}_{cls} \oplus \mathcal{X}_\mathcal{E},
$$
$$
\mathcal{X}_v^{(1)} = \mathcal{X}_v^{(0)} + \mathcal{P}.
$$
The paper states that a pre-trained Vision Transformer backbone is used, but the exact number of transformer blocks, heads, and the value of $D$ are not disclosed [2508.06701].

For audio, the input is mapped to a time-frequency representation $\mathcal{X}_{f} \in \mathbb{R}^{\mathcal{F} \times \mathcal{T}}$ and standardized to $\mathcal{X}'_{f} \in \mathbb{R}^{\mathcal{F}' \times \mathcal{T}'}$ through learnable projection using convolution, normalization, and pooling:
$$
\mathcal{X}'_{f} = \mathcal{F}_{lnp}(\mathcal{X}_f; \mathcal{F}', \mathcal{T}', \theta_d).
$$
Overlapping 2D convolutional patches of size $(p_f, p_t)$ and stride $(s_f, s_t)$ produce $\mathcal{M} = h \times w$ patches, which are flattened to tokens. Positional embeddings are adapted from Audio Spectrogram Transformer by bilinearly interpolating a base positional grid to $(h,w)$ and flattening it to $\mathcal{M} \times \mathcal{D}$. The embedded sequence includes both $[\mathrm{CLS}]$ and a distillation token $[\mathrm{DIST}]$, giving total length $\mathcal{M}+2$ [2508.06701].

Both branches use conventional multi-head self-attention:
$$
Q = X W_Q,\quad K = X W_K,\quad V = X W_V,
$$
$$
\mathrm{head}_i = \mathrm{softmax}\!\left(\frac{Q_i K_i^\top}{\sqrt{d_k}}\right) V_i,
$$
$$
\mathrm{MHA}(X) = \mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_h) W_O.
$$
Residual and feed-forward updates are given as
$$
Z = \mathrm{LayerNorm}(X + \mathrm{MHA}(X)),
$$
$$
\mathrm{FFN}(Z) = \mathrm{LayerNorm}(Z + \sigma(Z W_1 + b_1) W_2 + b_2).
$$
In the video branch, the block update is written
$$
\mathcal{Z}^{n} = \mathrm{softmax}\!\left(\frac{\mathcal{Q}^{n}(\mathcal{K}^{n})^\top}{\sqrt{d}}\right)\mathcal{V}^{n},
$$
$$
\mathcal{X}_v^{(n+1)} = \mathcal{F}_{mlp}(\mathcal{Z}^{n} + \mathcal{X}_v^{(n)}) + \left( \mathcal{Z}^{n} + \mathcal{X}_v^{(n)} \right).
$$
In the audio branch, the encoder layer is
$$
\mathcal{U} = \mathrm{LayerNorm}\!\left( \mathcal{X}_{pe} + \mathrm{MHA}(\mathcal{X}_{pe}) \right),
$$
$$
\mathcal{Z} = \mathrm{LayerNorm}\!\left( \mathcal{U} + \mathrm{MLP}(\mathcal{U}) \right).
$$

## 4. Fusion mechanisms and learning objective

The defining property of MMFformer is its use of explicit intermodal fusion rather than simple post hoc concatenation [2508.06701]. Three fusion strategies are implemented: Intermediate Transformer Fusion (IT), Late Transformer Fusion (LT), and Intermediate Attention Fusion (IA).

In Intermediate Transformer Fusion, each branch first applies Conv1D to obtain intermediate representations, after which transformer blocks exchange information via cross-attention before final pooling. In Late Transformer Fusion, cross-modal interaction occurs near the end of each branch; each branch queries its own features and attends to the other modality’s keys and values, and the outputs are pooled and concatenated prior to classification. Intermediate Attention Fusion uses attention-based interaction without direct feature mixing: dot-product attention highlights salient cross-modal features and produces attention vectors that refine each branch before pooling and concatenation [2508.06701].

The general cross-attention form given for video attending to audio is
$$
H_v' = \mathrm{softmax}\!\left(\frac{(H_v W_Q)(H_a W_K)^\top}{\sqrt{d_k}}\right) (H_a W_V).
$$
The paper’s LT and IT instantiation is written, for acoustic queries to visual keys and values, as
$$
\mathcal{O} = \mathrm{softmax}\!\left( \frac{ \mathcal{X}_a^{(o)} \mathcal{W}_q \cdot (\mathcal{X}_v^{(o)} \mathcal{W}_k)^\top }{ \sqrt{d} } \right)\big(\mathcal{X}_v^{(o)} \mathcal{W}_v\big).
$$
For IA, which omits value multiplication, the attention weights are
$$
\mathcal{O} = \mathrm{softmax}\!\left(\frac{\mathcal{X}_a^{(o)} \mathcal{W}_q \cdot (\mathcal{X}_v^{(o)} \mathcal{W}_k)^\top}{\sqrt{d}\right),
$$
and the visual attention vector is aggregated as
$$
v_v = \sum_{i=1}^{N_v} \mathcal{O}[:, i].
$$

The final classifier pools transformer outputs, typically by using the $[\mathrm{CLS}]$ token or global pooling, concatenates the modality representations, and applies a fully connected classifier with softmax activation for binary depression detection. The loss function is standard cross-entropy:
$$
\mathcal{L} = -\sum_{c \in \{\text{dep}, \text{non}\}} y_c \log p_c.
$$
The paper does not report class weighting, focal loss, probability-weighted late fusion, or logit-level linear fusion; it emphasizes learned fusion through cross-attention blocks instead [2508.06701].

## 5. Datasets, protocol, and empirical results

The empirical evaluation uses 10-fold cross-validation on D-Vlog and LMVD, and each experiment reports mean $\pm$ standard deviation over 10 runs [2508.06701]. D-Vlog provides acoustic features via OpenSMILE eGeMAPS and visual face landmarks via Dlib, sampled at 1 Hz. LMVD provides VGGish audio embeddings together with visual features including facial action units, landmarks, eye gaze, and head pose. Reported metrics include Accuracy, Precision, Recall, and F1-score for both weighted averages and unweighted averages. The source defines
$$
P = \frac{TP}{TP + FP}, \qquad R = \frac{TP}{TP + FN},
$$
$$
\mathrm{F1} = \frac{2 \cdot P \cdot R}{P + R}.
$$

| Dataset | Best configuration | Key reported result |
|---|---|---|
| D-Vlog | IT fusion | Acc 0.8108, Pr 0.8924, Rc 0.9380, F1 0.9092; UAF1 0.9239 |
| LMVD | LT fusion | Acc 0.8071, Pr 0.9013, Rc 0.9112, F1 0.9048; UAF1 0.9014 |

These results are contextualized by explicit comparisons with prior systems. On D-Vlog, prior F1 scores listed in the source include EMO-Mamba 0.7566, CAIINET 0.6655, STST 0.7500, Depressformer 0.5900, and XGBoost 0.7700, whereas MMFformer reaches 0.9092 [2508.06701]. On LMVD, the strongest prior method identified in the source is LMTformer with Acc 0.8276 and F1 0.8274, while MMFformer achieves F1 0.9048. The paper characterizes these as relative F1 improvements of 13.92% on D-Vlog and 7.74% on LMVD.

The modality breakdown is also informative. On D-Vlog, audio-only yields WAF1 $0.8955 \pm 0.018$, video-only yields $0.8704 \pm 0.021$, and multimodal variants yield LT $0.9046 \pm 0.012$, IT $0.9092 \pm 0.014$, and IA $0.9071 \pm 0.014$ [2508.06701]. On LMVD, audio-only yields $0.8575 \pm 0.011$, video-only yields $0.8437 \pm 0.043$, and multimodal variants yield LT $0.9048 \pm 0.013$, IT $0.9024 \pm 0.016$, and IA $0.9019 \pm 0.013$. These numbers support the paper’s claim that audio-only models consistently outperform video-only in these datasets, while fusion further improves performance by exploiting complementary facial information.

The paper also reports fusion-operation ablations using Add, Multiply, Concat, and Tensor Fusion. On D-Vlog, Concat performs best among these simple operations with WAF1 0.8833, Add is second-best with WAF1 0.8860 and the highest WAR 0.9112, and Tensor Fusion performs worst with WAF1 0.7357 [2508.06701]. On LMVD, Add performs best with WAF1 0.8998, Concat is second-best with WAF1 0.8833, and Tensor Fusion underperforms with WAF1 0.8680. The proposed IT, LT, and IA transformer-based fusion designs surpass these simpler fusion baselines on both datasets.

Cross-corpus experiments evaluate generalization under dataset shift. Training on D-Vlog and testing on LMVD, IA attains WAF1 0.8715, IT gives WAA 0.7170, and LT yields a lower WAF1 0.8529 [2508.06701]. Training on LMVD and testing on D-Vlog, IA reaches WAF1 0.8562 with WAR 0.9172, IT reaches WAF1 0.8561 with UAF1 0.8781, and LT is lowest with WAF1 0.8188. This suggests that the preferred fusion stage is dataset-dependent in-domain—IT is best for D-Vlog and LT for LMVD—whereas IA appears strongest under cross-corpus transfer.

## 6. Robustness, limitations, and broader significance

The training setup reported for MMFformer is relatively specific at the optimizer level and relatively sparse at the architectural level [2508.06701]. Optimization uses Adam with learning rate $1 \times 10^{-5}$, weight decay $0.1$, and epsilon $1 \times 10^{-8}$; training uses batch size 16, early stopping with patience of 15 epochs, and a maximum of 225 epochs, on two NVIDIA RTX A6000 GPUs with 48 GB each. The video branch uses a pre-trained ViT and the audio branch uses AST-style positional embeddings. However, learning-rate warmup, dropout, gradient clipping, freezing policy, transformer depth, number of heads, and detailed augmentation policies are not reported.

Several limitations are stated directly. Statistical significance testing, confusion matrices, and per-class breakdowns are not reported. Audio sampling and windowing specifics are not described. The paper does not provide detailed hyperparameters for layer count or head count. These omissions constrain architectural comparability and exact reproduction, although public code is reported at `https://github.com/rezwanh001/Large-Scale-Multimodal-Depression-Detection` [2508.06701].

The paper also identifies substantive data and deployment limitations. Because the labels derive from public vlogs, volunteered disclosures, annotator judgment, or volunteer labels validated by clinical professionals, the approach inherits potential label noise and demographic bias [2508.06701]. The source explicitly notes that using social media content for mental-health inference requires careful consent, privacy, and responsible deployment, and that the public nature of the datasets does not ensure informed consent for clinical inference. This places MMFformer within a technically successful but ethically sensitive research area.

Within the broader transformer literature, MMFformer is best understood as a domain-specific multimodal fusion architecture rather than a generic transformer family. The remote-sensing MMFormer focuses on HSI+LiDAR fusion and multiscale self-attention [2303.13101]; the medical mmFormer centers on hybrid modality-specific encoders, a modality-correlated encoder, and missing-modality compensation [2509.16436]; the environmental MMformer combines inverted variable-token processing, Adaptive Transferable Multi-head Attention, MC Dropout, and MAML-like training for multivariate time-series forecasting [2504.14050]; and FMMformer targets efficient sequence modeling through decomposed near-field and far-field attention [2108.02347]. Against that background, MMFformer’s distinctive contribution is the combination of a residual-learning transformer for video, a transformer encoder for audio, and three explicit cross-modal fusion strategies tuned to depression detection in naturalistic social-media data [2508.06701].

The future directions stated in the source are correspondingly application-specific: evaluation on raw, real-life data, addition of textual and physiological modalities, and the use of large language models for cross-domain representation and multimodal reasoning [2508.06701]. A plausible implication is that MMFformer’s current audiovisual formulation is not presented as the endpoint of multimodal depression modeling, but as a transformer-based baseline for richer multimodal clinical-inference pipelines.

Source: https://www.emergentmind.com/topics/mmfformer