---
title: 'MPDD: Age-Specific Multimodal Depression Detection'
url: https://www.emergentmind.com/topics/multimodal-personality-aware-depression-detection-challenge
type: topic
---

# MPDD: Age-Specific Multimodal Depression Detection

Searching arXiv for the challenge paper and closely related follow-up work.
Depression detection research has increasingly emphasized multimodal behavioral signals, but many benchmark resources have centered on young adults and have treated prediction as a direct mapping from audiovisual observations to depression labels. The Multimodal Personality-Aware Depression Detection Challenge (MPDD Challenge) was introduced to address this limitation by explicitly combining multimodal data with individual-difference factors, especially personality traits, and by separating evaluation into elderly and young cohorts [2505.10034]. The challenge defines a benchmark for personalized depression detection under age-specific conditions, provides a baseline fusion model, and has subsequently served as a testbed for methods that refine multimodal fusion, personality-guided representation learning, and cross-context generalization [2505.10034].

## 1. Origin, scope, and problem formulation

The MPDD Challenge was presented as “The First MPDD Challenge: Multimodal Personality-aware Depression Detection” [2505.10034]. Its stated motivation is that depression affects diverse age groups, while existing datasets and detection methods primarily focus on young adults and often neglect broader age variation and individual differences. The challenge therefore broadens automatic depression detection beyond young adults by introducing both elderly and young cohorts and by incorporating individual-difference factors—especially personality traits—to move beyond “direct” multimodal-to-depression mappings and toward personalized detection [2505.10034].

The benchmark is organized into two tracks. Track 1 uses the MPDD-Elderly dataset for detecting depression in older adults, and Track 2 uses the MPDD-Young dataset for detecting depression in younger participants [2505.10034]. This age-specific partition is central to the challenge design: the elderly and young subsets differ not only in participant demographics but also in recording protocols, auxiliary annotations, and depression labeling schemes.

A defining feature of the challenge is its explicit integration of personality-aware information into multimodal prediction. In the challenge formulation, audio and video signals are not treated as sufficient on their own; instead, they are complemented by structured individual information such as Big Five personality traits, demographic factors, and, in the elderly track, contextual variables including disease category, financial stress, and number of co-habiting family members [2505.10034]. This suggests a shift from purely behavior-driven recognition toward individualized inference, where the same observable cue may have different predictive significance depending on the subject profile.

## 2. Datasets, cohorts, and annotation protocols

The MPDD Challenge defines two age-specific datasets with different acquisition settings and annotation regimes [2505.10034].

For MPDD-Elderly, the challenge reports 94 elderly volunteers with mean age approximately 68, recruited from a hospital setting in China, many with chronic conditions [2505.10034]. The data comprise 564 segmented samples generated with 1 s and 5 s sliding windows, split into 337 training and 227 test clips with no subject overlap [2505.10034]. Annotations include depression labels from PHQ-9 in a five-class setting and HAMD-24 in binary and ternary settings, personality from BigFive-10 using one-hot top trait encoding, and additional labels for disease category, financial stress, and number of co-habiting family members [2505.10034]. The recording protocol consists of semi-structured interviews guided by HAMD/BDI questions with synchronized video and audio capture [2505.10034].

For MPDD-Young, the challenge reports 88 young adults with mean age approximately 21, described as non-clinical and recruited from a university [2505.10034]. This subset contains 528 samples, with 264 training and 264 test [2505.10034]. Depression annotations are based on PHQ-9 in binary and ternary forms, personality is again represented using BigFive-10 one-hot encoding, and demographic variables include age, gender, and native place [2505.10034]. The protocol follows a more controlled sequence: questionnaire, self-introduction, and two controlled text-reading tasks, with simultaneous video and audio capture [2505.10034].

A later challenge-participant study reports the training-set composition somewhat differently, describing MPDD-Elderly as 337 semi-structured interview recordings from 89 elderly participants with mean age \(62.8 \pm 11.0\), and MPDD-Young as 264 recordings from 88 university-age speakers with mean age \(20.0 \pm 2.2\) [2508.20805]. That work also lists explicit class distributions for the training split, including severe imbalance in the elderly quinary task, where the “very severe” class accounts for 0.9% [2508.20805]. The discrepancy in elderly participant count between the challenge overview and the participant study reflects differences in reported corpus granularity or subset accounting; the challenge description remains the primary definition of the official benchmark [2505.10034].

## 3. Modalities and personality-aware representations

The challenge provides multimodal features spanning audio, video, and text-derived personalized descriptions [2505.10034]. Audio waveforms are represented through MFCC with 13 dimensions, OpenSMILE descriptors of approximately 6 k dimensions, and Wav2Vec embeddings with 768 dimensions [2505.10034]. Video frames are represented through DenseNet-121 features with 1024 dimensions, OpenFace landmarks and AU intensities with \(68+17\) dimensions, and ResNet-50 embeddings with 2048 dimensions [2505.10034]. Textual prompts are generated by ChatGLM3 from each subject’s Big Five profile, stress information, and related attributes, then embedded by RoBERTa-large into a 1024-dimensional vector [2505.10034].

The individual-difference factors differ by track. Big Five personality traits are measured by BigFive-10, with scores for openness, conscientiousness, extraversion, agreeableness, and neuroticism, encoded as one-hot top trait(s) in the challenge release [2505.10034]. Depression severity is labeled using PHQ-9 and, for the elderly track, HAMD-24 [2505.10034]. Elderly-only auxiliary variables include disease category, financial stress, and family-member count, whereas the young track includes age, gender, and native place [2505.10034].

Subsequent work on the challenge retains the same overall principle but instantiates it differently. A participant system describes “personalized embedding” as raw trait descriptions tokenized and fed into pre-trained RoBERTa-large to produce a 1,024-dimensional personality embedding per participant [2508.20805]. Another follow-up method for the elderly track, “Personality-Enhanced Multimodal Depression Detection in the Elderly,” also uses ChatGLM3-generated text with RoBERTa-large to obtain a trait embedding \(p \in \mathbb{R}^d\), which is later fused with multimodal features via an explicit interaction module [2510.08004]. A later MPDD-Young method, P\(^3\)HF, instead prompts GPT-4 with demographic attributes and Big Five scores to generate a 1–2 sentence contextual description, encodes that description with BERT, and obtains a 768-dimensional “personality embedding” \(P_i\) [2511.12460].

These variants share the same conceptual premise: discrete personal attributes are first converted into a richer textual representation and then embedded with a pretrained language model. A plausible implication is that the challenge helped normalize an LLM-mediated pathway from structured subject metadata to dense conditioning vectors for multimodal affective modeling.

## 4. Baseline architecture and official evaluation

The official challenge baseline fuses audio and video modalities with individual-difference information in a staged pipeline [2505.10034]. The audio encoder is an LSTM over frame-level audio features, producing a final hidden vector \(a\). The video encoder is an LSTM over frame-level visual features, producing a final hidden vector \(v\). These are concatenated and passed to a small Transformer encoder to form a multimodal representation \(m\). A personalized embedding \(p \in \mathbb{R}^{1024}\), obtained from RoBERTa over the ChatGLM3-generated description, is concatenated with \(m\) to produce \(h\), and a two-layer MLP with softmax performs depression classification [2505.10034].

The challenge summary gives the multimodal fusion step as
\[
h_0 = \mathrm{concat}(a, v)
\]
\[
m = \mathrm{Transformer}(h_0)
\]
\[
h = \sigma(W_m m + W_p p + b),
\]
where \(W_m\) and \(W_p\) are learned weights and \(\sigma\) is ReLU [2505.10034]. An equivalent linear-fusion form is also stated:
\[
h = \sigma(W_a a + W_v v + W_p p + b).
\]
For binary, ternary, and quinary tasks, the loss is cross-entropy:
\[
L = -\sum_{i=1}^N \sum_{c=1}^C y_{i,c} \log \hat y_{i,c}.
\]
The challenge defines Accuracy as \(\mathrm{Acc}=(\mathrm{correct\ predictions})/(\mathrm{total\ samples})\) and \(F_1 = 2 \cdot \mathrm{Precision} \cdot \mathrm{Recall}/(\mathrm{Precision}+\mathrm{Recall})\), with weighted and unweighted versions used to account for class imbalance [2505.10034]. The task-level score is the average of weighted and unweighted variants, and the track-level score is the mean of its task-level scores [2505.10034].

The baseline results reported for 1 s windows with personalized features (“PF”) are summarized below.

| Track/task | Best reported baseline configuration | Reported result |
|---|---|---|
| MPDD-Elderly binary | MFCC + OpenFace + PF | \(W\_Acc=85.40\%\), \(W\_{F1}=85.71\%\) |
| MPDD-Elderly ternary | OpenSMILE + ResNet + PF | \(W\_Acc=55.49\%\), \(W\_{F1}=56.48\%\) |
| MPDD-Elderly quinary | OpenSMILE + DenseNet + PF | \(W\_Acc=45.79\%\), \(W\_{F1}=66.26\%\) |
| MPDD-Young binary | Wav2Vec + OpenFace + PF | \(W\_Acc=63.64\%\), \(W\_{F1}=59.96\%\) |
| MPDD-Young ternary | MFCC + DenseNet + PF | \(W\_Acc=49.66\%\), \(W\_{F1}=51.86\%\) |

The challenge also reports ablations showing that PF improves elderly binary \(W\_Acc\) by \(+16.03\) points, from \(69.37\) to \(85.40\), and improves young binary \(W\_Acc\) by \(+7.58\) points, with consistent gains of 2–13 points across tasks [2505.10034]. This is one of the clearest empirical claims attached to the benchmark: personality-aware embeddings materially improve prediction under the official baseline.

## 5. Follow-up methods on the benchmark

The MPDD Challenge rapidly became a target for method development, and the follow-up literature illustrates several distinct modeling strategies.

One challenge-participant study, “Exploring Machine Learning and Language Models for Multimodal Depression Detection,” compares XGBoost, a transformer-based architecture, and an LLM-based model on audio, video, and text features [2508.20805]. Its XGBoost configuration uses PCA to project each modality to \(k=50\) principal components and fuses them as \(Z=[Z_a \| Z_v]\), with objective “multi:softprob” and class weighting for imbalance [2508.20805]. The transformer projects modality features into a shared \(d=128\) latent space, employs two modality-specific transformer encoders with 2 layers and 4 heads, performs time-pooling with learned attention weights, and concatenates \(\tilde x_a\), \(\tilde x_v\), and \(z_t\) before two fully connected layers [2508.20805]. The LLM model uses LLaMA-2-7B extended via Emotion-LLaMA, modality projections into a 4096-dimensional shared space, and a structured multiple-choice prompt, with a two-stage fine-tuning procedure involving frozen backbone training followed by LoRA adaptation [2508.20805]. On the reported dev split, XGBoost reaches \(W_{F1}=94.29\%\) and \(U_{F1}=91.90\%\) for MPDD-Elderly 5 s binary, while the transformer achieves \(W_{F1}=95.83\%\) and \(U_{F1}=95.83\%\) on MPDD-Young 1 s binary; the LLM underperforms both [2508.20805]. That study concludes that scale alone does not guarantee better depression detection and that transformer-based fusion is the most consistent top performer across tasks [2508.20805].

A second line of work focuses specifically on the elderly track. “Personality-Enhanced Multimodal Depression Detection in the Elderly” introduces a multi-feature fusion approach based on a co-attention mechanism for audio, combining LLDs, MFCCs, and Wav2Vec, while visual features combine OpenFace, ResNet, and DenseNet representations [2510.08004]. Its multimodal stream produces per-utterance embeddings through LSTM encoders and Attentive Static Pooling, followed by a Transformer fusion block:
\[
f_s^* = F_s^*\bigl(\mathrm{Concat}(F_s^v(u_n^v), F_s^a(u_n^a))\bigr).
\]
The personality component is modeled by a Personality Traits & Multimodal Feature Interaction Module (PTMFIM), which uses binary correlation attention, triple interaction attention, and a gating regulator to combine a RoBERTa personality embedding with the fused multimodal representation [2510.08004]. On 1 s windows, the reported results improve over a baseline from 85.01 to 89.18 in binary accuracy and from 82.42 to 88.53 in binary \(F_1\), with corresponding gains for ternary and quinary tasks [2510.08004]. The ablation with the largest degradation is removal of PTMFIM, where binary accuracy drops from 89.18 to 86.29 and \(F_1\) from 88.53 to 85.93 [2510.08004]. This strengthens the challenge’s original claim that personality-aware fusion is not merely auxiliary but structurally important for performance.

A third line extends the benchmark toward graph-structured and domain-generalized modeling. “Personality-guided Public-Private Domain Disentangled Hypergraph-Former Network for Multimodal Depression Detection” proposes P\(^3\)HF for MPDD-Young [2511.12460]. The method first derives personality embeddings from GPT-4-generated text and BERT, then uses a gating mechanism to modulate audio and video streams:
\[
W_{\mathrm{gate}} = \sigma(W_p \tilde P + b_p),
\]
\[
A_k|P = \tilde A_k + \tilde A_k \odot W_{\mathrm{gate}}, \quad
V_k|P = \tilde V_k + \tilde V_k \odot W_{\mathrm{gate}}.
\]
It then builds a Hypergraph-Former over audio and video segments with positional encoding and multi-head self-attention, followed by public-private domain disentanglement with adversarial loss and HSIC regularization [2511.12460]. On MPDD-Young, P\(^3\)HF reports 82.17% accuracy and 81.39% weighted \(F_1\) for binary classification, and 76.29% accuracy and 74.61% weighted \(F_1\) for ternary classification, with improvements over DepMamba of \(+9.6\%/+9.9\%\) in binary and \(+8.4\%/+8.4\%\) in ternary [2511.12460]. Ablation shows that removing personality guidance reduces binary accuracy by approximately 5.5% [2511.12460].

Together, these studies indicate that the MPDD benchmark supports a broad range of model families—from gradient-boosted trees to transformer fusion modules and hypergraph-based architectures—while repeatedly reproducing the utility of personality-aware conditioning.

## 6. Technical themes, challenges, and research implications

Several technical themes recur across the challenge and follow-up systems.

First, multimodal fusion remains a central design variable. The official baseline uses simple concatenation with linear projection and a small Transformer encoder [2505.10034]. Later work explores more structured alternatives: co-attention among heterogeneous audio streams, explicit multimodal-personality interaction modules, and hypergraph reasoning for high-order cross-modal temporal relations [2510.08004; 2511.12460]. This suggests that the benchmark is not only about adding personality metadata, but also about testing how personality should interact with temporally structured audiovisual evidence.

Second, personality encoding is consistently treated as a learned dense representation derived from textualization of structured traits. In the challenge baseline, ChatGLM3 converts subject attributes into textual prompts that RoBERTa-large embeds to 1024 dimensions [2505.10034]. Later methods preserve the same general mechanism with different language models, including GPT-4 and BERT [2511.12460]. A plausible implication is that natural-language reformulation acts as an intermediate semantic layer, making discrete subject descriptors more compatible with pretrained text encoders and attention-based fusion modules.

Third, the benchmark foregrounds clinically and methodologically relevant heterogeneity. The elderly and young tracks differ in acquisition setting—hospital versus university/laboratory-like controlled capture—and this variability is explicitly identified as a source of domain shift [2505.10034]. The challenge summary also notes limited dataset size and class imbalance, especially in quinary classes [2505.10034]. The participant comparison study reaches the same conclusion, emphasizing the risk of overfitting and the difficulty of skewed quinary classification, particularly where the “very severe” class is only 0.9% of the elderly training set [2508.20805].

Fourth, the benchmark makes room for event-structured reasoning in the young cohort. P\(^3\)HF exploits the fact that each young subject has three recorded “events”—self-introduction and two reading tasks—and frames generalization partly as a problem of disentangling event-specific from event-invariant representations [2511.12460]. This is not part of the official baseline, but it demonstrates that the MPDD design can support research on behavioral-context disentanglement in addition to depression classification.

## 7. Limitations, open questions, and prospective directions

The challenge paper identifies three principal limitations: limited dataset size, class imbalance, and the possibility that hand-crafted fusion by concatenation underutilizes complex cross-modal and personality interactions [2505.10034]. It also notes variability in recording conditions between hospital and lab settings as a source of domain shift [2505.10034]. These points are echoed by follow-up work, which reports no formal statistical significance tests in one comparative study and highlights possible overfitting on small datasets [2508.20805].

Several future directions are explicitly proposed in the challenge description. These include attention-based cross-modal transformers or gated multimodal blocks for dynamic weighting of audio, visual, and personality signals; multi-task learning to jointly predict depression severity and auxiliary labels such as personality dimension scores; semi-supervised learning and data augmentation; interpretability methods to trace which modality or individual factor drives a decision; and expansion to text transcripts, physiological signals such as HRV and EEG, and contextual metadata [2505.10034]. Follow-up papers refine these possibilities. The elderly-track PTMFIM study notes that end-to-end fine-tuning of audio and video backbones was not explored and that personality embedding derived from a generative LLM may warrant sensitivity and noise analysis [2510.08004]. The comparative study proposes noise reduction, speaker diarization, cross-modal attention layers, domain-specific pre-training, and tighter integration of personality embeddings rather than simple concatenation [2508.20805].

A common misconception in multimodal mental-health modeling is that adding more modalities or a larger backbone will automatically improve detection. The MPDD-related evidence does not support that simplification. The participant comparison study reports that an LLM with 6.84 B parameters underperforms both XGBoost and a 1.06 M-parameter transformer on the examined settings [2508.20805]. Conversely, the challenge baseline and later systems repeatedly show that carefully structured personality-aware fusion can yield substantial gains even with relatively simple fusion components [2505.10034; 2510.08004].

The MPDD Challenge therefore occupies a specific place in depression detection research: it is a benchmark built around the hypothesis that multimodal behavioral signals should be interpreted in light of age-specific context and individual differences. Its continuing significance lies less in a single baseline score than in the methodological agenda it set—personalized multimodal inference, age-aware benchmarking, and explicit modeling of personality-conditioned depression manifestations [2505.10034].

Source: https://www.emergentmind.com/topics/multimodal-personality-aware-depression-detection-challenge