---
title: 'MLlm-DR: Multimodal LLM for Depression Recognition'
url: https://www.emergentmind.com/topics/multimodal-large-language-model-mllm-dr
type: topic
---

# MLlm-DR: Multimodal LLM for Depression Recognition

Searching arXiv for the named MLlm-DR paper and a few closely related multimodal LLM papers to ground the article in current research.
MLlm-DR is a multimodal large language model framework for explainable depression recognition from clinical interview videos. It combines a smaller instruction-tuned language model with a lightweight multimodal query module, termed LQ-former, to process audio, visual, and text inputs and to produce both aspect-level depression scores and natural-language evaluation rationales. The system is designed around interview-based assessment protocols such as PHQ-9 and PHQ-8, and it targets a central limitation of earlier multimodal depression models: they typically output scores without explicit clinical explanations, while general-purpose multimodal LLMs perform poorly when applied directly to interview data [2507.05591].

## 1. Clinical formulation and motivation

MLlm-DR models depression recognition at the aspect level rather than as a single opaque prediction. An interview is decomposed into clips \(C_i\), each corresponding to a psychological aspect such as sleep, appetite, or suicidal ideation. Each clip contains audio \(A\), visual input \(V\), and transcribed text \(T\). For each aspect, the model predicts a score \(s_i \in \{0,1,2,3\}\) together with a textual rationale, and the overall depression score is obtained by summing aspect scores:
$$
S = \sum_i s_i
$$
This mirrors the additive scoring procedure used in PHQ-style assessment [2507.05591].

The framework is motivated by three deficiencies in prior systems. First, many earlier multimodal depression recognizers are black-box models that output a score without explaining why a given symptom severity was assigned. Second, although such models fuse speech, facial, and textual features, they do not perform explicit symbolic or logical reasoning over interview content. Third, directly prompting general-purpose LLMs or multimodal LLMs is ineffective in this setting because depression datasets often do not release raw videos and because those models are not trained on clinical interview scoring tasks. A common misconception is that generic multimodal LLM competence transfers automatically to psychiatric assessment; MLlm-DR is explicitly designed against that assumption [2507.05591].

The framework is evaluated on two interview-based benchmarks with different structural properties. CMDC is a semi-structured Chinese Multimodal Depression Corpus with PHQ-9 labels and clear mapping from questions to aspects. E-DAIC-WOZ is based on PHQ-8 and consists of open-ended interviews without explicit aspect-question alignment. This difference is methodologically important because it changes how multimodal evidence can be localized and aggregated [2507.05591].

## 2. Core architecture

MLlm-DR consists of two main modules: a smaller LLM based on LLaMA-3-8B and an LQ-former that converts non-textual features into LLM-compatible embeddings. The smaller LLM is responsible for generating the aspect score and the accompanying evaluation rationale. The LQ-former is responsible for extracting depression-related information from speech and visual streams and injecting it into the language model [2507.05591].

| Component | Input | Function |
|---|---|---|
| Smaller LLM | Transcript, prompts, multimodal tokens | Generates score and rationale |
| LQ-A | HuBERT speech features | Maps audio to LLM embedding space |
| LQ-V | Visual features from OpenFace 2.0 or ResNet-50 | Maps visual input to LLM embedding space |
| MAA head | Final LLM hidden states | Produces regression score |

The LQ-former is a Transformer decoder with learnable queries \(Q \in \mathbb{R}^{n \times D}\), cross-attending to non-textual feature sequences \(F \in \mathbb{R}^{m \times D}\). Its core computation is
$$
\begin{aligned}
H &= \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V, \\
O &= \text{Projection}(H),
\end{aligned}
$$
where \(K = V \in \mathbb{R}^{m \times D}\) are modality features. Two separate instances are used: LQ-A for audio and LQ-V for visual features. Their outputs are inserted into the LLM input at special placeholders such as \<AudioHere\> and \<VideoHere\> [2507.05591].

The multimodal processing path is therefore adapter-like rather than end-to-end raw-signal ingestion. Audio is first encoded by HuBERT into 768-dimensional representations, and visual features are taken from dataset-provided OpenFace 2.0 or ResNet-50 descriptors. This design reflects privacy constraints: the model operates on pre-extracted interview features rather than raw video streams. A plausible implication is that MLlm-DR belongs to a class of clinical multimodal LLMs in which representational bottlenecks are imposed by data governance rather than by model capacity alone.

The architecture also includes a Multi-Head Attention Aggregation regression head. Let \(X \in \mathbb{R}^{L \times D}\) denote the final hidden states of the LLM. Splitting \(X\) across \(h\) heads yields \(x' \in \mathbb{R}^{L \times h \times d}\), after which the model computes
$$
\begin{aligned}
\alpha_i &= \text{softmax}\left(W_i x_i'^\top \right), \\
S_{\text{reg}} &= \text{FC}\left( \sum_{i=1}^{h} \left( \alpha_i \cdot x_i' \right) \right).
\end{aligned}
$$
This gives a continuous regression estimate complementary to the discrete score generated in text form [2507.05591].

## 3. Multimodal inputs, supervision, and training procedure

The input modalities are heterogeneous in both source and supervision. Speech is encoded with HuBERT, visual information comes from pre-extracted facial or deep features, and text comes from interview transcripts tokenized with the LLaMA-3 tokenizer. Because visual raw data are unavailable in these datasets, MLlm-DR is a feature-conditioned MLLM rather than a raw-video MLLM in the usual vision-language sense [2507.05591].

A central component of the method is the construction of a robust training dataset for rationale supervision. For each aspect-level dialogue segment with known label \(s_i \in \{0,1,2,3\}\), GPT-4o is prompted with the aspect name, the ground-truth score, and the dialogue content, and is required to produce output in a strict format:
- `Evaluation Result:`
- `Evaluation Reason:`

Because the true label is provided in the prompt, GPT-4o is used primarily to generate rationale text consistent with the gold score rather than to infer the score itself. This yields supervised pairs of interview content and clinically formatted explanations, which are then used to fine-tune the smaller LLM [2507.05591].

Training proceeds in two stages. In the first stage, the LLM is frozen and the LQ-former is trained so that non-textual embeddings become meaningful to the frozen decoder. In the second stage, the LQ-former is frozen and the LLM is fine-tuned with a joint objective:
$$
\mathcal{L}_{\text{joint}} = \mathcal{L}_{\text{lm}} + \mathcal{L}_{\text{mse}}.
$$
Here \(\mathcal{L}_{\text{lm}}\) supervises generation of both the score and the rationale, while \(\mathcal{L}_{\text{mse}}\) supervises the regression head against numeric depression scores [2507.05591].

Fine-tuning is parameter-efficient. The model uses LoRA on the query and value projection matrices of the LLaMA-3-8B attention layers, with \(r=16\), \(\alpha=32\), and dropout \(0.1\). Training uses a learning rate of \(1 \times 10^{-5}\) for 10 epochs on 2×A800 GPUs. This design choice is consistent with a broader pattern in multimodal adaptation: retain a strong pretrained language backbone and introduce a lightweight modality bridge rather than retraining the full model from scratch.

The datasets are structurally distinct. CMDC contains 78 subjects, with 12 interview questions manually aligned to 9 PHQ-9 aspects, producing \(78 \times 9 = 702\) aspect-level samples. E-DAIC-WOZ contains train/valid/test splits of 163/56/56 subjects and uses the full interview as input for each PHQ-8 aspect, producing \(163 \times 8 = 1304\) aspect-level samples for training. This difference matters because the LQ-former is more effective when it can summarize short, aspect-specific clips than when it must compress a long, open-ended interview [2507.05591].

## 4. Explainability and rationale generation

Explainability in MLlm-DR is implemented as structured natural-language generation rather than as post hoc saliency or attention visualization. The model is trained to emit both a numeric aspect score and a concise evaluation reason in a fixed template. The rationale is expected to refer to dialogue evidence and to align with PHQ item semantics, while speech and visual evidence enter implicitly through the multimodal embeddings inserted into the LLM context [2507.05591].

This rationale mechanism addresses a frequent criticism of earlier multimodal depression systems: even when they achieve strong predictive performance, their interpretability is weak relative to clinical workflows. MLlm-DR instead produces textual justifications that can be inspected directly. This does not imply causal transparency in a strict mechanistic sense, but it does provide an audit trail closer to psychiatric assessment practice.

Human evaluation was conducted on 100 test cases per dataset. Experts assessed output format, agreement between model predictions and expert scores, and the quality of rationales on a three-point scale. On CMDC, only 1.23% of outputs were misformatted, 87% of predictions were consistent with expert scores, and rationale ratings were 77% “fully agree,” 14% “reasonable,” and 9% “disagree.” On E-DAIC-WOZ, 3.12% of outputs were misformatted, 73% were consistent with expert scores, and rationale ratings were 64%, 19%, and 17%, respectively [2507.05591].

These results clarify another common misconception: explainability here is not merely a side effect of using an LLM. The rationales are produced because the model is explicitly trained on GPT-4o-generated, label-consistent explanations and because the output format is part of the supervision target. A plausible implication is that explanation quality depends at least as much on the curation of rationale data as on the underlying language model.

## 5. Empirical performance and ablation evidence

MLlm-DR is evaluated as both a classification system for depression screening and a regression system for PHQ score prediction. On CMDC, the text-only LLaMA-3-8B baseline achieved CCC \(0.37\), RMSE \(10.19\), MAE \(9.36\), Precision \(0.35\), Recall \(1.0\), and F1 \(0.52\). A text-only LoRA fine-tuned variant improved to CCC \(0.80\), RMSE \(4.61\), MAE \(4.08\), Precision \(1.0\), Recall \(0.83\), and F1 \(0.91\). The full multimodal MLlm-DR achieved CCC \(0.91\), RMSE \(3.10\), MAE \(2.61\), Precision \(1.0\), Recall \(1.0\), and F1 \(1.0\) [2507.05591].

On E-DAIC-WOZ, the text-only LLaMA-3-8B baseline achieved CCC \(0.54\), RMSE \(5.55\), MAE \(4.23\), Precision \(0.56\), Recall \(0.85\), and F1 \(0.68\). The text-only LoRA version improved to CCC \(0.69\), RMSE \(5.04\), MAE \(4.04\), Precision \(0.74\), Recall \(0.80\), and F1 \(0.77\). The full multimodal MLlm-DR achieved CCC \(0.72\), RMSE \(4.59\), MAE \(3.41\), Precision \(0.77\), Recall \(0.81\), and F1 \(0.79\), which the paper reports as state-of-the-art on the dataset [2507.05591].

Ablation results identify two decisive ingredients: the LQ-former and the joint objective. On CMDC, removing the joint loss reduced performance from CCC \(0.91\), RMSE \(3.10\), MAE \(2.61\) to CCC \(0.89\), RMSE \(3.58\), MAE \(3.08\). Removing the LQ-former reduced performance further to CCC \(0.87\), RMSE \(4.18\), MAE \(3.61\). On E-DAIC-WOZ, the full model achieved CCC \(0.72\), F1 \(0.79\), RMSE \(4.59\), MAE \(3.41\), whereas removing the LQ-former reduced this to CCC \(0.65\), F1 \(0.71\), RMSE \(4.86\), MAE \(3.78\) [2507.05591].

The modality-specific ablations are also instructive. On CMDC, removing audio LQ-A yielded CCC \(0.87\), RMSE \(3.99\), MAE \(3.55\), whereas removing visual LQ-V yielded CCC \(0.90\), RMSE \(3.52\), MAE \(2.90\). On E-DAIC-WOZ, removing LQ-A produced CCC \(0.66\), RMSE \(4.80\), MAE \(3.71\), while removing LQ-V produced CCC \(0.70\), RMSE \(4.55\), MAE \(3.56\). This suggests that audio is particularly informative on short, aspect-aligned clips, while the contribution of visual cues becomes relatively more competitive on long, open-ended interviews.

The isolated LQ-former analysis reinforces this interpretation. With the LLM frozen and no text input, the combined LQ-former on CMDC achieved CCC \(0.85\), Precision \(1.0\), Recall \(1.0\), F1 \(1.0\), RMSE \(3.79\), and MAE \(3.06\). On E-DAIC-WOZ, the same setting achieved CCC only \(0.22\), RMSE \(7.24\), and MAE \(6.38\). The paper attributes this gap to the difficulty of compressing long interviews with fixed learnable queries, which is a substantive limitation rather than a mere optimization detail [2507.05591].

## 6. Broader context, misconceptions, and limitations

Within the broader multimodal LLM literature, MLlm-DR exemplifies a design pattern in which a pretrained language model is preserved and non-text modalities are introduced through lightweight interfaces rather than through wholesale architectural replacement. This strategy is shared, in different forms, by discrete multimodal language models that expand a text vocabulary to include speech or image tokens [2406.06582], by visual-expert systems that add modality-specific attention parameters while preserving the text pathway [2406.18193], by parameter-efficient multimodal adaptation via low-rank modules [2406.09617], and by compact domain-adapted MLLMs for autonomous driving [2501.05081]. Distillation-oriented frameworks such as LLaVA-KD likewise emphasize transferring multimodal competence into smaller backbones without altering the core student architecture [2410.16236].

MLlm-DR differs from these systems in task and data regime. Its multimodal evidence is not raw image-text data but pre-extracted speech and visual interview features under privacy constraints. Its output is not captioning, VQA, or general dialogue, but PHQ-style scores and clinically shaped rationales. This makes it a domain-specific MLLM rather than a general-purpose one, even though it inherits the backbone-and-adapter logic common to current multimodal LLM design.

Several objective limitations remain. The rationale supervision dataset is constructed primarily from text, which may bias the model when nonverbal evidence conflicts with verbal reports. The underlying datasets are relatively small, limiting confidence about generalization across populations, languages, and interview protocols. Fine-grained labels for speech and visual data are absent, so the model cannot localize which exact frames or acoustic events support a rationale. The paper therefore does not claim deployment readiness; it explicitly identifies the need for larger multimodal depression datasets, finer-grained speech and visual supervision, and broader clinical validation [2507.05591].

A final misconception is that rationales produced by a clinical MLLM necessarily reflect grounded multimodal evidence in a fully transparent way. MLlm-DR provides human-readable justifications and strong empirical alignment with expert judgments, but its explanations are still generated by a learned decoder operating over latent multimodal summaries. The significance of the framework lies less in solving interpretability outright than in showing that domain-adapted multimodal LLMs can couple competitive depression recognition with structured, clinically reviewable explanatory text.

Source: https://www.emergentmind.com/topics/multimodal-large-language-model-mllm-dr