MLlm-DR: Multimodal LLM for Depression Recognition

• MLlm-DR is a multimodal framework that integrates audio, visual, and textual inputs to produce clinically interpretable depression scores and natural-language rationales based on PHQ protocols.
• It combines a smaller instruction-tuned LLM with a lightweight LQ-former to convert non-textual features into embeddings, improving explainability over traditional black-box models.
• Empirical results demonstrate state-of-the-art performance through a dual-stage training process and joint supervision that enhances both classification and regression metrics.

Searching arXiv for the named MLlm-DR paper and a few closely related multimodal LLM papers to ground the article in current research. MLlm-DR is a multimodal LLM framework for explainable depression recognition from clinical interview videos. It combines a smaller instruction-tuned LLM with a lightweight multimodal query module, termed LQ-former, to process audio, visual, and text inputs and to produce both aspect-level depression scores and natural-language evaluation rationales. The system is designed around interview-based assessment protocols such as PHQ-9 and PHQ-8, and it targets a central limitation of earlier multimodal depression models: they typically output scores without explicit clinical explanations, while general-purpose multimodal LLMs perform poorly when applied directly to interview data (Zhang et al., 8 Jul 2025).

1. Clinical formulation and motivation

MLlm-DR models depression recognition at the aspect level rather than as a single opaque prediction. An interview is decomposed into clips $C_i$, each corresponding to a psychological aspect such as sleep, appetite, or suicidal ideation. Each clip contains audio $A$, visual input $V$, and transcribed text $T$. For each aspect, the model predicts a score $s_i \in \{0,1,2,3\}$ together with a textual rationale, and the overall depression score is obtained by summing aspect scores:

$S = \sum_i s_i$

This mirrors the additive scoring procedure used in PHQ-style assessment (Zhang et al., 8 Jul 2025).

The framework is motivated by three deficiencies in prior systems. First, many earlier multimodal depression recognizers are black-box models that output a score without explaining why a given symptom severity was assigned. Second, although such models fuse speech, facial, and textual features, they do not perform explicit symbolic or logical reasoning over interview content. Third, directly prompting general-purpose LLMs or multimodal LLMs is ineffective in this setting because depression datasets often do not release raw videos and because those models are not trained on clinical interview scoring tasks. A common misconception is that generic multimodal LLM competence transfers automatically to psychiatric assessment; MLlm-DR is explicitly designed against that assumption (Zhang et al., 8 Jul 2025).

The framework is evaluated on two interview-based benchmarks with different structural properties. CMDC is a semi-structured Chinese Multimodal Depression Corpus with PHQ-9 labels and clear mapping from questions to aspects. E-DAIC-WOZ is based on PHQ-8 and consists of open-ended interviews without explicit aspect-question alignment. This difference is methodologically important because it changes how multimodal evidence can be localized and aggregated (Zhang et al., 8 Jul 2025).

2. Core architecture

MLlm-DR consists of two main modules: a smaller LLM based on LLaMA-3-8B and an LQ-former that converts non-textual features into LLM-compatible embeddings. The smaller LLM is responsible for generating the aspect score and the accompanying evaluation rationale. The LQ-former is responsible for extracting depression-related information from speech and visual streams and injecting it into the LLM (Zhang et al., 8 Jul 2025).

Component	Input	Function
Smaller LLM	Transcript, prompts, multimodal tokens	Generates score and rationale
LQ-A	HuBERT speech features	Maps audio to LLM embedding space
LQ-V	Visual features from OpenFace 2.0 or ResNet-50	Maps visual input to LLM embedding space
MAA head	Final LLM hidden states	Produces regression score

The LQ-former is a Transformer decoder with learnable queries $Q \in \mathbb{R}^{n \times D}$, cross-attending to non-textual feature sequences $F \in \mathbb{R}^{m \times D}$. Its core computation is

$$\begin{aligned} H &= \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V, \ O &= \text{Projection}(H), \end{aligned}$$

where $K = V \in \mathbb{R}^{m \times D}$ are modality features. Two separate instances are used: LQ-A for audio and LQ-V for visual features. Their outputs are inserted into the LLM input at special placeholders such as <AudioHere> and <VideoHere> (Zhang et al., 8 Jul 2025).

The multimodal processing path is therefore adapter-like rather than end-to-end raw-signal ingestion. Audio is first encoded by HuBERT into 768-dimensional representations, and visual features are taken from dataset-provided OpenFace 2.0 or ResNet-50 descriptors. This design reflects privacy constraints: the model operates on pre-extracted interview features rather than raw video streams. A plausible implication is that MLlm-DR belongs to a class of clinical multimodal LLMs in which representational bottlenecks are imposed by data governance rather than by model capacity alone.

The architecture also includes a Multi-Head Attention Aggregation regression head. Let $A$0 denote the final hidden states of the LLM. Splitting $A$1 across $A$2 heads yields $A$3, after which the model computes

$A$4

This gives a continuous regression estimate complementary to the discrete score generated in text form (Zhang et al., 8 Jul 2025).

3. Multimodal inputs, supervision, and training procedure

The input modalities are heterogeneous in both source and supervision. Speech is encoded with HuBERT, visual information comes from pre-extracted facial or deep features, and text comes from interview transcripts tokenized with the LLaMA-3 tokenizer. Because visual raw data are unavailable in these datasets, MLlm-DR is a feature-conditioned MLLM rather than a raw-video MLLM in the usual vision-language sense (Zhang et al., 8 Jul 2025).

A central component of the method is the construction of a robust training dataset for rationale supervision. For each aspect-level dialogue segment with known label $A$5, GPT-4o is prompted with the aspect name, the ground-truth score, and the dialogue content, and is required to produce output in a strict format:

• Evaluation Result:
• Evaluation Reason:

Because the true label is provided in the prompt, GPT-4o is used primarily to generate rationale text consistent with the gold score rather than to infer the score itself. This yields supervised pairs of interview content and clinically formatted explanations, which are then used to fine-tune the smaller LLM (Zhang et al., 8 Jul 2025).

Training proceeds in two stages. In the first stage, the LLM is frozen and the LQ-former is trained so that non-textual embeddings become meaningful to the frozen decoder. In the second stage, the LQ-former is frozen and the LLM is fine-tuned with a joint objective:

$A$6

Here $A$7 supervises generation of both the score and the rationale, while $A$8 supervises the regression head against numeric depression scores (Zhang et al., 8 Jul 2025).

Fine-tuning is parameter-efficient. The model uses LoRA on the query and value projection matrices of the LLaMA-3-8B attention layers, with $A$9, $V$0, and dropout $V$1. Training uses a learning rate of $V$2 for 10 epochs on 2×A800 GPUs. This design choice is consistent with a broader pattern in multimodal adaptation: retain a strong pretrained language backbone and introduce a lightweight modality bridge rather than retraining the full model from scratch.

The datasets are structurally distinct. CMDC contains 78 subjects, with 12 interview questions manually aligned to 9 PHQ-9 aspects, producing $V$3 aspect-level samples. E-DAIC-WOZ contains train/valid/test splits of 163/56/56 subjects and uses the full interview as input for each PHQ-8 aspect, producing $V$4 aspect-level samples for training. This difference matters because the LQ-former is more effective when it can summarize short, aspect-specific clips than when it must compress a long, open-ended interview (Zhang et al., 8 Jul 2025).

4. Explainability and rationale generation

Explainability in MLlm-DR is implemented as structured natural-language generation rather than as post hoc saliency or attention visualization. The model is trained to emit both a numeric aspect score and a concise evaluation reason in a fixed template. The rationale is expected to refer to dialogue evidence and to align with PHQ item semantics, while speech and visual evidence enter implicitly through the multimodal embeddings inserted into the LLM context (Zhang et al., 8 Jul 2025).

This rationale mechanism addresses a frequent criticism of earlier multimodal depression systems: even when they achieve strong predictive performance, their interpretability is weak relative to clinical workflows. MLlm-DR instead produces textual justifications that can be inspected directly. This does not imply causal transparency in a strict mechanistic sense, but it does provide an audit trail closer to psychiatric assessment practice.

Human evaluation was conducted on 100 test cases per dataset. Experts assessed output format, agreement between model predictions and expert scores, and the quality of rationales on a three-point scale. On CMDC, only 1.23% of outputs were misformatted, 87% of predictions were consistent with expert scores, and rationale ratings were 77% “fully agree,” 14% “reasonable,” and 9% “disagree.” On E-DAIC-WOZ, 3.12% of outputs were misformatted, 73% were consistent with expert scores, and rationale ratings were 64%, 19%, and 17%, respectively (Zhang et al., 8 Jul 2025).

These results clarify another common misconception: explainability here is not merely a side effect of using an LLM. The rationales are produced because the model is explicitly trained on GPT-4o-generated, label-consistent explanations and because the output format is part of the supervision target. A plausible implication is that explanation quality depends at least as much on the curation of rationale data as on the underlying LLM.

5. Empirical performance and ablation evidence

MLlm-DR is evaluated as both a classification system for depression screening and a regression system for PHQ score prediction. On CMDC, the text-only LLaMA-3-8B baseline achieved CCC $V$5, RMSE $V$6, MAE $V$7, Precision $V$8, Recall $V$9, and F1 $T$0. A text-only LoRA fine-tuned variant improved to CCC $T$1, RMSE $T$2, MAE $T$3, Precision $T$4, Recall $T$5, and F1 $T$6. The full multimodal MLlm-DR achieved CCC $T$7, RMSE $T$8, MAE $T$9, Precision $s_i \in \{0,1,2,3\}$0, Recall $s_i \in \{0,1,2,3\}$1, and F1 $s_i \in \{0,1,2,3\}$2 (Zhang et al., 8 Jul 2025).

On E-DAIC-WOZ, the text-only LLaMA-3-8B baseline achieved CCC $s_i \in \{0,1,2,3\}$3, RMSE $s_i \in \{0,1,2,3\}$4, MAE $s_i \in \{0,1,2,3\}$5, Precision $s_i \in \{0,1,2,3\}$6, Recall $s_i \in \{0,1,2,3\}$7, and F1 $s_i \in \{0,1,2,3\}$8. The text-only LoRA version improved to CCC $s_i \in \{0,1,2,3\}$9, RMSE $S = \sum_i s_i$0, MAE $S = \sum_i s_i$1, Precision $S = \sum_i s_i$2, Recall $S = \sum_i s_i$3, and F1 $S = \sum_i s_i$4. The full multimodal MLlm-DR achieved CCC $S = \sum_i s_i$5, RMSE $S = \sum_i s_i$6, MAE $S = \sum_i s_i$7, Precision $S = \sum_i s_i$8, Recall $S = \sum_i s_i$9, and F1 $Q \in \mathbb{R}^{n \times D}$0, which the paper reports as state-of-the-art on the dataset (Zhang et al., 8 Jul 2025).

Ablation results identify two decisive ingredients: the LQ-former and the joint objective. On CMDC, removing the joint loss reduced performance from CCC $Q \in \mathbb{R}^{n \times D}$1, RMSE $Q \in \mathbb{R}^{n \times D}$2, MAE $Q \in \mathbb{R}^{n \times D}$3 to CCC $Q \in \mathbb{R}^{n \times D}$4, RMSE $Q \in \mathbb{R}^{n \times D}$5, MAE $Q \in \mathbb{R}^{n \times D}$6. Removing the LQ-former reduced performance further to CCC $Q \in \mathbb{R}^{n \times D}$7, RMSE $Q \in \mathbb{R}^{n \times D}$8, MAE $Q \in \mathbb{R}^{n \times D}$9. On E-DAIC-WOZ, the full model achieved CCC $F \in \mathbb{R}^{m \times D}$0, F1 $F \in \mathbb{R}^{m \times D}$1, RMSE $F \in \mathbb{R}^{m \times D}$2, MAE $F \in \mathbb{R}^{m \times D}$3, whereas removing the LQ-former reduced this to CCC $F \in \mathbb{R}^{m \times D}$4, F1 $F \in \mathbb{R}^{m \times D}$5, RMSE $F \in \mathbb{R}^{m \times D}$6, MAE $F \in \mathbb{R}^{m \times D}$7 (Zhang et al., 8 Jul 2025).

The modality-specific ablations are also instructive. On CMDC, removing audio LQ-A yielded CCC $F \in \mathbb{R}^{m \times D}$8, RMSE $F \in \mathbb{R}^{m \times D}$9, MAE $$\begin{aligned} H &= \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V, \ O &= \text{Projection}(H), \end{aligned}$$0, whereas removing visual LQ-V yielded CCC $$\begin{aligned} H &= \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V, \ O &= \text{Projection}(H), \end{aligned}$$1, RMSE $$\begin{aligned} H &= \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V, \ O &= \text{Projection}(H), \end{aligned}$$2, MAE $$\begin{aligned} H &= \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V, \ O &= \text{Projection}(H), \end{aligned}$$3. On E-DAIC-WOZ, removing LQ-A produced CCC $$\begin{aligned} H &= \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V, \ O &= \text{Projection}(H), \end{aligned}$$4, RMSE $$\begin{aligned} H &= \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V, \ O &= \text{Projection}(H), \end{aligned}$$5, MAE $$\begin{aligned} H &= \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V, \ O &= \text{Projection}(H), \end{aligned}$$6, while removing LQ-V produced CCC $$\begin{aligned} H &= \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V, \ O &= \text{Projection}(H), \end{aligned}$$7, RMSE $$\begin{aligned} H &= \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V, \ O &= \text{Projection}(H), \end{aligned}$$8, MAE $$\begin{aligned} H &= \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V, \ O &= \text{Projection}(H), \end{aligned}$$9. This suggests that audio is particularly informative on short, aspect-aligned clips, while the contribution of visual cues becomes relatively more competitive on long, open-ended interviews.

The isolated LQ-former analysis reinforces this interpretation. With the LLM frozen and no text input, the combined LQ-former on CMDC achieved CCC $K = V \in \mathbb{R}^{m \times D}$0, Precision $K = V \in \mathbb{R}^{m \times D}$1, Recall $K = V \in \mathbb{R}^{m \times D}$2, F1 $K = V \in \mathbb{R}^{m \times D}$3, RMSE $K = V \in \mathbb{R}^{m \times D}$4, and MAE $K = V \in \mathbb{R}^{m \times D}$5. On E-DAIC-WOZ, the same setting achieved CCC only $K = V \in \mathbb{R}^{m \times D}$6, RMSE $K = V \in \mathbb{R}^{m \times D}$7, and MAE $K = V \in \mathbb{R}^{m \times D}$8. The paper attributes this gap to the difficulty of compressing long interviews with fixed learnable queries, which is a substantive limitation rather than a mere optimization detail (Zhang et al., 8 Jul 2025).

6. Broader context, misconceptions, and limitations

Within the broader multimodal LLM literature, MLlm-DR exemplifies a design pattern in which a pretrained LLM is preserved and non-text modalities are introduced through lightweight interfaces rather than through wholesale architectural replacement. This strategy is shared, in different forms, by discrete multimodal LLMs that expand a text vocabulary to include speech or image tokens (Trinh et al., 2024), by visual-expert systems that add modality-specific attention parameters while preserving the text pathway (She et al., 2024), by parameter-efficient multimodal adaptation via low-rank modules (Palaskar et al., 2024), and by compact domain-adapted MLLMs for autonomous driving (Zheng et al., 9 Jan 2025). Distillation-oriented frameworks such as LLaVA-KD likewise emphasize transferring multimodal competence into smaller backbones without altering the core student architecture (Cai et al., 2024).

MLlm-DR differs from these systems in task and data regime. Its multimodal evidence is not raw image-text data but pre-extracted speech and visual interview features under privacy constraints. Its output is not captioning, VQA, or general dialogue, but PHQ-style scores and clinically shaped rationales. This makes it a domain-specific MLLM rather than a general-purpose one, even though it inherits the backbone-and-adapter logic common to current multimodal LLM design.

Several objective limitations remain. The rationale supervision dataset is constructed primarily from text, which may bias the model when nonverbal evidence conflicts with verbal reports. The underlying datasets are relatively small, limiting confidence about generalization across populations, languages, and interview protocols. Fine-grained labels for speech and visual data are absent, so the model cannot localize which exact frames or acoustic events support a rationale. The paper therefore does not claim deployment readiness; it explicitly identifies the need for larger multimodal depression datasets, finer-grained speech and visual supervision, and broader clinical validation (Zhang et al., 8 Jul 2025).

A final misconception is that rationales produced by a clinical MLLM necessarily reflect grounded multimodal evidence in a fully transparent way. MLlm-DR provides human-readable justifications and strong empirical alignment with expert judgments, but its explanations are still generated by a learned decoder operating over latent multimodal summaries. The significance of the framework lies less in solving interpretability outright than in showing that domain-adapted multimodal LLMs can couple competitive depression recognition with structured, clinically reviewable explanatory text.