HAREN-CTC: Hierarchical Speech Depression Detector

• HAREN-CTC is a speech-based depression detection architecture that integrates layered SSL representations with CTC-based temporal supervision.
• It reorganizes multiple SSL layers into adaptive shallow and deep embeddings and fuses them via cross-attention to capture both acoustic and semantic cues.
• Empirical results show superior macro F1-scores on DAIC-WOZ and MODMA, validating its effective multi-task learning and generalization capabilities.

Searching arXiv for the HAREN-CTC paper and closely related context. Searching arXiv for HAREN-CTC. HAREN-CTC is a speech-based depression detection architecture that integrates multi-layer self-supervised speech representations within a multitask learning framework and uses Connectionist Temporal Classification (CTC) to model sparse temporal supervision. It is designed to address a central difficulty in speech-based depression detection: depressive cues are sparse, heterogeneous, and irregularly distributed over time, while conventional systems often rely on a single self-supervised learning (SSL) layer or a single best-performing representation. HAREN-CTC instead reorganizes hierarchical SSL features, fuses complementary levels through cross-attention, and combines global classification with alignment-aware temporal supervision. In reported experiments, it achieves macro F1-scores of 0.81 on DAIC-WOZ and 0.82 on MODMA under an upper-bound setting, and it also improves generalization under five-fold cross-validation (Li et al., 5 Oct 2025).

1. Definition and problem setting

HAREN-CTC, described in the source as Hierarchical Acoustic Representation Encoding Network with Connectionist Temporal Classification, is a model for depression detection from speech that uses pretrained SSL representations such as WavLM and HuBERT. Its motivation is the observation that most existing speech-based depression detection methods either use only the final SSL layer or search for a single best layer, which can overfit to dataset-specific effects and fail to exploit the hierarchical structure of speech representations (Li et al., 5 Oct 2025).

The model targets two related problems. First, depressive speech cues are not uniformly present throughout an utterance; they may be brief, weak, and temporally irregular. Second, SSL encoders produce layered representations whose lower layers tend to encode more acoustic and prosodic structure, while higher layers encode more abstract information. HAREN-CTC is constructed to use both kinds of information rather than forcing a fixed-layer selection.

A common misconception is to treat the model primarily as a CTC system in the automatic speech recognition sense. In HAREN-CTC, CTC is not used for transcript decoding. Instead, it functions as an alignment-aware supervisory signal for sparse depression-related temporal structure. The primary task remains session-level depression classification, with CTC acting as an auxiliary temporal objective (Li et al., 5 Oct 2025).

2. Hierarchical representation organization

The first major component is the Hierarchical Adaptive Clustering (HAC) module, which groups selected SSL hidden states into two complementary embeddings, described as “shallow” and “deep.” The model collects selected hidden representations from $m$ encoder layers, $H_{i_1}, \ldots, H_{i_m}$, and introduces a trainable assignment matrix $G \in \mathbb{R}^{m \times 2}$ so that each selected layer receives a probabilistic assignment to one of two clusters (Li et al., 5 Oct 2025).

The assignments are normalized by softmax: $$P_{l,k} = \frac{\exp(G_{l,k})}{\sum_{k'=1}^2 \exp(G_{l,k'})}$$

The clustered representations are then formed by weighted aggregation: $$U^{(k)} = \sum_{l=1}^m P_{l,k} S_l, \quad k \in \{\text{shallow}, \text{deep}\}$$

The source states that $G$ is initialized so that early layers favor the shallow group and later layers favor the deep group, using exponential decay $\alpha$. This preserves an initial inductive bias while allowing data-driven reassignment during training. The stated purpose is to avoid the brittleness of fixed-layer selection and to learn complementary embeddings that reflect both low-level acoustic/prosodic structure and higher-level contextual information (Li et al., 5 Oct 2025).

This organization is central to the model’s claim of “hierarchical” learning. Rather than averaging or concatenating all layers without structure, HAC explicitly partitions them into two adaptive groups. A plausible implication is that the model is intended to regularize representation use across datasets by preventing overcommitment to a single layer, but the empirical claim made directly in the source is that single-layer models are highly unstable and perform worse in both evaluation settings (Li et al., 5 Oct 2025).

3. Cross-modal fusion and inter-layer dependency modeling

The second major component is the Cross-Modal Fusion (CMF) module, which models inter-layer dependencies through cross-attention. The grouped representations are denoted $U^{\text{shallow}} \in \mathbb{R}^{T \times d}$ and $U^{\text{deep}} \in \mathbb{R}^{T \times d}$. The deep group acts as queries, while the shallow group acts as keys and values (Li et al., 5 Oct 2025).

The source gives the cross-attention formulation as: $$Q = W_Q U^{\text{deep}}, \quad K = W_K U^{\text{shallow}}, \quad V = W_V U^{\text{shallow}}$$

$H_{i_1}, \ldots, H_{i_m}$0

This module is intended to fuse high-level and low-level cues dynamically. According to the source, the deep group is treated as more semantic and the shallow group as more acoustic. Cross-attention therefore allows the model to relate abstract contextual patterns to temporally localized acoustic realizations. The fused representation $H_{i_1}, \ldots, H_{i_m}$1 is then used by both the global depression classifier and the temporal CTC head (Li et al., 5 Oct 2025).

This design distinguishes HAREN-CTC from approaches that either pool SSL representations directly or use monolithic fusion. In the reported interpretation, CMF enables the model to capture subtle or localized depressive cues that might be obscured in global pooled features. That interpretive claim appears explicitly in the provided description and aligns with the broader architectural choice of pairing hierarchical aggregation with attention-based fusion (Li et al., 5 Oct 2025).

4. CTC supervision for sparse depressive cues

The CTC component addresses the fact that depression-related cues are sparse and do not come with frame-level annotations. The source states that HAREN-CTC uses CTC to provide alignment-aware supervision without requiring precise timing labels. This is the architectural role of “CTC” in the model name (Li et al., 5 Oct 2025).

Pseudo-labels are generated using HuBERT hidden states. Specifically, HuBERT is used to extract hidden states, for example layer 12 output, and these states are clustered with K-means using $H_{i_1}, \ldots, H_{i_m}$2 to produce discrete token sequences for each utterance. For depressive samples, cluster tokens are reindexed so that their token span does not overlap with non-depressed samples. After collapsing consecutive duplicates in CTC style, the result is a target sequence $H_{i_1}, \ldots, H_{i_m}$3 without explicit frame alignment (Li et al., 5 Oct 2025).

The CTC objective is written as: $H_{i_1}, \ldots, H_{i_m}$4

The model is trained with a multitask loss that combines binary focal loss and CTC loss: $H_{i_1}, \ldots, H_{i_m}$5

The binary focal loss is given as: $H_{i_1}, \ldots, H_{i_m}$6

In this formulation, the focal component addresses class imbalance in global depression classification, while the CTC component encourages the fused representation to learn temporal structure that is compatible with sparse and irregular depressive markers. The source explicitly states that CTC is well suited to this setting because it sums over valid alignments and therefore does not require frame-level labels (Li et al., 5 Oct 2025).

5. SSL backbone usage and training configuration

HAREN-CTC uses WavLM-Large as a frozen feature extractor and obtains per-layer hidden representations $H_{i_1}, \ldots, H_{i_m}$7, with $H_{i_1}, \ldots, H_{i_m}$8 for WavLM-Large. Instead of choosing a single layer, the model stacks and clusters multiple layers through HAC, then fuses them through CMF. For CTC labeling, the configuration described in the source uses HuBERT-Large (Li et al., 5 Oct 2025).

The evaluation is conducted on two datasets. DAIC-WOZ is described as English clinical interviews with $H_{i_1}, \ldots, H_{i_m}$9, including 56 depressed samples. MODMA is described as Mandarin interviews with $G \in \mathbb{R}^{m \times 2}$0, including 23 depressed samples. Two evaluation settings are reported: a performance upper-bound setting using official splits, and a generalization setting using five-fold stratified cross-validation. The reported metrics are macro F1 score, recall, and precision (Li et al., 5 Oct 2025).

This training and evaluation protocol is important because it separates within-split performance from cross-fold generalization. The source emphasizes that many earlier methods can appear competitive under standard splits while remaining sensitive to layer choice or dataset composition. HAREN-CTC is presented as addressing that issue through hierarchical feature integration rather than layer selection heuristics (Li et al., 5 Oct 2025).

6. Empirical results, ablations, and comparative position

Under the performance upper-bound evaluation, HAREN-CTC reports the following results:

Dataset	Model	Macro F1	Recall	Precision
DAIC-WOZ	HAREN-CTC	0.81	0.81	0.81
MODMA	HAREN-CTC	0.82	0.83	0.86

The same table in the source places HAREN-CTC above DMPF, DALF, Vlad-GRU, and other prior systems on the listed datasets. On DAIC-WOZ, DMPF is reported at macro F1 $G \in \mathbb{R}^{m \times 2}$1, DALF at $G \in \mathbb{R}^{m \times 2}$2, and Vlad-GRU at $G \in \mathbb{R}^{m \times 2}$3. On MODMA, DMPF is reported at $G \in \mathbb{R}^{m \times 2}$4 and Vlad-GRU at $G \in \mathbb{R}^{m \times 2}$5 (Li et al., 5 Oct 2025).

Under the generalization evaluation, the source reports macro F1 values of $G \in \mathbb{R}^{m \times 2}$6 on DAIC-WOZ and $G \in \mathbb{R}^{m \times 2}$7 on MODMA for HAREN-CTC. In the same comparison, DepAudioNet is reported at $G \in \mathbb{R}^{m \times 2}$8 on DAIC-WOZ and $G \in \mathbb{R}^{m \times 2}$9 on MODMA, while Speechformer is reported at $$P_{l,k} = \frac{\exp(G_{l,k})}{\sum_{k'=1}^2 \exp(G_{l,k'})}$$0 on DAIC-WOZ (Li et al., 5 Oct 2025).

The ablation summary in the source states three points. First, single-layer models are highly unstable and worse across both settings. Second, removing HAC causes a large drop in performance. Third, removing CTC also yields a large drop. These ablations are presented as evidence that the model’s gains are not attributable to WavLM features alone, but to the joint use of hierarchical grouping, cross-attentive fusion, and alignment-aware supervision (Li et al., 5 Oct 2025).

7. Interpretation, scope, and limitations

HAREN-CTC occupies a specific methodological position within speech-based depression detection. It is neither a pure end-to-end classifier over a single SSL layer nor a temporal model with explicit frame labels. Its defining claim is that robust depression detection benefits from three ingredients used together: hierarchical SSL feature utilization, adaptive shallow/deep organization, and CTC-based temporal supervision (Li et al., 5 Oct 2025).

Several interpretive points follow directly from the source. The model is intended to be less dependent on chance layer selection than prior single-layer systems. It is also intended to balance recall and precision more effectively than some earlier baselines. Because it uses a frozen SSL backbone, it treats pretrained speech representations as a stable substrate and learns the depression-specific structure primarily in the HAC, CMF, and multitask heads (Li et al., 5 Oct 2025).

The reported evidence remains bounded by the datasets and settings described. DAIC-WOZ and MODMA are relatively small corpora, and the model’s CTC targets are pseudo-labels derived from HuBERT hidden states and K-means clustering rather than direct clinical temporal annotations. This suggests that HAREN-CTC should be understood as a weakly supervised alignment-aware architecture rather than a system trained on explicit depression-cue timestamps. Within that scope, the reported contribution is a state-of-the-art macro F1 of 0.81 on DAIC-WOZ and 0.82 on MODMA, together with improved cross-validation performance and ablation evidence that both HAC and CTC are materially important to the model’s behavior (Li et al., 5 Oct 2025).