WEE-Therapy: Ensemble AudioLLM for Counseling Analysis

• WEE-Therapy is a domain-adaptive AudioLLM that uses a weak encoder ensemble to capture emotional cues, therapeutic techniques, and crisis risk in counseling dialogues.
• It employs a dual-routing mechanism that combines data-independent and data-dependent expert selection to enhance feature fusion with minimal overhead.
• Evaluated on tasks like emotion recognition, technique classification, risk detection, and summarization, it shows significant performance gains over single-encoder baselines.

WEE-Therapy is a multi-task AudioLLM for psychological counseling dialogue analysis that introduces a Weak Encoder Ensemble (WEE) mechanism. It supplements a “strong” base encoder with a pool of lightweight, specialized encoders and uses a dual-routing strategy that combines stable, data-independent domain knowledge with dynamic, data-dependent expert selection. The framework is evaluated on emotion recognition, technique classification, risk detection, and dialogue summarization, where it reports significant performance gains across all tasks with minimal parameter overhead (Kang et al., 24 Sep 2025).

1. Problem Setting and Scope

WEE-Therapy is situated in computational psychology, specifically in the analysis of counseling dialogues. Its stated motivation is that existing audio LLMs often rely on a single speech encoder pre-trained on general data and therefore struggle to capture domain-specific features such as complex emotions and professional techniques. Within this formulation, the target problem is not generic speech understanding but counseling-analysis under clinically relevant objectives, including negative-affect recognition, counselor-technique identification, crisis-risk detection, and summary generation (Kang et al., 24 Sep 2025).

The framework is therefore best understood as a domain-adaptive AudioLLM rather than a narrowly defined classifier. It is explicitly multi-task, and its design assumes that counseling speech contains heterogeneous signals that are not optimally represented by one encoder alone. This suggests a mixture-of-experts interpretation in which specialization is introduced at the encoder layer, not only at the decoder or task-head level.

2. Encoder Composition and Feature Fusion

The model defines a strong base encoder $E_b$, exemplified by Whisper-large-v3 with approximately $637$ million parameters. For an input audio segment $a_i$, the base representation is

$$z_{i,base}=E_b(a_i)\in\mathbb{R}^{d_{base}\times T}.$$

Alongside the base encoder, WEE-Therapy maintains a pool of $M$ weak encoders $\{E_1,\dots,E_M\}$, with examples including Whisper-tiny, HuBERT-base, and Emotion-finetuned-HuBERT. Each weak encoder produces features with the same temporal length $T$, typically of dimension $d_w$. The weak-encoder contribution is split into a data-independent component and a data-dependent component:

$$z_{i,indep}=\sum_{k=1}^M r_{indep}[k]\,E_k(a_i), \qquad z_{i,dep}=\sum_{k=1}^M r_{i,dep}[k]\,E_k(a_i).$$

An equivalent high-level additive formulation is

$F(a_i)=E_b(a_i)+\sum_{k=1}^M r_k(a_i)\,E_k(a_i),$

with

$637$0

In the reported implementation, the three streams are concatenated along the feature dimension after projection, while the additive decomposition provides an equivalent conceptual view (Kang et al., 24 Sep 2025).

This decomposition is central to the framework’s design. The base encoder provides a strong general representation, while the weak encoders inject targeted inductive biases. The architecture does not treat the weak encoders as replacements for the base model; rather, they act as specialized supplements.

3. Dual-Routing Mechanism

The WEE mechanism consists of two routing paths. The first is static, or data-independent, and associates each weak encoder $637$1 with a learnable domain-prior weight $637$2. The routing scores are produced by a softmax,

$637$3

followed by a KeepTop1 operator,

$637$4

The second route is dynamic, or data-dependent. It first mean-pools the base-encoder output over time,

$637$5

then projects the pooled representation through $637$6 to obtain an $637$7-dimensional score vector,

$637$8

Again, sparsity is enforced through KeepTop1:

$637$9

The paper keeps the two routes distinct and adds their outputs in feature space. Because the selections are top-1 in both routes, only a single static and a single dynamic weak encoder are active per input. A common misconception is to read WEE-Therapy as a dense ensemble over all weak encoders; the reported mechanism is instead sparse, with routing explicitly constructed to force single-expert selection in each path (Kang et al., 24 Sep 2025).

4. Multi-Task Objective and Evaluation Setting

WEE-Therapy is trained on four counseling-analysis tasks. The losses are defined as follows: $a_i$0 for emotion-state classification, $a_i$1 for counselor-technique classification, $a_i$2 for crisis-risk detection, and $a_i$3 for dialogue summarization. The joint optimization objective is

$a_i$4

The $a_i$5 coefficients are tuned on a development set. The paper uses roughly equal weighting except for a small routing auxiliary term of weight $a_i$6 on the WEE-routing entropy/diversity losses (Kang et al., 24 Sep 2025).

The evaluation suite spans four datasets or task collections. Emotion Recognition uses DAIC-WOZ with 189 clinical interview sessions, approximately 7 hours, and approximately 2,500 utterances. Technique Classification uses a simulated dataset with approximately 5,000 counselor–client exchanges and 6 techniques labeled. Crisis Risk Detection uses a self-annotated set with approximately 400 segments with high-risk labels and heavy class imbalance. Dialogue Summarization uses approximately 200 full dialogues with human summaries. The reported metrics are Macro F1 for emotion recognition, Accuracy and Macro F1 for technique classification, Precision@5 for crisis-risk detection, ROUGE-L recall for dialogue summarization, and an overall human/GPT-4 holistic score from 0 to 5 (Kang et al., 24 Sep 2025).

The selection of tasks and metrics indicates that WEE-Therapy is intended as a unified clinical-analysis pipeline rather than a collection of independent benchmarks. This suggests an emphasis on deployment scenarios in which affective inference, technique coding, risk triage, and documentation are operationally linked.

5. Empirical Performance and Computational Profile

The main reported baselines are Whisper-only, HuBERT-only, Data-Indep only, and Data-Dep only. On the four principal metrics, WEE-Therapy reports the following results: ER $a_i$7 Macro F1, CTC $a_i$8 Accuracy, CMD $a_i$9 Precision@5, and DS $$z_{i,base}=E_b(a_i)\in\mathbb{R}^{d_{base}\times T}.$$0 ROUGE-L recall. The corresponding Whisper-only results are $$z_{i,base}=E_b(a_i)\in\mathbb{R}^{d_{base}\times T}.$$1, $$z_{i,base}=E_b(a_i)\in\mathbb{R}^{d_{base}\times T}.$$2, $$z_{i,base}=E_b(a_i)\in\mathbb{R}^{d_{base}\times T}.$$3, and $$z_{i,base}=E_b(a_i)\in\mathbb{R}^{d_{base}\times T}.$$4, yielding improvements of $$z_{i,base}=E_b(a_i)\in\mathbb{R}^{d_{base}\times T}.$$5 points on emotion recognition, $$z_{i,base}=E_b(a_i)\in\mathbb{R}^{d_{base}\times T}.$$6 points on technique classification accuracy, $$z_{i,base}=E_b(a_i)\in\mathbb{R}^{d_{base}\times T}.$$7 points on crisis-risk detection, and $$z_{i,base}=E_b(a_i)\in\mathbb{R}^{d_{base}\times T}.$$8 points on summarization. The paper states that all four tasks see consistent, statistically significant gains (Kang et al., 24 Sep 2025).

The ablations also separate the contributions of the two routing modes. Data-independent routing alone improves over the single-encoder baselines, and data-dependent routing alone improves further, but the full WEE configuration remains best on all four reported tasks. This pattern is consistent with the intended division of labor between stable domain priors and input-adaptive expert selection.

The computational design is explicitly lightweight relative to the base model. The weak encoders total approximately $$z_{i,base}=E_b(a_i)\in\mathbb{R}^{d_{base}\times T}.$$9 million parameters, combining Whisper-tiny ($M$0 million), HuBERT-base ($M$1 million), and Emotion-HuBERT ($M$2 million). Because only one static and one dynamic weak encoder run per input, the additional forward cost is approximately one weak encoder, or the compute equivalent of $M$3 to $M$4 million parameters. Inference overhead is reported as approximately $M$5 to $M$6 over the base encoder alone. During training, most of the $M$7 million-parameter base encoder and the $M$8 billion-parameter LLM are frozen; only the routers, a small adapter/projection, and LoRA-injected LLM weights are updated, which keeps fine-tuning efficient (Kang et al., 24 Sep 2025).

6. Interpretation, Applications, and Terminological Ambiguity

The framework attributes distinct representational roles to the weak encoders. Whisper-tiny captures overall speech content quickly, HuBERT-base captures phonetic and acoustic nuances, and Emotion-finetuned HuBERT abstracts fine emotional cues such as pitch and prosody. Static routing is described as ensuring that emotion cues are always injected, while dynamic routing allows the model to select, for example, the phonetic or content expert when the client is speaking rapidly or using novel terminology (Kang et al., 24 Sep 2025).

From this perspective, WEE-Therapy functions as a practical mechanism for bridging the domain gap of general AudioLLMs without full re-pretraining. The stated application implications are more objective, consistent analysis of real counseling sessions, early warning of crisis risk, automated post-session summaries for supervisors, and a low-cost fine-tuning recipe requiring only a few million trainable parameters. Future work is identified as extending to larger real-world counseling corpora, more fine-grained expert pools such as silence or sigh detectors, multi-head routing, and real-time deployment in tele-mental-health platforms (Kang et al., 24 Sep 2025).

The term “WEE-Therapy” also appears in a separate guide derived from “Therapist-Exoskeleton-Patient Interaction: An Immersive Gait Therapy” (Küçüktabak et al., 21 Jul 2025), where it is expanded as “Wearable Exoskeleton Embodiment” Therapy and refers to a bidirectional pHRHI gait-rehabilitation paradigm involving therapist and patient exoskeleton coupling. That usage concerns lower-limb exoskeleton control, spring-damper virtual coupling, and treadmill-based post-stroke gait training, whereas the WEE-Therapy of (Kang et al., 24 Sep 2025) concerns Weak Encoder Ensemble design for counseling-dialogue AudioLLMs. The shared label therefore denotes two distinct technical constructs in the provided literature, and conflating them obscures both the computational-psychology context of the former and the rehabilitation-robotics context of the latter.