Structure Matters: Evaluating Multi-Agents Orchestration in Generative Therapeutic Chatbots

Abstract: While LLMs excel at open-ended dialogue, effective psychotherapy requires structured progression and adherence to clinical protocols, making the design of psychotherapist chatbots challenging. We investigate how different LLM-based designs shape perceived therapeutic dialogue in a chatbot grounded in the Self-Attachment Technique (SAT), a novel self-administered psychotherapy rooted in attachment theory. We compare three architectural variants: (1) a multi-agent system utilizing finite state machine aligned with therapeutic stages and a shared long-term memory, (2) a single-agent using identical knowledge-base and the same prompts, and (3) an unguided LLM. In an eight-day randomized controlled trial (RCT) with N=66 Farsi-speaking participants, balanced across the three chatbots, the multi-agent system is perceived as significantly more natural and human-like than the other variants and achieves higher ratings across most other metrics. These findings demonstrate that for therapeutic AI, architectural orchestration is as critical as prompt engineering in fostering natural, engaging dialogue.

• The paper demonstrates that a 12-state, memory-augmented multi-agent FSM significantly improved perceived naturalness over equivalent single-agent and unguided GPT-4o systems, scoring 3.955 versus 3.043 and 3.211 on a five-point scale.
• The randomized eight-day trial with 66 Farsi-speaking participants held prompts, knowledge, interface, and model constant, isolating orchestration as the main architectural difference and reducing concerns that content drove the result.
• The findings suggest that shorter, more frequent turns and explicit therapeutic state management can improve conversational flow, but architecture alone did not significantly increase trust, empathy, memory, satisfaction, or clinical outcomes.

Overview

This paper reports a randomized controlled trial (RCT) examining how architectural orchestration of LLM-based conversational agents affects perceived therapeutic quality in a chatbot grounded in the Self-Attachment Technique (SAT), an attachment-theory-based, self-administered psychotherapy. The central claim is that system architecture—not merely prompt engineering or the underlying model—materially shapes user experience: a multi-agent finite state machine (FSM) design was perceived as significantly more natural and human-like than both a single-agent control with identical prompts and knowledge, and an unguided GPT-4o placebo (2603.00774).

Background and motivation

Prior SAT chatbots relied on rule-based dialogue and emotion classification, enforcing therapeutic structure but lacking generative flexibility, contextual adaptation, long-term personalization, and multilingual support (2603.00774). Broader work on mental health conversational agents—CBT systems, Woebot-style interventions, FSM-driven systems such as MindfulDiary and ChaCha, and emerging multi-agent frameworks—has largely emphasized feasibility and acceptability while rarely comparing architectural paradigms under controlled conditions (2603.00774). The paper positions itself against this gap: it embeds a context-aware LLM within a multi-agent, FSM-driven SAT architecture and evaluates it via one of the first RCTs of a Farsi-language SAT chatbot.

System design

All three conditions run on a Django REST backend using GPT-4o as the conversational engine, deployed through an identical React.js web interface, with interactions conducted in Farsi while prompts and structure were authored in English. The three conditions are:

• Alpha (Intervention): a 12-state FSM aligned with SAT stages, organized into four phases (Initiation, Exploration, Intervention, Conclusion). State transitions use two mechanisms: LLM-as-Judge sufficiency detection, which evaluates whether accumulated user responses satisfy emotional/informational criteria before advancing, and intent-based routing over more than 40 affirmative/negative Farsi patterns via a fine-tuned BERT classifier. Alpha maintains shared long-term memory through prompt-guided LLM summarization every three messages, and delivers exercises via a hierarchical adaptive RAG pipeline combining constraint-based filtering (by protocol day and therapeutic stage) with LLM semantic re-ranking and personalization.
• Beta (Active Control): a single agent given the exact same prompts collapsed into one sequential system prompt, plus the same SAT knowledge base and static exercise schedule, but no FSM enforcement.
• Gamma (Placebo Control): a minimal single-agent "digital companion" with no SAT knowledge or structured goals.

This design isolates architecture from content: Alpha and Beta share prompts and knowledge, so differences are attributable to orchestration rather than material delivered.

Study and results

The eight-day between-subjects study enrolled $N=66$ Farsi-speaking participants (Alpha $n=22$, Beta $n=23$, Gamma $n=21$; mean age 27.2). Baseline distress measures were balanced ($\eta^2 \leq 0.053$), 71% reported high LLM familiarity, and 74% had no prior SAT exposure. Analysis used one-way ANOVA with permutation-derived p-values (5,000 iterations).

The primary finding is a significant effect on perceived naturalness/human-like tone: Alpha scored $\mu=3.955$ versus Beta's 3.043 and Gamma's 3.211 ($F=7.017$, $p_{perm}=0.0018$, $\eta^2=0.187$)—a nearly one-point gain on a 5-point scale, with architecture explaining roughly 19% of rating variance. Pairwise comparisons showed Alpha outperforming both controls, which did not differ from each other, indicating that explicit multi-agent state management contributes beyond content alone. Baseline usability metrics were equivalent across groups (ease of use: $F=0.204$, $n=22$0), isolating the effect to relational quality rather than functional UX.

Metric	Alpha	Beta	Gamma	$n=22$1	$n=22$2
Naturalness*	3.955	3.043	3.211	7.017	0.187
Trust	3.136	2.739	2.737	0.737	0.024
Empathy	3.727	3.435	3.895	1.357	0.043
Memory	3.682	3.348	3.263	1.061	0.034
Satisfaction	3.636	3.348	3.474	0.520	0.017
Off-topic replies (lower better)	1.727	2.304	2.000	1.481	0.046
Ease of use	4.000	3.826	3.789	0.204	0.007

*Significant at $n=22$3.

Secondary metrics trended in Alpha's favor for most items (trust, memory coherence, satisfaction, fewer off-topic replies) but were not statistically significant. Notably, Gamma scored highest on perceived empathy (3.895 vs. Alpha's 3.727), a pattern the authors interpret as evidence that naturalness, empathy, trust, and personalization are distinct dimensions of the therapeutic alliance that architectural improvements alone do not jointly improve.

Chat log analysis offers a mechanistic account. Normalized per participant, Alpha generated approximately 20.9 agent messages versus 14.6 (Beta) and 9.8 (Gamma)—43% and 113% increases—while agent messages were substantially shorter (229.8 characters vs. 408.7 and 635.1). The agent-to-user message length ratio was lowest in Alpha (7.9:1 vs. 10.5:1 and 13.4:1). The authors argue this reflects a shift from monologic, lecture-style responses toward rapid turn-taking that emulates human therapeutic conversation, challenging the assumption that chatbots should maximize information per turn. Lexicon-based sentiment analysis further showed Alpha's agent messages carried the highest mean sentiment score (0.629 vs. 0.480 and 0.465), with nearly 80% classified as positive.

Discussion

The paper's strongest claim is that architectural orchestration is as critical as prompt engineering for therapeutic AI. Because therapy is staged and process-driven, distributing stages across specialized agents with shared memory and grounded expertise produced measurably more natural interactions than identical content delivered by a single unstructured agent. The consistent ordering Alpha > Beta > Gamma across most metrics suggests structural scaffolding shapes perceived interaction quality even when users are unaware of system design. At the same time, the authors concede that gains in naturalness did not extend to emotional attunement dimensions, implying that architecture and prompt/content design may govern different facets of the therapeutic alliance.

Limitations and open questions

The authors identify several constraints. The eight-day window captures only early impressions, leaving unresolved whether perceived naturalness translates into sustained engagement or clinical benefit. Single-item survey measures limit construct validity; validated multi-item instruments such as the Working Alliance Inventory were not used. The sample was Persian-speaking, highly educated (93.9% holding at least a Bachelor's degree), and recruited partly through the authors' networks, constraining generalizability; the English-authored architecture also raises cross-linguistic questions. Therapeutic outcomes and skill acquisition were not directly assessed, so subjective interaction quality remains unlinked to objective clinical measures. Finally, the collected interaction data could support preference-based alignment (e.g., PPO or DPO), though this was not performed.

Conclusion

Holding content constant across conditions, this RCT demonstrates that multi-agent, memory-augmented architectures with explicit state management yield significantly more natural, human-like therapeutic dialogue than single-agent or unguided alternatives, alongside non-significant improvements in relational metrics. The results indicate that advancing therapeutic chatbots requires attention to system-level orchestration—in addition to capable base models and refined prompts—to support structured therapeutic progression.