PatientHub: Modular Simulation Framework
- PatientHub is a unified, modular framework that standardizes the design and benchmarking of LLM-based patient simulations across mental health and counseling applications.
- It decomposes patient simulations into clear abstractions—agents, evaluators, generators, and events—to enable consistent interaction modeling and evaluation.
- By consolidating heterogeneous protocols and metrics, the framework facilitates reproducible experiments and fair cross-method comparisons in simulation research.
PatientHub is a unified, modular framework for designing, running, and evaluating large-language-model-based patient simulations. It was introduced to address a specific methodological problem in patient-simulation research: prior work had become fragmented across incompatible data formats, profile schemas, prompts, orchestration assumptions, and evaluation metrics, making reproduction, reuse, and fair comparison unnecessarily difficult. In response, PatientHub standardizes the definition, composition, and deployment of simulated patients, while providing a common substrate for cross-method benchmarking, custom evaluation, and rapid prototyping of new simulators (Sabour et al., 12 Feb 2026).
1. Research context and intended scope
PatientHub emerged in a research setting where simulated patients had become useful for counselor training, therapeutic assessment, and synthetic dialogue generation, but where the literature varied substantially in what counted as a “patient,” how interactions unfolded, and how outputs were judged. Existing systems spanned counseling-style multi-turn conversations, interview-style psychiatric assessment, single-session and multi-session protocols, single-agent prompting, multi-agent prompting, and fine-tuned models. Evaluation was equally heterogeneous, ranging from automatic scoring to expert judgments and bespoke task-specific heuristics (Sabour et al., 12 Feb 2026).
The framework is positioned in mental-health and counseling research, but its abstractions are deliberately broader. It uses “client” and “patient” interchangeably to cover both clinically diagnosed patients and help-seeking individuals without a formal diagnosis. This scope matters because it lets PatientHub host methods aimed at cognitive behavioral therapy, motivational interviewing, general psychotherapy, psychiatric diagnosis, clinical diagnosis, and communication training within one infrastructure (Sabour et al., 12 Feb 2026).
A central motivation is that simulation is operationally attractive when real patient data are privacy-sensitive, scarce, or risky to use for iterative model development. PatientHub therefore treats simulation not as a single model family, but as an infrastructure problem: how to specify personas, run interactions, log sessions, and evaluate outputs under shared protocols (Sabour et al., 12 Feb 2026).
2. Core abstractions and system design
PatientHub standardizes simulation around four abstractions: agents, evaluators, generators, and events. This decomposition separates participants, workflow, and scoring, allowing heterogeneous methods to coexist without being forced into one prompt template or one dialogue style (Sabour et al., 12 Feb 2026).
At the agent level, the framework includes clients and therapists. A client agent is defined mainly by a persona profile, a response-generation specification, and, optionally, an internal latent state that can evolve with dialogue context. The persona may include demographic attributes, medical history, and behavioral tendencies. Therapist agents follow the same high-level interface and can represent either human trainees or LLM-based supporters (Sabour et al., 12 Feb 2026).
The evaluator abstraction provides a standardized LLM-as-a-judge interface. PatientHub supports four evaluation paradigms: Binary, Scalar, Categorical, and Extraction. Binary evaluation covers deterministic pass/fail criteria, such as whether self-harm was mentioned. Scalar evaluation covers Likert-style scoring, such as consistency from 1 to 10. Categorical evaluation returns explicit labels such as good, neutral, or bad. Extraction identifies concrete evidence, such as turns that contradict a profile. The paper emphasizes Extraction as especially useful because it supports actionable debugging rather than only assigning a score (Sabour et al., 12 Feb 2026).
Generators are used to create structured simulation inputs, especially synthetic profiles. This is important because several prior methods relied on ad hoc profile synthesis procedures. In PatientHub, profile generation becomes a first-class modular step, which makes simulation instances reproducible and easier to compare across methods (Sabour et al., 12 Feb 2026).
Events define how interactions unfold. PatientHub represents interaction protocols as graphs specifying turn-taking, session structure, and termination criteria. A therapy-session event, for example, may have the therapist initiate the interaction, the client respond, and a moderator enforce a turn cap or detect whether the session should stop. This event abstraction is what lets the same framework support multi-turn counseling, interview-style assessment, and multi-session simulations (Sabour et al., 12 Feb 2026).
The implementation stack is explicit. PatientHub is written in Python; Hydra is used for configuration management; Burr is used to represent and execute events as a graph; character data are stored as JSON records; prompts are stored as YAML templates rendered with Jinja; model calls go through LiteLLM; and structured outputs can be validated against Pydantic schemas when the underlying model supports them (Sabour et al., 12 Feb 2026).
3. Coverage of prior simulation methods
A major claim of PatientHub is that it is not tied to one simulator family. The framework includes implementations of 11 representative simulators covering multiple domains, tasks, and interaction regimes (Sabour et al., 12 Feb 2026).
| Method | Domain / task | Interaction style |
|---|---|---|
| PATIENT- | CBT, counselor training | single-agent prompting, multi-turn |
| Roleplay-doh | general psychotherapy, counselor training | multi-agent prompting, multi-turn |
| Eeyore | general psychotherapy, counselor training | SFT + DPO, multi-turn |
| ClientCAST | general psychotherapy, psychotherapy evaluation | single-agent prompting, multi-turn |
| AnnaAgent | general psychotherapy, realistic simulation | multi-agent prompting, multi-session |
| SimPatient | motivational interviewing, counselor training | multi-agent prompting, multi-session |
| Consistent MI | motivational interviewing, consistent simulation | multi-agent prompting, multi-turn |
| PSYCHE | general psychotherapy, psychiatric diagnosis | single-agent prompting, interview-style |
| SAPS | clinical medicine, clinical diagnosis | multi-agent prompting, interview-style |
| TalkDep | general psychotherapy, depression diagnosis | single-agent prompting, interview-style |
| Adaptive-VP | clinical medicine, communication training | multi-agent prompting, multi-turn |
This breadth is methodologically important. PatientHub can host simple prompt-based simulators such as PATIENT-, multi-agent systems such as AnnaAgent, and fine-tuned models such as Eeyore under the same orchestration and evaluation pipeline. A practical consequence is that differences in reported performance can be attributed more cleanly to simulator design rather than to incompatible prompts, logging, or scoring scripts (Sabour et al., 12 Feb 2026).
The framework also includes two newly prototyped Patient- variants intended to demonstrate extensibility. -COT adds a turn-level internal reasoning step involving emotion, trust, and response planning, with trust modeled across five levels, –. -Doh adds self-refinement: it drafts a response, evaluates it against existing dimensions such as consistency, realism, and pedagogical utility, and rewrites if necessary while preserving the original essence (Sabour et al., 12 Feb 2026).
4. Standardized workflow and evaluation methodology
PatientHub’s workflow begins by creating or loading structured profiles, potentially via generators. Client and therapist agents are then instantiated from a chosen method implementation and associated prompts or models. An event graph specifies how they interact. The conversation is executed through the orchestrator, all turns are logged in a unified format, and evaluators can score the transcript during runtime or offline. Because logs are standardized, the same conversation can be rescored under different rubrics or judge models (Sabour et al., 12 Feb 2026).
The framework’s evaluation methodology has two major uses: profile-quality assessment and simulator benchmarking. For synthesized personas, the authors first unified the fields required across selected simulation methods and built a common character generator. They then sampled 20 conversations from ESC, a human emotional-support dataset, and generated 20 corresponding profiles. These conversations covered CBT-suitable topics such as alcohol abuse and ongoing depression (Sabour et al., 12 Feb 2026).
Profile quality was then assessed with an Extraction-style evaluator using four dimensions: Completeness, Coherence, Realism, and Pedagogical Utility. Extracted feedback was used to prompt GPT-4o to revise profiles, after which revisions were manually verified. The paper presents a case in which evaluator feedback identified weak coping-strategy specification and an underexplained tension between a life-coach background and self-doubt; the revised profile explicitly reframed this as impostor syndrome and maladaptive drafting-and-deleting behavior (Sabour et al., 12 Feb 2026).
For simulator benchmarking, the framework uses a shared therapy-session event capped at 15 turns, with a moderator reminding the therapist to wrap up at turn 13. Five client simulators—PATIENT-, Roleplay-doh, Eeyore, ClientCAST, and AnnaAgent—were compared under two therapist conditions: a CBT therapist and a deliberately bad therapist. The CBT therapist prompt instructed the model to roleplay as a professional CBT therapist, progress through reflection, cognitive-model identification, Socratic questioning, cognitive restructuring, and possible behavioral activation, remain concise, never diagnose, handle safety crises explicitly, begin with “Hello, what’s on your mind today?”, and keep replies under 3 sentences. The bad therapist prompt instead instructed the model to be dismissive, invalidating, cold, and unhelpful (Sabour et al., 12 Feb 2026).
The benchmarking rubric has three top-level dimensions with nine sub-aspects. Consistency includes Factual Consistency, Self-Consistency, and Psychological Alignment. Realism includes Naturalness, Emotional Depth, Appropriate Resistance, and Absence of Self-Curing. Pedagogical Utility includes Feedback Quality and Learning Opportunities. A Scalar evaluator scores each aspect on a 5-point Likert scale. Additional reporting includes average response length, average number of generated tokens per turn, and average API cost per session (Sabour et al., 12 Feb 2026).
The experimental setup is standardized. The domain is CBT. All agents except Eeyore use GPT-4o as the backbone model; Eeyore uses the authors’ released fine-tuned Llama-3.1-8B model. All methods share the same inference hyperparameters: Temperature , max tokens (Sabour et al., 12 Feb 2026).
5. Empirical findings and extensibility
The most prominent empirical finding is that standardized benchmarking does not reward architectural complexity automatically. A relatively simple prompt-based method, PATIENT-0, achieved the highest consistency and pedagogical utility across both therapist conditions. Under the CBT therapist, it scored 4.75 on Factual Consistency, 4.45 on Self-Consistency, 4.45 on Psychological Alignment, 4.00 on Feedback Quality, and 4.05 on Learning Opportunities (Sabour et al., 12 Feb 2026).
Other methods exhibited different strengths. AnnaAgent was strongest on Naturalness in targeted therapy, which is consistent with its design emphasis on realistic multi-session counseling with long-term memory and dynamic emotional or cognitive evolution. Eeyore remained competitive despite shorter outputs and a smaller prompt footprint, which suggests that a fine-tuned model can remain viable even when compared against runtime-prompted GPT-4o-based simulators (Sabour et al., 12 Feb 2026).
The bad-therapist condition exposed a more diagnostic pattern. Appropriate Resistance increased across methods, which indicates more realistic pushback against invalidating therapy. At the same time, pedagogical utility declined sharply, suggesting that realistic reaction to poor therapy is not the same thing as usefulness for training. This distinction becomes a broader methodological claim of the paper: realism and pedagogical value are not identical objectives and should not be collapsed into one score (Sabour et al., 12 Feb 2026).
The extensibility demonstrations reinforce this point. For the two new Patient-1 variants, the authors generated 80 additional conversations, described as 2 clients 3 4 therapists 5 6 profiles from Patient-7. 8-COT improved Factual Consistency in both therapist settings and produced considerably shorter responses, which the authors interpret as trust-sensitive moderation of disclosure. 9-Doh improved realism-related behavior, especially Appropriate Resistance and Absence of Self-Curing with the bad therapist, but did not consistently improve pedagogical utility (Sabour et al., 12 Feb 2026).
A broader implication is that PatientHub makes it easier to isolate what a new simulator variant actually changes. Because logging, orchestration, and evaluation are already standardized, modifications can be localized to prompting logic, internal state updates, or refinement loops rather than hidden inside a new end-to-end implementation (Sabour et al., 12 Feb 2026).
6. Limitations, ethics, and relation to the broader ecosystem
The empirical study is deliberately modest. It uses only 20 synthetic profiles and 280 conversation sessions overall. All reported metrics rely on GPT-4o as LLM judge, which introduces possible bias because GPT-4o also serves as the simulator backbone for most methods. The paper notes that LLM judgments may depend on the judge model and rubric phrasing and may correlate with surface features such as response length or formality rather than genuine clinical realism. It therefore recommends explicit reporting of judge models and rubrics, sensitivity analysis across judges or rubric variants, and validation by human experts where possible (Sabour et al., 12 Feb 2026).
Current demonstrations also have a domain bias. They focus on multi-turn counseling and do not yet support multi-party care teams, long-horizon follow-up, crisis intervention, or tool-augmented clinicians. Those scenarios would require richer abstractions for coordination and persistent state (Sabour et al., 12 Feb 2026).
The ethical discussion is cautious. PatientHub itself does not require real patient records, but users who supply their own data are responsible for consent, de-identification, legal compliance, and institutional review requirements. Even simulated interactions can produce harmful advice, inappropriate escalation, or privacy leakage. Automatically generated personas may also encode or amplify biases. The framework is therefore presented as research infrastructure that benefits from controlled use and supervision, not as an unevaluated direct-to-user therapeutic system (Sabour et al., 12 Feb 2026).
A plausible implication is that PatientHub occupies a different layer from patient-facing care-delivery platforms such as the TreC-based prescription-app framework (Osmani et al., 2017), the WhatsApp-mediated remote-care EHR CARE-link (Adjei et al., 3 Jun 2026), or local multimodal model hubs such as MedFoundationHub (Li et al., 28 Aug 2025). Those systems focus on care coordination, patient engagement, or secure clinical model deployment; PatientHub focuses instead on simulation infrastructure for training, assessment, and benchmarking. The distinction is important because “hub” in this context does not denote a clinical portal or a hospital operations platform, but a reproducible research substrate for patient-centered dialogue methods.
Its lasting significance lies in that infrastructural role. PatientHub is not introduced as a single superior simulator. It is a common language for specifying simulated patients, therapist counterparts, profile generators, event-driven dialogue protocols, prompt templates, unified logging, and rubric-based evaluation. By consolidating heterogeneous simulators into one modular and reproducible pipeline, it lowers the barrier to method development and makes cross-method, cross-model benchmarking materially easier (Sabour et al., 12 Feb 2026).