Optimal Question Selection from a Large Question Bank for Clinical Field Recovery in Conversational Psychiatric Intake

Abstract: Psychiatric intake is a sequential, high-stakes information-gathering process in which clinicians must decide what to ask, in what order, and how to interpret incomplete or ambiguous responses under limited time. Despite growing interest in conversational AI for healthcare, there is still limited infrastructure for conversational AI in this application. Accordingly, we formulate this task as a question-selection problem with clinically grounded questions, known target information, and controllable patient difficulty. We also introduce a task-specific question-selection benchmark based on a bank of 655 clinician-authored intake questions and corresponding synthetic patient vignettes with 5 different behavioral conditions. In our evaluation, we compare random questioning, a clinical psychiatric intake form baseline, and an LLM-guided adaptive policy across 300 interview sessions spanning four patients and five behavioral conditions. Across the benchmark, the clinically ordered fixed form substantially outperforms random questioning, and the LLM-guided policy achieves the strongest overall recovery. The advantage of adaptation grows sharply under patient behavior that is less amenable to field recovery, especially under guarded-concise conditions. These findings suggest that performance in conversational clinical systems depends not only on language understanding after information is disclosed, but also on whether the system reaches the right topics within a limited interaction budget. More broadly, the benchmark provides a controlled framework for studying how clinical structure and adaptive follow-up contribute to information recovery in interactive clinical machine learning.

• The paper introduces a controlled benchmark for sequential question selection in conversational psychiatric intake.
• It evaluates LLM-guided, form-based, and random policies, with the LLM-guided approach achieving 95.4% ± 8.1% mean accuracy.
• Results underscore the importance of adaptive, efficient questioning to robustly recover clinical information under challenging patient behaviors.

Optimal Question Selection for Psychiatric Intake: A Benchmark Study

Introduction

The paper "Optimal Question Selection from a Large Question Bank for Clinical Field Recovery in Conversational Psychiatric Intake" (2604.22067) addresses the machine learning problem of sequential question selection for psychiatric intake interviews. Psychiatric intake, unlike structured data collection in other medical domains, is characterized by its adaptive nature: clinicians must dynamically decide which questions to ask based on incomplete, subjective, and often ambiguous patient responses, all within stringent time constraints. The study formalizes this challenge as an information recovery task, introducing a rigorously constructed benchmark based on a large, clinically vetted question bank and a set of synthetic patient profiles with controllable behavioral parameters.

Benchmark Design and Problem Formulation

The core contribution is the development of a controlled, clinically grounded benchmark for evaluating question selection policies in conversational psychiatric intake.

A bank of 655 clinician-authored intake questions is constructed through participatory design with practicing psychiatrists. Each question is mapped to one or more of 10 binary target fields, reflecting discrete, high-value clinical outcomes such as presence of suicidality, substance use, or prior psychiatric treatment. Patient behavior is simulated using GPT-4o-based agents, with each synthetic patient vignette instantiated under five behavioral conditions spanning axes of cooperativeness (forthcoming vs. guarded) and verbosity (talkative vs. concise), in addition to a default setting. Ground-truth field values are known for each synthetic vignette, enabling objective evaluation.

Figure 1: The benchmark framework spans clinical question bank curation, synthetic patient instantiation under five behavioral conditions, and scoring of conversational policies over fixed-budget interviews.

The psychiatric intake process is modeled as a sequential decision problem: at each conversational turn $t$, the agent selects a question $q_t$ from the pool of unanswered items, observes the patient (agent) response, and the process repeats up to a fixed turn budget ($T=20$). The objective for any policy $\pi$ is to maximize mean recovery accuracy over all $K=10$ target fields:

$$\max_{\pi \in \Pi_T}\ \mathbb{E}\left[\frac{1}{K}\sum_{k=1}^{K}\mathbf{1}\!\left[\hat{g}_{\pi}(f_k)=g^{\ast}(f_k)\right]\right]$$

The evaluation pipeline involves an automated LLM-based judge for binary field extraction as well as manual review for hallucination detection and process failures.

Figure 2: Sessions are scored by both automated LLM-based post-hoc extraction and manual review for hallucination and unsupported inference.

Question-Selection Policies

The study implements and evaluates three distinct question-selection strategies:

• Random baseline: Uniform random selection of unanswered questions at each turn, providing a lower bound on policy efficacy.
• Form-based baseline: Fixed, clinically ordered sequence reflecting standard psychiatric intake protocols (e.g., SCID, MINI), with early emphasis on safety/risk screening, per-domain budget allocation, and intra-domain prioritization.
• LLM-guided policy: At each turn, an LLM is presented with the conversation history, unrecovered fields, and a random sample of 40 candidate questions; it selects the next question deemed most informative for target recovery, adapting dynamically to dialogue context.

Experimental Results

Comprehensive evaluation across 300 interview sessions (3 policies × 4 patients × 5 behaviors × 5 runs per cell) yields several key findings.

Overall Performance:

The LLM-guided policy achieves the highest overall mean accuracy (95.4% ± 8.1%), with the form-based baseline at 84.8% ± 15.8% and the random baseline at 51.7% ± 19.6%. Precision is uniformly high for all strategies, but coverage and accuracy diverge sharply, highlighting that bottlenecks arise from question selection rather than information extraction post-disclosure.

Behavioral Condition Robustness:

While the form-based and LLM-guided policies maintain high accuracy under default and cooperative patient behaviors, the LLM-guided policy exhibits substantial robustness under the most challenging (guarded + concise) condition, retaining 89% mean accuracy compared to dramatic degradation for form-based (59.5%) and random (26.5%).

Figure 3: Mean accuracy remains robust for the LLM-guided and form-based policies except under the hardest guarded + concise condition, where LLM-guided selection is notably more resilient.

Figure 4: Across all behavioral variants, LLM-guided outperforms form-based and random, with the largest gap manifesting under guarded + concise conditions.

Per-session variability is also minimized for the LLM-guided policy in difficult conditions, pointing to increased reliability.

Figure 5: In the hardest behavioral condition (guarded + concise), the LLM-guided policy demonstrates both superior accuracy and reduced variance compared to baselines.

Conversational Efficiency:

Analysis of accuracy versus conversational word count indicates that the LLM-guided policy not only achieves the highest recovery but does so efficiently in terms of dialogue cost; in contrast, random policy achieves poor informational yield despite using the most words.

Figure 6: The LLM-guided selection attains near-maximal recovery per unit of conversational effort, combining high accuracy with cost-effective questioning.

Patient-Level Heterogeneity:

Evaluation across patient cases with differing diagnostic complexity shows consistent superiority of LLM-guided selection, with the form-based policy competitive only for cooperative or verbose profiles.

Implications

This work substantiates several critical implications for machine learning in conversational healthcare domains:

• Question selection is an independent lever for performance: Beyond language understanding, the ability to prioritize, adapt, and re-target questions in real time is central to efficient recovery of clinically critical information.
• Clinical structure explains most gain over naïve approaches: Fixed, evidence-based ordering resolves much of the inefficiency seen in random questioning, but is insufficient in the presence of challenging patient behaviors.
• Adaptation is essential for robustness: The LLM-guided policy's advantage is magnified for terse, guarded, or otherwise uncooperative patient profiles, mirroring real-world high-risk clinical scenarios.
• Synthetic benchmarks are essential for controlled evaluation: The framework enables isolation of behavioral difficulty, policy adaptation, and other variables that are difficult to disentangle in live clinical data.

On the practical front, these findings argue that future AI-driven psychiatric intake support systems should incorporate both strong clinical structuring and real-time adaptive mechanisms. The approach can be extended to richer patient behavior models, more granular targets, and ultimately serves as a foundation for clinician-training and AI-evaluation platforms using synthetic but clinically realistic interactions.

Limitations and Future Directions

The benchmark relies on synthetic patients, constraining immediate clinical generalizability. The question bank, while grounded in institutional practice, reflects a finite abstraction (10 binary targets). Evaluation does not include a human clinician baseline or consider unconstrained free-form question generation, which may be more effective but is currently less controllable.

Future work should expand to broader diagnostic constructs, larger and more diverse patient pools, and inclusion of clinical end-users for external validation. Comparing constrained bank-based adaptive policies to both human clinicians and fully open-ended LLM strategies constitutes a logical progression, as does the integration of these benchmarks into simulation-based training environments and real-world clinical workflows.

Conclusion

This paper advances the formalization and empirical evaluation of optimal question selection in conversational psychiatric intake. The presented benchmark and results systematically delineate the roles of clinical structure and adaptive questioning in maximizing information recovery under interactive constraints. The LLM-guided policy exhibits strong robustness, especially under adverse patient behavioral conditions, underscoring the importance of adaptability for clinical AI agents. The methodological paradigm established here is extensible to a broad class of interactive healthcare tasks requiring strategic information gathering and paves the way for further integration of ML-based support systems in high-stakes, real-world clinical settings.