Question-Selection Benchmark

• Question-Selection Benchmark is a framework for sequentially selecting clinically vetted questions to recover psychiatric intake targets within a strict 20-turn dialogue budget.
• It leverages 655 clinician-authored questions, 10 binary target fields, and synthetic patient vignettes under varied behavioral conditions to simulate and evaluate clinical information recovery.
• Empirical results demonstrate that LLM-guided adaptive policies substantially outperform fixed and random questioning by enhancing accuracy and efficiency in eliciting critical patient data.

The Question-Selection Benchmark is a task-specific benchmark for budgeted information recovery in conversational psychiatric intake. It formulates psychiatric intake as a sequential question-selection problem with clinically grounded questions, known target information, and controllable patient difficulty, using a vetted bank of 655 clinician-authored intake questions, 10 binary target fields, and synthetic patient vignettes evaluated under five behavioral conditions (Gui et al., 23 Apr 2026). In this formulation, performance depends not only on language understanding after disclosure, but also on whether the system reaches the right topics within a fixed interaction budget of $T = 20$ turns. Across 300 interview sessions, the benchmark compares random questioning, a clinically ordered fixed form, and an LLM-guided adaptive policy; the clinically ordered form substantially outperforms random questioning, and the LLM-guided policy achieves the strongest overall recovery, with its advantage growing sharply under guarded–concise patient behavior (Gui et al., 23 Apr 2026).

1. Clinical motivation and benchmark scope

Psychiatric intake is presented as a sequential, adaptive information-gathering process under uncertainty. Clinicians must decide what to ask next based on partial, subjective, or ambiguous responses, and missing the right question can mean missing critical information such as suicidal ideation, substance use, prior treatment, or relevant medical and social context. The benchmark therefore centers on field recovery, defined as recovery of a clinically meaningful binary field when the transcript contains sufficient evidence to correctly infer its true value at the end of the session, and on the interaction budget, defined as a maximum of 20 questions with early stopping if all target fields are recovered (Gui et al., 23 Apr 2026).

The benchmark’s target information consists of $K = 10$ binary evaluation targets defined by clinician input to reflect safety, risk, symptom characterization, and context necessary for diagnosis and treatment planning. These fields are evaluated against authored vignettes whose ground-truth values are known a priori. This design yields exact scoring of recovery rather than approximate annotation. Behavioral difficulty is controlled through five response styles: Default, Forthcoming–Talkative, Forthcoming–Concise, Guarded–Talkative, and Guarded–Concise. Forthcoming styles are amenable to single-pass screening, while guarded styles require follow-ups or reframing; talkativeness increases incidental disclosure, whereas conciseness reduces incidental mentions (Gui et al., 23 Apr 2026).

This framing makes the benchmark a study of interactive clinical machine learning rather than merely a study of post-hoc transcript interpretation. The central claim is that conversational clinical systems must be evaluated not only by whether they can interpret disclosed information, but also by whether they can allocate a limited conversational budget across clinically meaningful topics and adapt as the dialogue evolves (Gui et al., 23 Apr 2026).

2. Benchmark construction and controlled difficulty

The question bank contains $N = 655$ clinician-authored intake questions, reduced from 812 raw items and sourced from a large Department of Psychiatry. Its domains include psychiatric history, substance use, medical history, family history, social context, and risk assessment. The benchmark explicitly restricts actions to real intake questions in order to preserve clinical validity and safety, and the native spreadsheet order was not treated as a meaningful interview sequence (Gui et al., 23 Apr 2026).

The patient side of the benchmark is built from synthetic vignettes. Four evaluation patients were selected from 12 authored profiles to maximize variation in diagnosis, severity, communication style, and field distribution; diagnoses include MDD, GAD, PTSD, with one severe MDD case including current suicidal ideation. The patients are labeled Eval001–Eval004. Each profile includes a structured vignette, ground-truth values for all 10 binary fields, and a communication style. GPT-4o-family models simulate patient responses conditioned on the vignette, behavioral condition, and dialogue context (Gui et al., 23 Apr 2026).

The five behavioral conditions operationalize recoverability. In Default, the patient’s native communication style is unaltered. In Forthcoming–Talkative, answers are direct and elaborative; in Forthcoming–Concise, direct but brief. In Guarded–Talkative, the patient initially defers or partially answers before yielding on follow-up and elaborates when answering. In Guarded–Concise, the patient both defers or partially answers and remains brief, producing low incidental disclosure and the highest difficulty for recovery (Gui et al., 23 Apr 2026).

This controlled construction separates two variables that are often conflated in conversational evaluation: what information exists in the patient state, and how readily that information can be elicited. A plausible implication is that the benchmark isolates acquisition difficulty independently from downstream extraction difficulty.

3. Formal task formulation and scoring protocol

The benchmark casts intake as budgeted information recovery via sequential question selection from $Q$ over $K = 10$ target fields with $T = 20$ turns. The latent state includes the patient vignette, ground-truth fields, and behavior, while the observable state is the dialogue history

$H_t = \{(q_1, r_1), \ldots, (q_{t-1}, r_{t-1})\}.$

The action space is the set of unanswered questions from the 655-item bank, the observation is the patient response generated by the GPT-4o simulator, and the horizon is $B = T = 20$ with early stopping when all fields are recovered (Gui et al., 23 Apr 2026).

The paper states the end-of-session objective as

$$\max_{\pi \in \Pi_T}\ \mathbb{E}\left[\frac{1}{K}\sum_{k=1}^{K}\mathbf{1}\!\left[\hat{g}_{\pi}(f_k)=g^{\ast}(f_k)\right]\right].$$

It also gives a budgeted reward-maximization form,

$$\pi^* = \arg\max_{\pi} \ \mathbb{E}_{\tau\sim \pi}\Bigg[\sum_{t=1}^{B} R(s_t, a_t, o_t)\Bigg],$$

with $K = 10$0 and reward tied to incremental field recovery. The field recovery metric is

$K = 10$1

No explicit belief-state update is defined; instead, a lightweight in-loop heuristic tracks unrecovered fields via conservative keyword matching, while final scoring relies on a post-session LLM judge (Gui et al., 23 Apr 2026).

Evaluation uses a full factorial design: 3 policies $K = 10$2 4 patients $K = 10$3 5 behavioral conditions $K = 10$4 5 runs, for 300 interview sessions. Stopping occurs at 20 turns or when all 10 fields are recovered. Scoring is automated by a post-session GPT-4o judge that returns, per field, a mentioned flag, extracted value in $K = 10$5, confidence score, and evidence quote; extractions with confidence $K = 10$6 are discarded. Reported transcript-level metrics are Accuracy, Coverage, Precision, F1 score, False label rate, and Efficiency, with results summarized as mean $K = 10$7 standard deviation across sessions. No formal hypothesis tests or $K = 10$8-values are reported (Gui et al., 23 Apr 2026).

Manual review separately flags patient-agent hallucinations, defined as contradictions or unsupported inventions relative to the vignette, and algorithm hallucinations, defined as unsupported inferences not present in the transcript. These are reported separately from the quantitative metrics (Gui et al., 23 Apr 2026).

4. Policies, baselines, and empirical results

The benchmark evaluates three policies. The random baseline samples an unanswered question uniformly from the bank at each turn, without repetition and without structure or adaptation. The clinically ordered fixed-form baseline is a deterministic, clinician-informed interview order that prioritizes safety early, then psychiatric history, medical history, and social or family context; it incorporates domain priority, per-domain turn caps, subdomain caps, variable-key grouping to avoid redundant variants, and binary-before-open ordering. The LLM-guided adaptive policy samples up to 40 unanswered candidate questions uniformly at random from the bank, presents them together with the conversation history and the list of unrecovered fields to GPT-4o, and has the model select exactly one candidate judged most likely to recover new information (Gui et al., 23 Apr 2026).

Policy	Accuracy	Mean turns
Random	51.7% ± 19.6	20.0
Form-based	84.8% ± 15.8	20.0
LLM-guided	95.4% ± 8.1	18.3 ± 3.1

Across the 300 sessions, random questioning yields Coverage 52.4% ± 19.5, Precision 98.4% ± 6.9, F1 66.1% ± 18.8, False label rate 1.6% ± 6.9, Efficiency 0.0090 ± 0.0040, and Mean words 645.2 ± 318.7. The form-based policy yields Coverage 85.5% ± 15.8, Precision 99.2% ± 2.8, F1 90.9% ± 11.5, False label rate 0.8% ± 2.8, Efficiency 0.0197 ± 0.0067, and Mean words 489.7 ± 213.3. The LLM-guided policy yields Coverage 96.0% ± 7.8, Precision 99.4% ± 2.5, F1 97.5% ± 4.6, False label rate 0.6% ± 2.5, Efficiency 0.0207 ± 0.0097, and Mean words 589.8 ± 309.9 (Gui et al., 23 Apr 2026).

The comparative gains are explicit. Fixed-form improves over random by +33.1 percentage points in accuracy, from 51.7% to 84.8%, and the paper attributes most of this gain to clinical structure. LLM-guided improves over fixed-form by +10.6 points, from 84.8% to 95.4%, and the paper attributes this increment to adaptation (Gui et al., 23 Apr 2026).

Behavioral-condition analysis is central. The LLM-guided policy is best in every condition, ranging from 89.0% ± 11.2 under Guarded–Concise to 99.0% ± 3.1 under Default. The fixed form remains strong in non-guarded settings, at approximately 92–93%, but collapses under Guarded–Concise to 59.5% ± 18.2. Random ranges from 26.5% ± 13.5 in Guarded–Concise to 64.0% ± 14.3 in more amenable conditions. Under Guarded–Concise, the LLM–form gap expands to 29 points, approximately 89% versus approximately 60% (Gui et al., 23 Apr 2026).

Patient-level heterogeneity is also reported. For Eval001 (MDD+GAD), accuracy is 47.2% ± 19.0 for random, 80.8% ± 16.1 for form, and 94.4% ± 8.2 for LLM. For Eval002 (MDD+SI), the figures are 55.6% ± 21.8, 82.4% ± 14.5, and 96.0% ± 8.2. For Eval003 (GAD), they are 45.2% ± 16.6, 83.2% ± 17.3, and 94.0% ± 9.6. For Eval004 (PTSD), they are 58.8% ± 18.6, 92.8% ± 13.1, and 97.2% ± 6.1 (Gui et al., 23 Apr 2026).

5. Interpretation, clinical significance, and observed failure patterns

The benchmark’s principal interpretive result is that all strategies maintain uniformly high precision, approximately 98–99%, while coverage and accuracy drive the differences across policies. The paper therefore characterizes the bottleneck as elicitation rather than interpretation: once information is present in the transcript, systems mislabel relatively rarely, but non-adaptive policies often fail to reach the right topics within the budget (Gui et al., 23 Apr 2026).

The clinical fixed form performs well because it reaches high-yield domains early, imposes a rational sequence under budget, and avoids redundant screening variants. The adaptive policy adds a further advantage because it can condition on evolving dialogue state, unexpected disclosures, partial answers, and local topic shifts; it can re-ask or reframe when patients defer or partially answer, pivot to better-targeted follow-ups, and exploit incidental disclosures intelligently without wasting turns on low-yield topics (Gui et al., 23 Apr 2026).

The paper’s illustrative Guarded–Concise example makes this distinction concrete. A fixed form may screen for self-harm, receive a brief answer such as “Not really,” and move on, leaving critical fields unrecovered. The adaptive policy instead follows with a targeted reframing—asking whether there have been moments this week when the patient felt better off dead or had fleeting thoughts of self-harm—and thereby recovers suicidal ideation. A subsequent adaptive question about recent stressors can recover social context and symptom exacerbation under the same budget (Gui et al., 23 Apr 2026).

This suggests that benchmark difficulty is not reducible to linguistic opacity alone. It is partly a property of sequential control: different response styles change the value of follow-up, reframing, and topical persistence. In that sense, the benchmark operationalizes conversational competence as a topic-allocation problem under severe budget constraints.

6. Limitations, safety considerations, and relation to broader benchmark research

The benchmark’s limitations are explicit. Results reflect benchmark performance rather than direct clinical efficacy because the patients are synthetic; the vignettes and simulated behavior may not capture the full variability of real patients. The evaluation set comprises four patients selected from 12 authored profiles and only 10 binary targets, emphasizing controlled comparison rather than population-level generalization. The patient simulator, selector, and judge all rely on GPT-4o, so shared biases could affect both policy behavior and evaluation. There is no human clinician baseline, and the bank’s clinical vetting mitigates unsafe or inappropriate questioning but does not remove the need for rigorous governance and clinician oversight in deployment (Gui et al., 23 Apr 2026).

Reproducibility is partial. The paper reports the full-factorial protocol, turn budget, temperatures, and metrics, but it does not report public release of data or code, nor licensing details. Reproduction would require access to the vetted question bank, authored vignettes, and the evaluation pipeline, including the LLM judge and prompts. The framework is nevertheless presented as extensible to additional patients, targets, policies, and behaviors, with possible future work including unconstrained human-led interviews, clinician-in-the-loop evaluation, and additional safety or robustness checks (Gui et al., 23 Apr 2026).

In broader benchmark research, the term question selection names several distinct evaluation problems. In SelQA, it denotes answer sentence selection and answer triggering within open-domain selection-based question answering rather than sequential conversational acquisition (Jurczyk et al., 2016). In Scales++ and in the multiple-regression formulation of efficient benchmarking, it denotes selecting a small, representative subset of benchmark items that predicts full-benchmark performance with high fidelity and low compute cost (Bean et al., 30 Oct 2025); (Bowyer et al., 25 May 2026). In computerized adaptive testing, question selection is the policy that chooses the next item based on current proficiency estimates, and recent work studies how bi-level optimization can amplify selection bias and how debiasing modules such as Cross-Attribute Examinee Retrieval and Selective Mixup-based Regularization affect worst-group accuracy and fairness (Tian et al., 19 Nov 2025). Clinical reliability benchmarks such as SCARE, by contrast, treat question answerability classification as a post-hoc safety layer for EHR question answering, coupling selection or triage with SQL verification and correction (Lee et al., 13 Nov 2025).

These neighboring literatures clarify the specificity of the psychiatric intake benchmark. It is neither a sentence-selection corpus, nor a tiny-benchmark subset-selection method, nor a post-hoc answerability classifier. Its distinctive contribution is a controlled framework in which clinically vetted question actions, known latent targets, controllable patient behavior, and a fixed conversational budget jointly define a measurable information-recovery problem (Gui et al., 23 Apr 2026).