---
title: Question-Selection Benchmark
url: https://www.emergentmind.com/topics/question-selection-benchmark
type: topic
---

# Question-Selection Benchmark

The Question-Selection Benchmark is a task-specific benchmark for budgeted information recovery in conversational psychiatric intake. It formulates psychiatric intake as a sequential question-selection problem with clinically grounded questions, known target information, and controllable patient difficulty, using a vetted bank of 655 clinician-authored intake questions, 10 binary target fields, and synthetic patient vignettes evaluated under five behavioral conditions [2604.22067]. In this formulation, performance depends not only on language understanding after disclosure, but also on whether the system reaches the right topics within a fixed interaction budget of $T = 20$ turns. Across 300 interview sessions, the benchmark compares random questioning, a clinically ordered fixed form, and an LLM-guided adaptive policy; the clinically ordered form substantially outperforms random questioning, and the LLM-guided policy achieves the strongest overall recovery, with its advantage growing sharply under guarded–concise patient behavior [2604.22067].

## 1. Clinical motivation and benchmark scope

Psychiatric intake is presented as a sequential, adaptive information-gathering process under uncertainty. Clinicians must decide what to ask next based on partial, subjective, or ambiguous responses, and missing the right question can mean missing critical information such as suicidal ideation, substance use, prior treatment, or relevant medical and social context. The benchmark therefore centers on **field recovery**, defined as recovery of a clinically meaningful binary field when the transcript contains sufficient evidence to correctly infer its true value at the end of the session, and on the **interaction budget**, defined as a maximum of 20 questions with early stopping if all target fields are recovered [2604.22067].

The benchmark’s target information consists of $K = 10$ binary evaluation targets defined by clinician input to reflect safety, risk, symptom characterization, and context necessary for diagnosis and treatment planning. These fields are evaluated against authored vignettes whose ground-truth values are known a priori. This design yields exact scoring of recovery rather than approximate annotation. Behavioral difficulty is controlled through five response styles: Default, Forthcoming–Talkative, Forthcoming–Concise, Guarded–Talkative, and Guarded–Concise. Forthcoming styles are amenable to single-pass screening, while guarded styles require follow-ups or reframing; talkativeness increases incidental disclosure, whereas conciseness reduces incidental mentions [2604.22067].

This framing makes the benchmark a study of interactive clinical machine learning rather than merely a study of post-hoc transcript interpretation. The central claim is that conversational clinical systems must be evaluated not only by whether they can interpret disclosed information, but also by whether they can allocate a limited conversational budget across clinically meaningful topics and adapt as the dialogue evolves [2604.22067].

## 2. Benchmark construction and controlled difficulty

The question bank contains $N = 655$ clinician-authored intake questions, reduced from 812 raw items and sourced from a large Department of Psychiatry. Its domains include psychiatric history, substance use, medical history, family history, social context, and risk assessment. The benchmark explicitly restricts actions to real intake questions in order to preserve clinical validity and safety, and the native spreadsheet order was not treated as a meaningful interview sequence [2604.22067].

The patient side of the benchmark is built from synthetic vignettes. Four evaluation patients were selected from 12 authored profiles to maximize variation in diagnosis, severity, communication style, and field distribution; diagnoses include MDD, GAD, PTSD, with one severe MDD case including current suicidal ideation. The patients are labeled Eval001–Eval004. Each profile includes a structured vignette, ground-truth values for all 10 binary fields, and a communication style. GPT-4o-family models simulate patient responses conditioned on the vignette, behavioral condition, and dialogue context [2604.22067].

The five behavioral conditions operationalize recoverability. In Default, the patient’s native communication style is unaltered. In Forthcoming–Talkative, answers are direct and elaborative; in Forthcoming–Concise, direct but brief. In Guarded–Talkative, the patient initially defers or partially answers before yielding on follow-up and elaborates when answering. In Guarded–Concise, the patient both defers or partially answers and remains brief, producing low incidental disclosure and the highest difficulty for recovery [2604.22067].

This controlled construction separates two variables that are often conflated in conversational evaluation: what information exists in the patient state, and how readily that information can be elicited. A plausible implication is that the benchmark isolates acquisition difficulty independently from downstream extraction difficulty.

## 3. Formal task formulation and scoring protocol

The benchmark casts intake as budgeted information recovery via sequential question selection from $Q$ over $K = 10$ target fields with $T = 20$ turns. The latent state includes the patient vignette, ground-truth fields, and behavior, while the observable state is the dialogue history
$$
H_t = \{(q_1, r_1), \ldots, (q_{t-1}, r_{t-1})\}.
$$
The action space is the set of unanswered questions from the 655-item bank, the observation is the patient response generated by the GPT-4o simulator, and the horizon is $B = T = 20$ with early stopping when all fields are recovered [2604.22067].

The paper states the end-of-session objective as
$$
\max_{\pi \in \Pi_T}\ \mathbb{E}\left[\frac{1}{K}\sum_{k=1}^{K}\mathbf{1}\!\left[\hat{g}_{\pi}(f_k)=g^{\ast}(f_k)\right]\right].
$$
It also gives a budgeted reward-maximization form,
$$
\pi^* = \arg\max_{\pi} \ \mathbb{E}_{\tau\sim \pi}\Bigg[\sum_{t=1}^{B} R(s_t, a_t, o_t)\Bigg],
$$
with $B = 20$ and reward tied to incremental field recovery. The field recovery metric is
$$
\mathrm{RecoveryRate}=\frac{\sum_{i=1}^{K}\mathbf{1}[\text{field } i \text{ recovered}]}{K}.
$$
No explicit belief-state update is defined; instead, a lightweight in-loop heuristic tracks unrecovered fields via conservative keyword matching, while final scoring relies on a post-session LLM judge [2604.22067].

Evaluation uses a full factorial design: 3 policies $\times$ 4 patients $\times$ 5 behavioral conditions $\times$ 5 runs, for 300 interview sessions. Stopping occurs at 20 turns or when all 10 fields are recovered. Scoring is automated by a post-session GPT-4o judge that returns, per field, a mentioned flag, extracted value in $\{\text{true}, \text{false}, \text{null}\}$, confidence score, and evidence quote; extractions with confidence $< \tau = 0.8$ are discarded. Reported transcript-level metrics are Accuracy, Coverage, Precision, F1 score, False label rate, and Efficiency, with results summarized as mean $\pm$ standard deviation across sessions. No formal hypothesis tests or $p$-values are reported [2604.22067].

Manual review separately flags patient-agent hallucinations, defined as contradictions or unsupported inventions relative to the vignette, and algorithm hallucinations, defined as unsupported inferences not present in the transcript. These are reported separately from the quantitative metrics [2604.22067].

## 4. Policies, baselines, and empirical results

The benchmark evaluates three policies. The **random baseline** samples an unanswered question uniformly from the bank at each turn, without repetition and without structure or adaptation. The **clinically ordered fixed-form baseline** is a deterministic, clinician-informed interview order that prioritizes safety early, then psychiatric history, medical history, and social or family context; it incorporates domain priority, per-domain turn caps, subdomain caps, variable-key grouping to avoid redundant variants, and binary-before-open ordering. The **LLM-guided adaptive policy** samples up to 40 unanswered candidate questions uniformly at random from the bank, presents them together with the conversation history and the list of unrecovered fields to GPT-4o, and has the model select exactly one candidate judged most likely to recover new information [2604.22067].

| Policy | Accuracy | Mean turns |
|---|---:|---:|
| Random | 51.7% ± 19.6 | 20.0 |
| Form-based | 84.8% ± 15.8 | 20.0 |
| LLM-guided | 95.4% ± 8.1 | 18.3 ± 3.1 |

Across the 300 sessions, random questioning yields Coverage 52.4% ± 19.5, Precision 98.4% ± 6.9, F1 66.1% ± 18.8, False label rate 1.6% ± 6.9, Efficiency 0.0090 ± 0.0040, and Mean words 645.2 ± 318.7. The form-based policy yields Coverage 85.5% ± 15.8, Precision 99.2% ± 2.8, F1 90.9% ± 11.5, False label rate 0.8% ± 2.8, Efficiency 0.0197 ± 0.0067, and Mean words 489.7 ± 213.3. The LLM-guided policy yields Coverage 96.0% ± 7.8, Precision 99.4% ± 2.5, F1 97.5% ± 4.6, False label rate 0.6% ± 2.5, Efficiency 0.0207 ± 0.0097, and Mean words 589.8 ± 309.9 [2604.22067].

The comparative gains are explicit. Fixed-form improves over random by +33.1 percentage points in accuracy, from 51.7% to 84.8%, and the paper attributes most of this gain to clinical structure. LLM-guided improves over fixed-form by +10.6 points, from 84.8% to 95.4%, and the paper attributes this increment to adaptation [2604.22067].

Behavioral-condition analysis is central. The LLM-guided policy is best in every condition, ranging from 89.0% ± 11.2 under Guarded–Concise to 99.0% ± 3.1 under Default. The fixed form remains strong in non-guarded settings, at approximately 92–93%, but collapses under Guarded–Concise to 59.5% ± 18.2. Random ranges from 26.5% ± 13.5 in Guarded–Concise to 64.0% ± 14.3 in more amenable conditions. Under Guarded–Concise, the LLM–form gap expands to 29 points, approximately 89% versus approximately 60% [2604.22067].

Patient-level heterogeneity is also reported. For Eval001 (MDD+GAD), accuracy is 47.2% ± 19.0 for random, 80.8% ± 16.1 for form, and 94.4% ± 8.2 for LLM. For Eval002 (MDD+SI), the figures are 55.6% ± 21.8, 82.4% ± 14.5, and 96.0% ± 8.2. For Eval003 (GAD), they are 45.2% ± 16.6, 83.2% ± 17.3, and 94.0% ± 9.6. For Eval004 (PTSD), they are 58.8% ± 18.6, 92.8% ± 13.1, and 97.2% ± 6.1 [2604.22067].

## 5. Interpretation, clinical significance, and observed failure patterns

The benchmark’s principal interpretive result is that all strategies maintain uniformly high precision, approximately 98–99%, while coverage and accuracy drive the differences across policies. The paper therefore characterizes the bottleneck as **elicitation** rather than interpretation: once information is present in the transcript, systems mislabel relatively rarely, but non-adaptive policies often fail to reach the right topics within the budget [2604.22067].

The clinical fixed form performs well because it reaches high-yield domains early, imposes a rational sequence under budget, and avoids redundant screening variants. The adaptive policy adds a further advantage because it can condition on evolving dialogue state, unexpected disclosures, partial answers, and local topic shifts; it can re-ask or reframe when patients defer or partially answer, pivot to better-targeted follow-ups, and exploit incidental disclosures intelligently without wasting turns on low-yield topics [2604.22067].

The paper’s illustrative Guarded–Concise example makes this distinction concrete. A fixed form may screen for self-harm, receive a brief answer such as “Not really,” and move on, leaving critical fields unrecovered. The adaptive policy instead follows with a targeted reframing—asking whether there have been moments this week when the patient felt better off dead or had fleeting thoughts of self-harm—and thereby recovers suicidal ideation. A subsequent adaptive question about recent stressors can recover social context and symptom exacerbation under the same budget [2604.22067].

This suggests that benchmark difficulty is not reducible to linguistic opacity alone. It is partly a property of sequential control: different response styles change the value of follow-up, reframing, and topical persistence. In that sense, the benchmark operationalizes conversational competence as a topic-allocation problem under severe budget constraints.

## 6. Limitations, safety considerations, and relation to broader benchmark research

The benchmark’s limitations are explicit. Results reflect benchmark performance rather than direct clinical efficacy because the patients are synthetic; the vignettes and simulated behavior may not capture the full variability of real patients. The evaluation set comprises four patients selected from 12 authored profiles and only 10 binary targets, emphasizing controlled comparison rather than population-level generalization. The patient simulator, selector, and judge all rely on GPT-4o, so shared biases could affect both policy behavior and evaluation. There is no human clinician baseline, and the bank’s clinical vetting mitigates unsafe or inappropriate questioning but does not remove the need for rigorous governance and clinician oversight in deployment [2604.22067].

Reproducibility is partial. The paper reports the full-factorial protocol, turn budget, temperatures, and metrics, but it does not report public release of data or code, nor licensing details. Reproduction would require access to the vetted question bank, authored vignettes, and the evaluation pipeline, including the LLM judge and prompts. The framework is nevertheless presented as extensible to additional patients, targets, policies, and behaviors, with possible future work including unconstrained human-led interviews, clinician-in-the-loop evaluation, and additional safety or robustness checks [2604.22067].

In broader benchmark research, the term **question selection** names several distinct evaluation problems. In SelQA, it denotes answer sentence selection and answer triggering within open-domain selection-based question answering rather than sequential conversational acquisition [1606.08513]. In Scales++ and in the multiple-regression formulation of efficient benchmarking, it denotes selecting a small, representative subset of benchmark items that predicts full-benchmark performance with high fidelity and low compute cost [2510.26384]; [2605.25773]. In computerized adaptive testing, question selection is the policy that chooses the next item based on current proficiency estimates, and recent work studies how bi-level optimization can amplify selection bias and how debiasing modules such as Cross-Attribute Examinee Retrieval and Selective Mixup-based Regularization affect worst-group accuracy and fairness [2511.15241]. Clinical reliability benchmarks such as SCARE, by contrast, treat question answerability classification as a post-hoc safety layer for EHR question answering, coupling selection or triage with SQL verification and correction [2511.17559].

These neighboring literatures clarify the specificity of the psychiatric intake benchmark. It is neither a sentence-selection corpus, nor a tiny-benchmark subset-selection method, nor a post-hoc answerability classifier. Its distinctive contribution is a controlled framework in which clinically vetted question actions, known latent targets, controllable patient behavior, and a fixed conversational budget jointly define a measurable information-recovery problem [2604.22067].

Source: https://www.emergentmind.com/topics/question-selection-benchmark