HealthBench-Hard Evaluation Suite

• HealthBench-Hard is an evaluation suite that stresses large language models in demanding healthcare settings using physician-crafted criteria.
• It employs an adversarial selection process on 5,000 cases to isolate 1,000 of the most failure-prone and ambiguous clinical scenarios.
• The benchmark highlights gaps in context-awareness, completeness, and safety, driving advances in adaptive clinical dialogue and deployment.

HealthBench-Hard is a rigorously constructed evaluation suite within the HealthBench benchmark ecosystem, designed to measure the limits of LLM performance, reliability, and clinical safety in the most challenging healthcare scenarios. It operationalizes medical difficulty through physician-authored rubrics, adversarial case selection, and high-density ambiguity, thus exposing worst-case model behaviors and framing open research questions in alignment, reasoning, and workflow completeness. HealthBench-Hard is referenced in multiple high-impact LLM, reinforcement learning, and evaluation studies, and now serves as the canonical stress-test for frontier and open-source healthcare LLMs.

1. Definition, Scope, and Rationale

HealthBench-Hard is the "hard" subset of the broader HealthBench evaluation framework, specifically curated to create a persistent and unsaturated challenge for cutting-edge health AI systems. It comprises exactly 1,000 open-ended conversations or queries sourced from various clinical domains—emergencies, global health, context-seeking, expertise tailoring, and more—where top-performing LLMs remain far from saturation, with best model scores historically ranging from 0.10 (GPT-3.5 Turbo) up to approximately 0.32 (OpenAI o3) and, in recent advances, slightly higher for agentic or retrieval-augmented models (Arora et al., 13 May 2025, Ravichandran et al., 29 Aug 2025).

Distinct from the general HealthBench (5,000 cases) and the HealthBench Consensus (3,671 cases × 34 consensus criteria), HealthBench-Hard was explicitly created to:

• Encompass the most failure-prone, ambiguous, or information-deficient cases for LLMs.
• Provide an unsaturated sub-benchmark with performance headroom, maximizing sensitivity to incremental model improvements.
• Illuminate error patterns in context-awareness, completeness, safety, and global context adaptation.

HealthBench-Hard has also been adapted and referenced by related benchmarks in machine translation (as the healthcare slice of HardMTBench (Li et al., 27 May 2026)) and in professional clinical dialogue (HealthBench Professional (Hicks et al., 30 Apr 2026)) as the standard for "difficult" or "adversarial" clinical cases.

2. Dataset Construction and Selection Methodology

The construction of HealthBench-Hard employs a systematic adversarial filtering pipeline:

• Source Pool: 5,000 HealthBench conversations spanning routine to complex clinician-patient and research interactions; each is annotated with multiple physician-authored rubrics totaling >48,000 unique criteria.
• Difficulty Ranking: Five contemporary LLMs (o3, Grok 3, Gemini 2.5 Pro, Claude 3.7 Sonnet, Llama 4 Maverick) are scored on all cases. Any sample for which all models scored ≤ 0 is removed.
• Selection: The 1,000 samples with lowest mean model score across models are selected.
• Scenario Types include: emergency referrals (e.g., acute decompensation), hedging/uncertainty (insufficient detail, intent ambiguity), context-seeking (information elicitation need), global health (resource limitations), structured data tasks, expertise-tailored communication (layperson/professional), and response depth variability (Arora et al., 13 May 2025, Ravichandran et al., 29 Aug 2025).

In HealthBench Professional (Hicks et al., 30 Apr 2026), "Hard" cases undergo further enrichment: adversarial selection (red teaming), three-phase multi-physician rubric vetting, Likert-based difficulty labeling (1–2 on 1–7 scale), resulting in ~3.5–8× higher prevalence of hard examples versus the underlying distribution.

Concrete selection criteria (from "Decomposing Physician Disagreement in HealthBench" (Borgohain et al., 26 Feb 2026)) include:

• Mean pass-rate in the midrange (e.g., 0.25–0.75), where physician disagreement ≥ 25% and continuous-strength disagreement $1 - |2\,\mathrm{pass\_fraction} - 1| > 0.6$.
• Presence of annotated reducible uncertainty (missing context, ambiguous phrasing).
• Pairwise disagreement $D_{pw} = 2n_\text{pass} n_\text{fail}/[n(n-1)]$, optionally used to select the most divergent quartile.

3. Rubric Design, Behavioral Axes, and Scoring

HealthBench-Hard evaluation is rubric-driven, with each case accompanied by 5–10 physician-written criteria assigned point values $p_{ij}\in[-10,10]$, categorizing desired (positive) and undesired (negative) model behaviors. The main behavioral axes are:

1. Accuracy: Concordance with current standards of care and medical evidence, signaling uncertainty as needed.
2. Completeness: Inclusion of all relevant data, actions, warnings, and follow-up steps.
3. Context-Awareness: Sensitivity to user identity, missing details, prior context, geographical/practice constraints, and adaptive information-seeking.
4. Communication Quality: Audience-specific clarity, language calibration, and instructional structure.
5. Instruction Following: Obedience to explicit format or content constraints, without sacrificing safety or accuracy.

Consensus and case-specific rubrics both appear. Scoring proceeds as follows, with each response $r_i$ and criterion $j$:

$$s_i = \frac{\sum_{j=1}^{M_i} \mathbf{1}\{r_{ij}\}\,p_{ij}} {\sum_{j=1}^{M_i} \max(0,\,p_{ij})}$$

where $s_i$ is clipped to [0,1]. The overall HealthBench-Hard (HB-Hard) score is the mean across $N=1,000$ cases, each possibly weighted equally or by axis-specific schemes (Arora et al., 13 May 2025, Ravichandran et al., 29 Aug 2025, Zhou et al., 13 Nov 2025). In HealthBench Professional, a length-penalized adjustment is used:

$$s^{\text{len}}_i = s_i - \hat\beta_L(\ell_i - 2000),\quad \hat\beta_L = 2.94 \times 10^{-5}$$

Inter-rater agreement across physician and model annotators is high (Cohen’s $\kappa\approx0.7-0.8$ in meta-evaluations).

4. Error Patterns, Disagreement Structure, and Uncertainty Decomposition

HealthBench-Hard uniquely concentrates physician and LLM ambiguity:

• The dataset's inverted-U error profile (disagreement vs. mean pass rate) peaks for borderline-quality outputs (mean pass rate $D_{pw} = 2n_\text{pass} n_\text{fail}/[n(n-1)]$0), with 30–40% case-level disagreement, and returns to $D_{pw} = 2n_\text{pass} n_\text{fail}/[n(n-1)]$1 at the extremes (Borgohain et al., 26 Feb 2026).
• Variance decomposition identifies residual (case-level) variance as the dominant source: rubric identity explains 15.8% of pass/fail label variance but only 3.6–6.9% of disagreement; physician identity explains just 2.4%; residual/case-level variance constitutes 81.8%–96.4%.
• Statistical modeling shows no reduction in residual disagreement with HealthBench metadata (z=–0.22, p=0.83), rubric language (pseudo-$D_{pw} = 2n_\text{pass} n_\text{fail}/[n(n-1)]$2), medical specialty (0/300 significant Tukey pairs), triage surface features (AUC=0.58), or embeddings (AUC=0.485).
• Uncertainty categories (Borgohain et al., 26 Feb 2026): Reducible uncertainty—cases with missing information or ambiguous phrasing—more than doubles the odds of disagreement (OR=2.55, 95%CI[2.13,3.06], p<10⁻²⁴), though this factors accounts for only ≈3% of total disagreement variance. Irreducible clinical ambiguity (genuine medical uncertainty) has no effect (OR=1.01, p=0.90).

Actionable strategies proven to reduce disagreement in “hard” cases are prompt/rubric enrichment, disambiguation of criteria, and context-sufficiency checking.

5. Quantitative Results and Model Comparisons

HealthBench-Hard consistently reveals wide headroom for improvement and pronounced differences between model families and approaches.

Model/System	HealthBench-Hard Score
GPT-3.5 Turbo	0.10
Claude 3.7 Sonnet	0.14
GPT-4o	0.16
o1	0.21
GPT-4.1	0.27
o3	0.32
DR.INFO (RAG/agentic, n=1,000)	0.51
GPT-5 (“thinking” mode)	0.46
Baichuan-M2 (32B, SOTA open-source)	0.347
GPT-OSS-120B (teacher)	0.30
Qwen3-32B (base)	0.12
Qwen3-32B (KD + MuSeR)	0.431
ChatGPT for Clinicians (Pro.)	0.59*
Human physicians (Pro.)	0.437*

*For HealthBench Professional Hard, which is aligned in definition and difficulty with HealthBench-Hard (Hicks et al., 30 Apr 2026).

Error analysis reveals:

• Context Awareness and Completeness are the lowest-scoring axes, even for state-of-the-art models (e.g., DR.INFO: Context 0.35, Completeness 0.43) (Ravichandran et al., 29 Aug 2025).
• Common failure categories: omission of clarifying questions, underescalation/overescalation in emergencies, incomplete documentation, and misinterpretation of resource constraints.
• Retrieval-augmented, dynamic-reasoning, and self-refinement strategies yield significant empirical gains (MuSeR: +11.6 pts over KD baseline on Qwen3-32B) (Zhou et al., 13 Nov 2025).

6. Extended Applications: Professional, Translation, and Low-Power ML

• HealthBench Professional-Hard (Hicks et al., 30 Apr 2026): Hard enrichment via red teaming, adversarial selection, and real-world clinical chat logs yields a subset (N=269 difficult cases in N=525) with outsized difficulty. The top model (ChatGPT for Clinicians) achieves 59.0 compared to 43.7 for human physicians, with advantage especially pronounced on adversarially selected examples.
• HardMTBench (Healthcare domain) (Li et al., 27 May 2026): Incorporates a "HealthBench-Hard" slice into domain-specialized Chinese-English translation. With 1,666 directional items and a focus on terminology density, the hardest cases increase cross-system metric spread (GEMBA-DA 75–98), highlighting that fluency metrics mask terminology and adequacy errors, and top LLMs exhibit only ~61% term accuracy, substandard for clinical deployment.
• TinyML/Low-power (Samakovlis et al., 2024): BiomedBench applications such as Bio-BPFree and SeizDetCNN are tied directly to HealthBench-Hard conditions, specifying constraints like <100KiB RAM, <10ms latency, and battery lifetimes >7 days.

7. Benchmark Design Principles and Future Directions

HealthBench-Hard exemplifies a set of evolving best practices for healthcare AI evaluation:

• Adversarial enrichment: Deliberately upsampling examples most difficult for current LLMs to avoid benchmark saturation and artificially high scores.
• Granularity in error annotation: Fine-grained rubric criteria enable quantification of not just factual inaccuracy, but incomplete context gathering, instruction violation, and communication style mismatches.
• Relevance to practical deployment: Evaluation axes and scenario types are directly drawn from real-world clinician workflows, not synthetic vignettes.
• Physics of clinical disagreement: The benchmark exposes the structural limits imposed by reducible and irreducible uncertainties.
• Extensible framework: HealthBench-Hard’s selection protocols, scoring methods, and enrichment strategies have been adopted beyond core diagnosis (translation, wearable AI, refusal-compliance).
• Headroom for continuous improvement: The unsaturated, long-tail distribution of case difficulty ensures continued relevance for future model generations (Arora et al., 13 May 2025, Borgohain et al., 26 Feb 2026, Hicks et al., 30 Apr 2026).

A plausible implication is that as LLM capabilities evolve, benchmarks like HealthBench-Hard will need dynamic adversarial expansion, finer-grained uncertainty annotation, and possibly real-world clinical outcome validation to remain effective.

---

References:

(Arora et al., 13 May 2025, Ravichandran et al., 29 Aug 2025, Zhou et al., 13 Nov 2025, Borgohain et al., 26 Feb 2026, Hicks et al., 30 Apr 2026, Li et al., 27 May 2026, Samakovlis et al., 2024)