Evaluation format, not model capability, drives triage failure in the assessment of consumer health AI

Abstract: Ramaswamy et al. reported in \textit{Nature Medicine} that ChatGPT Health under-triages 51.6\% of emergencies, concluding that consumer-facing AI triage poses safety risks. However, their evaluation used an exam-style protocol -- forced A/B/C/D output, knowledge suppression, and suppression of clarifying questions -- that differs fundamentally from how consumers use health chatbots. We tested five frontier LLMs (GPT-5.2, Claude Sonnet 4.6, Claude Opus 4.6, Gemini 3 Flash, Gemini 3.1 Pro) on a 17-scenario partial replication bank under constrained (exam-style, 1,275 trials) and naturalistic (patient-style messages, 850 trials) conditions, with targeted ablations and prompt-faithful checks using the authors' released prompts. Naturalistic interaction improved triage accuracy by 6.4 percentage points ($p = 0.015$). Diabetic ketoacidosis was correctly triaged in 100\% of trials across all models and conditions. Asthma triage improved from 48\% to 80\%. The forced A/B/C/D format was the dominant failure mechanism: three models scored 0--24\% with forced choice but 100\% with free text (all $p < 10^{-8}$), consistently recommending emergency care in their own words while the forced-choice format registered under-triage. Prompt-faithful checks on the authors' exact released prompts confirmed the scaffold produces model-dependent, case-dependent results. The headline under-triage rate is highly contingent on evaluation format and should not be interpreted as a stable estimate of deployed triage behavior. Valid evaluation of consumer health AI requires testing under conditions that reflect actual use.

• The paper demonstrates that triage underperformance stems from constrained evaluation protocols rather than intrinsic model deficiencies.
• It compares forced-choice versus naturalistic interaction methods across five frontier LLMs, revealing a 6.4% aggregate improvement in triage accuracy with free-text input.
• Ablation studies and prompt-faithful replications confirm that exam-style constraints distort real-world emergency recommendations in health AI.

Evaluation Format Determines Apparent Triage Failure in Consumer Health AI: A Mechanistic Assessment

Introduction

This paper challenges prior conclusions regarding the triage safety of consumer health LLM systems, specifically critiquing the methodology underlying recent high-profile claims of elevated emergency under-triage rates in ChatGPT Health. The authors systematically decompose the evaluation scaffold employed by Ramaswamy et al. and empirically demonstrate that measured under-triage emerges not from model incapacity, but from the interaction of forced-choice, exam-style evaluation protocols with the output dynamics of state-of-the-art LLMs. The analysis is executed as a controlled, partial replication using five frontier LLMs and a suite of ablation and prompt-faithful techniques.

Methods

The experimental design juxtaposed two evaluation conditions: (1) a "constrained" exam-style protocol that mirrors that of Ramaswamy et al., involving forced discretization into predefined A/B/C/D triage categories, deliberate knowledge suppression, and the prohibition of clarifying questions, and (2) a "naturalistic" protocol, which utilizes patient-style free-text input without structural or interactional constraints. The main test set consisted of 17 researcher-authored clinical scenarios representing a range of urgency and symptom reporting. Across conditions, over 2,000 LLM inferences were elicited from GPT-5.2, Claude Sonnet 4.6, Claude Opus 4.6, Gemini 3 Flash, and Gemini 3.1 Pro. Targeted ablations isolated individual scaffold elements, and prompt-faithful checks assessed whether findings generalized to the exact prompts used in the original study.

Results

Naturalistic evaluation yielded a statistically significant improvement in triage accuracy across all tested models (aggregate +6.4 percentage points, $p = 0.015$), notably restoring accurate emergency recommendations in scenarios where constrained evaluation had artificially suppressed performance. In particular, cases involving diabetic ketoacidosis (DKA) were always triaged correctly (100% accuracy), contradicting the failed performance previously attributed to the model. The most striking finding emerged in the asthma exacerbation scenario: forced-choice conditions produced extreme under-triage in three of five models (0–24% accuracy), while free-text input restored 100% correct recommendations in all three ($p < 10^{-8}$). Ablations confirmed that forced discretization—requiring the model to select among abstract lettered choices—was both necessary and sufficient to induce measured failure; other scaffold elements had lesser and inconsistent impact.

Prompt-faithful evaluation on the released Ramaswamy et al. prompt set reproduced the finding that measured performance is both model- and prompt-dependent. Removing scaffold constraints produced heterogeneous effects across model families and scenarios, reinforcing the claim that this format acts as a behaviorally active intervention.

Discussion

The primary implication is that high-profile claims of widespread emergency under-triage by LLM-based systems—in particular, the reported 51.6% under-triage rate—are not robust to variations in evaluation protocol and should not be interpreted as indicative of deployed product behavior. The forced-choice, non-interactive protocols used in much of the recent benchmarking of consumer health LLMs are misaligned with real-world use, wherein consumers present unstructured data, employ lay language, and frequently require, and receive, model-initiated clarification.

The results underscore the necessity of aligning evaluation protocol with actual deployment characteristics. Suppressing model knowledge and clarification not only underutilizes the core generative and interactive affordances of LLMs, but systematically distorts safety-critical behavior estimates. This has direct regulatory and policy implications: regulatory and clinical benchmarking of health AI must account for protocol-induced artifacts or risk disqualifying models (or deploying them with the wrong safety profile) based on inapposite artificial task designs.

Implications and Future Directions

From a methodological perspective, this work compels reevaluation of current triage benchmarking practice and reporting standards. It shows that claims about "unsafe under-triage rates" in consumer health LLMs are not transferable to real-world conditions without explicit validation under naturalistic, multi-turn, patient-style protocols. As commercial deployments increasingly leverage persistent user context, multi-turn clarification, and refined conversational memory, the gap between exam-style testbeds and production interaction will only widen.

Future research should prioritize full multi-turn, mis-reported, and ambiguous symptom studies that mirror real patient behaviors, facilitated by larger scenario banks and, eventually, actual user interaction logs. Further, as new models gain functionality for context persistence and domain adaptation, benchmarking must keep pace by updating evaluation frameworks rather than relying on static protocol artifacts.

Conclusion

Observed triage failures in LLM-based consumer health products, specifically under the exam-style evaluation scaffolds prevalent in current literature, do not indicate model incapacity but rather are artifacts of protocol misalignment. Forced-choice, non-interactive formats systematically mask clinically appropriate recommendations that are otherwise accessible through naturalistic interaction. Rigorous safety evaluation of deployed consumer health AI necessitates protocols that capture the realities of end-user interaction rather than artifacts of exam-style legacy benchmarking.