Measuring Epistemic Resilience of LLMs Under Misleading Medical Context

Abstract: LLMs now reach expert-level scores on medical licensing exams, encouraging the assumption that high scores imply safe medical judgment while patients increasingly use them for health advice. We show this assumption is fragile: when misleading context is injected into questions that LLMs originally answer correctly, they abandon the correct answer. We call the ability to maintain correct judgment under adversarial context epistemic resilience, and introduce MedMisBench to measure it. MedMisBench contains 10,932 medical question items and 48,889 misleading context-option pairs spanning medical reasoning, agentic capability, and patient-journey evaluation. Across 11 model configurations, mean accuracy falls from 71.1% on original questions to 38.0% under focused misleading context, with 51.5% attack success. The most damaging injections are formal, rule-like fabrications: authority-framed falsehoods reach 69.5% attack success and exception-poisoning claims reach 64.1%. A 14-member clinical panel from 7 countries identified serious potential harm in 38.2% of reviewed cases. MedMisBench exposes a structural blind spot in LLM evaluation in medical settings: existing benchmarks measure what models know, but not whether they preserve correct medical judgment under misleading context.

• The paper shows that LLMs' accuracy drops from 71.1% to 38.0% under focused misleading contexts, highlighting a critical gap in epistemic resilience.
• It employs a rigorously constructed benchmark with 10,932 medical QA items and 48,889 misleading pairs, systematically testing five content corruptions and three provenance frames.
• Mitigation strategies, including external evidence retrieval and defensive prompts, partially reduce misleading impact but do not fully safeguard clinical reasoning.

Measuring Epistemic Resilience of LLMs in the Medical Domain: An Expert Analysis of "MedMisBench: Measuring Epistemic Resilience of LLMs Under Misleading Medical Context" (2606.12291)

Introduction

This work interrogates the foundational assumption underlying the deployment of LLMs in healthcare: that high performance on "clean" medical benchmarks is indicative of reliable and safe clinical reasoning in deployment. The study introduces the concept of epistemic resilience—a model's ability to preserve correct medical judgment when exposed to plausible but false contextual information—and establishes MedMisBench, a comprehensive evaluation suite explicitly designed to measure this attribute at scale.

The benchmark is constructed from 10,932 rigorously filtered medical question-answer items, paired with 48,889 misleading context-option pairs. MedMisBench systematically decomposes misleading context into five medically grounded content corruption types and three provenance frames, revealing model vulnerabilities to information contamination that are not captured by standard accuracy metrics. Evaluated across 11 model configurations, including frontier commercial, open-weight, and domain-specific medical models, the study identifies structural deficits in current paradigms and quantifies the associated clinical risks through a 14-member, seven-country clinician panel. The implications are substantial for both AI safety and the future integration of LLMs in real-world medical environments.

MedMisBench Construction and Methodology

MedMisBench's methodological design addresses two axes of misleading medical input: content corruption and provenance framing. Content corruptions span relationship/sequence inversion, threshold/reference corruption, cue remapping, spurious anchoring, and exception poisoning. Provenance framing encodes neutral false statements, patient-reported beliefs, and authority-like sources (e.g., guidelines).

Items were extracted from multiple high-value medical QA benchmarks, each vetted via LLM-based applicability filtering and human expert validation to ensure that misleading injections preserve semantic coherence while remaining false and contextually plausible. The adversarial context generation pipeline produces both focused (Type 1) and mixed evidence (Type 2) delivery protocols, enabling fine-grained assessment of both direct and arbitration-based resilience. This leads to a reproducible, answer-grounded evaluation protocol that is both scalable and interpretable.

MedMisBench validity was systematically established through clinician review, yielding a composite item-quality score of 1.76/2.00, with high fidelity in attack-type targeting, answer preservation, and clinical plausibility.

Figure 1: MedMisBench turns clean medical QA into paired resilience tests: filtered answer-grounded items receive option-aligned misleading context and focused Type~1 or mixed Type~2 delivery.

Main Empirical Results: Epistemic Resilience under Adversarial Context

The central numerical findings demonstrate a decoupling between clean accuracy and actual deployment resilience. Across models, clean accuracy averaged 71.1% but dropped precipitously to 38.0% under focused Type~1 context injection, with mean attack success rate (ASR) of 51.5%. Notably, 45.4% of these failures are targeted attack success events, indicating direct uptake of the specific adversarial claim rather than generic answer degradations.

Figure 2: Clean accuracy overstates epistemic resilience. Mean accuracy falls from 71.1\% clean to 38.0\% under Type~1, while Type~2 returns to 70.5\% without eliminating ASR failures.

The Type~2 (all-option) protocol, where both truthful and misleading context are present, resulted in a markedly lower ASR (18.7%) and near-restoration of accuracy (70.5%). However, residual resilience failures remain, particularly in open-weight and domain-specific models, which had up to 52.0% Type~2 ASR versus <10% for the best commercial models.

Figure 3: Focused delivery drives most resilience loss. Type~1 averages 51.5\% ASR versus 18.7\% for Type~2, showing that one plausible false claim is especially damaging.

Epistemic resilience did not uniformly correlate with increased "reasoning effort" model settings. For some models, more elaborate reasoning chains increased susceptibility to authoritative or rule-like falsehoods, contradicting the expectation that deliberative reasoning alone is sufficient for resilience.

Taxonomy Analysis: Content and Provenance Effects

The stratified resilience analysis revealed that authority-framed and formal, rule-like falsehoods were the most destructive. Authority-framed false claims had a mean Type~1 ASR of 69.5%, and neutral statements were similar at 65.2%; in contrast, patient-framed misleading claims yielded a relatively low 18.5% ASR. Content-wise, exception poisoning (64.1%) and threshold/reference corruption (60.9%) had far more impact than spurious anchoring (20.9%), establishing a clear risk calibration by adversarial content type.

Clinical Risk Assessment

A key contribution is the clinician panel's harm review of model responses under misleading context. 38.2% of reviewed tasks were classified as "worst-case"—incorrect final answer with material uptake of injection and serious clinical harm potential. An additional 46.1% exhibited wrong answers with lesser but non-negligible harm. Only 5.6% of model outputs both recovered the correct answer and explicitly rejected the false context. Inter-rater agreement metrics (Gwet's AC2 0.94–0.95 for critical dimensions) support the robustness of these findings.

Figure 4: Clinicians review both item validity and response harm using the case, injected context, model output, and Rubric A/B controls.

Robustness and Mitigation Strategies

Intervention studies demonstrated partial but incomplete mitigation. External evidence search (via web retrieval and self-verification) dramatically reduced Type~1 ASR for high-capacity models (e.g., Gemini-3.1-pro Type~1 ASR fell from 81.5% to 16.1% on HLE tasks) but was less effective for less capable configurations. Defensive prompt engineering (explicit warnings about possible misinformation) reduced Type~1 ASR by up to 14 points, but substantial residual errors persisted—indicating that current LLMs frequently fail to operationalize caution even with explicit instructions.

Figure 5: Generator choice does not explain the main pattern: Type~1 remains more damaging than Type~2 after replacing the injection generator.

Alternative injection generations and provenance assignment shuffling did not materially affect the direction or strength of the resilience gap, confirming the robustness of the observed phenomena against methodological variance.

Implications and Future Directions

The findings have critical implications for both LLM safety evaluation in medicine and the broader ambition of reliable, deployment-ready AI agents in high-stakes domains. First, standard "clean" benchmarks fail to capture failure modes that arise from misleading but plausible context, leading to an overestimation of the reliability and safety of current models. Second, the most damaging adversarial interventions involve authority or rule-like statements, highlighting the importance of epistemological robustness to explicit protocol-level errors. Third, mitigation frameworks that rely solely on increased deliberation, retrieval augmentation, or basic cautionary instructions are insufficient, particularly for lower-tier or non-commercial models.

Practical consequence: Any proposed clinical deployment of LLMs—even those with strong clean-benchmark performance—requires explicit epistemic resilience testing. Theoretical consequence: The failures elucidate a gap in current learning and prompting schemas for LLMs, which lack structured mechanisms for evidentiary adjudication and robust rejection of plausible yet false context.

Prospective research avenues include open-ended resilience testing (beyond answer-grounded QA), multi-modal adverse context incorporation, adversarial training for context robustness, and mechanistic interpretability for resilience failures. For AI safety as a field, MedMisBench offers a blueprint for adversarial evaluation toolkits in other verticals where misinformation uptake is consequential.

Conclusion

MedMisBench establishes that LLMs, including state-of-the-art commercial models, are highly susceptible to misleading medical context, often abandoning correct predictions under minimal, plausible context corruption. This epistemic resilience gap is greatest for authoritative and rule-like falsehoods. Standard clean accuracy metrics conceal the severity of this issue—serious clinical harm is common in misled model outputs. Mitigation strategies offer partial relief but do not obviate the need for explicit, adversarially-oriented evaluation and response-level clinical oversight prior to any deployment in health-critical settings. This work crystallizes epistemic resilience as a central desideratum for the evaluation and safe integration of LLMs in medicine, motivating further advances in robust model architectures and adversarially informed benchmarking.