EH-Benchmark: Ophthalmic Hallucination Evaluation

• EH-Benchmark is a specialized benchmark that uses over 27,000 multiple-choice questions to evaluate hallucinations in medical LLMs for ophthalmology.
• It features a three-stage multi-agent traceable reasoning workflow that enhances diagnostic audibility and enables self-correction in eye disease analysis.
• The benchmark distinctly separates visual understanding from logical composition hallucinations, allowing targeted evaluation of lesion detection and clinical reasoning.

EH-Benchmark is a specialized benchmark and accompanying reasoning framework for evaluating and mitigating hallucinations in Medical LLMs (MLLMs) for ophthalmology. It combines a multimodal benchmark of over 27,000 multiple-choice questions with a three-stage, multi-agent, top-down traceable reasoning workflow intended to expose hallucinations in eye-disease diagnosis and visual reasoning while making the diagnostic process auditable and self-correcting (Pan et al., 24 Jul 2025).

1. Motivation and Clinical Scope

EH-Benchmark is motivated by the use of MLLMs in ophthalmic diagnosis, where hallucinations can produce confident but medically incorrect statements about lesions, disease stage, or treatment. The benchmark describes these errors as arising from limited ophthalmic domain knowledge, weak lesion-level visual grounding and localization, reliance on language priors instead of actual image content, and lack of structured, multimodal ophthalmic training data. In this setting, the benchmark is designed not only to measure correctness, but to separate failures of visual perception from failures of logical reasoning (Pan et al., 24 Jul 2025).

The benchmark is explicitly ophthalmology-specific. Its target tasks span fundus photography, optical coherence tomography, and case-report reasoning, with emphasis on diseases and findings such as diabetic retinopathy, glaucoma, age-related macular degeneration, retinal lesions, optic disc and cup structure, artery-vein identification, and OCT-visible pathological structures. This domain focus distinguishes it from generic medical evaluation suites that emphasize recognition or question answering without a fine-grained taxonomy of hallucination mechanisms.

A central design principle is the separation of hallucinations into two primary classes. The first, Visual Understanding Hallucination, concerns cases in which the model misidentifies, miscounts, or mislocalizes what is visually present. The second, Logical Composition Hallucination, concerns cases in which the model may recognize findings but combines them incorrectly, violates medical logic, or mishandles temporal and clinical context. This separation structures both the benchmark and the proposed mitigation pipeline.

2. Corpus Composition and Benchmark Construction

EH-Benchmark aggregates 13 datasets across 3 modalities: Color Fundus Photography (CFP), Optical Coherence Tomography (OCT), and Text. The benchmark spans lesion analysis, disease classification, severity grading, anatomical localization, and case-based multimodal reasoning. The CFP and OCT components draw from datasets including Retinal-Lesions, DDR, IDRiD, FGADR, G1020, LES-AV, OIMHS, OCT5k, OCTDL, ODIR-5K, GRAPE, and PAPILA, while the text-heavy reasoning component is constructed from more than 200 published, peer-reviewed ophthalmic case reports (Pan et al., 24 Jul 2025).

For the visual hallucination component, labels are inherited from curated datasets and provide ground truth for lesion presence, lesion type, lesion location, lesion counts, disease labels, and disease stage. For the logical-composition component, the benchmark constructs 806 complex, context-rich questions from case reports. These questions require integration of clinical history, imaging findings, interventions, and disease course. Two medically trained reviewers evaluate each question and its answer choices for reliability, completeness, and hallucination type, and low-quality questions are removed. The benchmark states that all answers are traceable to specific sentences in the original case reports.

The resulting benchmark contains over 27K multiple-choice questions, with approximately 26K assigned to Visual Understanding Hallucination (A1) and 806 assigned to Logical Composition Hallucination (A2). Evaluation uses strict regular-expression matching on the chosen option. This structure turns the benchmark into a closed-form diagnostic instrument rather than an open-ended generation benchmark.

3. Hallucination Taxonomy

The taxonomy introduced by EH-Benchmark is organized around two main classes and multiple clinically specific subclasses. Visual hallucinations are divided by granularity into instance-level and pathological-level errors. Logical-composition hallucinations are divided into reasoning at the instance, pathological, and clinical decision levels (Pan et al., 24 Jul 2025).

Class	Scope	Subtypes or levels
Visual Understanding Hallucination (A1)	Visual grounding failures	Numerical Error, Categorical Error, Position Error, Diagnosis-Type Error, Stage-Level Error
Logical Composition Hallucination (A2)	Reasoning and knowledge-integration failures	Instance-level reasoning, Pathological-level reasoning, Clinical decision-level reasoning

Within A1, Numerical Error concerns counting features such as microaneurysms, hemorrhages, or exudates. Categorical Error concerns confusion between lesion or structure types, such as soft versus hard exudates or artery versus vein. Position Error concerns incorrect localization, including superior versus inferior, nasal versus temporal, or foveal versus peripapillary reasoning. At the pathological level, Diagnosis-Type Error captures incorrect disease classification, and Stage-Level Error captures incorrect severity grading, including diabetic retinopathy grade and glaucoma severity.

Within A2, the instance level examines whether local reasoning about specific findings is coherent. The pathological level tests whether recognized findings are combined into correct disease-level conclusions. The clinical decision level tests therapeutic and prognostic reasoning, including treatment response, follow-up interpretation, and progression logic. This division permits a distinction between “seeing wrong” and “reasoning wrong,” even when the same answer surface form is incorrect.

4. Tasks, Inputs, and Evaluation Protocols

All tasks in EH-Benchmark are multiple-choice. This design is used to support clear correctness labels, standardized scoring across models, and direct assignment of errors to hallucination categories. Major task families include lesion localization, lesion counting, lesion-type identification, disease classification, disease severity grading, optic cup and disc recognition, artery-vein recognition, OCT structure localization, and case-report question answering over multimodal clinical narratives (Pan et al., 24 Jul 2025).

The benchmark evaluates close-ended outputs with Accuracy, Precision, Recall, and F1-score. The reported metric granularity follows the hallucination taxonomy. For A1 instance-level evaluation, Categorical Error uses F1, Precision, and Recall, while Position Error and Numerical Error use Accuracy. For A1 pathological-level evaluation, Diagnosis-Type Error uses F1, Precision, and Recall, while Stage-Level Error uses Accuracy. For A2, the instance-level, pathological-level, and clinical decision-level reasoning tasks use F1, Precision, and Recall.

A further robustness analysis is conducted on 1,250 representative A1 and A2 questions. For each question, four candidate answers are produced and scored by similarity to ground truth along five clinical dimensions: etiology, location, vascular involvement, course or stage, and morphology. Qwen3-235B acts as both generator and evaluator; the correct answer receives reward 4 and unrelated answers receive 0. Distractor options are perturbed through synonym substitution and random shuffling, and the analysis reports consistency, accuracy change, and total reward before and after perturbation. This part of the benchmark is intended to measure sensitivity to semantically close but clinically distinct answer options.

5. Agent-Driven Top-Down Traceable Reasoning Workflow

EH-Benchmark is paired with a three-stage multi-agent framework designed to mitigate hallucinations: the Knowledge-Level Retrieval stage, the Task-Level Case Studies stage, and the Result-Level Validation stage. The underlying LLM core in the described system is GPT-4.1. The first stage is implemented by a Retrieval-Augmented Generation agent that operates over a curated set of ophthalmology URLs and resources, retrieves and segments source text, constructs a vector database, and returns context-aware outputs grounded in retrieved passages (Pan et al., 24 Jul 2025).

The second stage is orchestrated by a Decision Agent. It parses the user query and available images, identifies modality and task complexity, selects tools from a toolkit, and determines the order of invocation. The toolset includes a Diagnose Tool for 18 ophthalmic conditions from CFP and OCT, a Lesion Detection Tool for diabetic-retinopathy lesions in CFP, a Fundus Localization Tool for optic disc and optic cup segmentation and cup-to-disc ratio computation, an OCT Localization Tool for segmenting choroid, retina, and macular hole structures, and a DR Severity Diagnose Tool for five-stage diabetic-retinopathy classification according to the International Clinical Classification of Diabetic Retinopathy. Each tool returns structured outputs such as probabilities, coordinates, segmented regions, lesion categories, and confidence scores.

The third stage is handled by an Evaluation Agent that emulates a senior ophthalmologist and assesses the workflow along three dimensions: correctness, completeness, and workflow adherence. It returns a tuple of the form $$(is\_correct,\ is\_complete,\ is\_followed,\ feedback)$$. If the executed workflow does not follow the planned sequence, the system re-enters the decision stage. If correctness or completeness fails, the feedback identifies missing or problematic tools, and only those tools are re-invoked. This validate-retry loop is presented as the mechanism by which the framework becomes transparent, traceable, and self-correcting rather than purely generative.

6. Empirical Performance and Robustness

EH-Benchmark evaluates 14 LLMs under zero-shot settings. The model set includes generalist LLMs and MLLMs such as Qwen2.5-14B, Qwen2.5-32B, Qwen2.5-72B, LLaVA-1.5-7B, LLaVA-1.5-13B, GPT-4o, GPT-4.1, InternVL2.5-2B, InternVL3-2B, InternVL3-8B, and DeepSeek-V3, as well as medical models including HuatuoGPT-V-7B, MedGemma-4B, and HealthGPT. The proposed system is evaluated as a multi-agent framework built on GPT-4.1 plus ophthalmic tools (Pan et al., 24 Jul 2025).

On A1 Visual Understanding Hallucination, the reported gains are largest for lesion and disease-level grounding. The multi-agent system achieves Cat-E F1 = 0.787, Precision = 0.791, Recall = 0.786; Pos-E Accuracy = 0.633; Num-E Accuracy = 0.480; Diag-E F1 = 0.662; and Sta-E Accuracy = 0.530. By comparison, GPT-4o is reported with Cat-E F1 = 0.085, and the text notes that GPT-4o and GPT-4.1 are particularly prone to Categorical and Diagnosis-Type errors while performing somewhat better on Position and Stage-level errors.

On A2 Logical Composition Hallucination, GPT-4.1 is described as the strongest single-model baseline, with Instance-level F1 = 0.651, Precision = 0.615, Recall = 0.800; Pathological-level F1 = 0.896, Precision = 0.875, Recall = 0.924; and Clinical Decision F1 = 0.921, Precision = 0.914, Recall = 0.930. The multi-agent framework achieves Instance-level F1 = 0.700, Precision = 0.664, Recall = 0.811; Pathological-level F1 = 0.885, Precision = 0.903, Recall = 0.873; and Clinical Decision F1 = 0.919, Precision = 0.920, Recall = 0.921. The benchmark therefore presents the agent framework as strongest on visual-grounding failures and competitive on higher-level clinical reasoning, while also providing substantially greater traceability.

The ablation study attributes distinct roles to retrieval, tools, and agent coordination. Relative to baseline GPT-4o, adding RAG alone yields A2 Instance F1 +5.40% and A2 Pathological F1 +1.13%. Adding tools alone yields A2 Instance F1 +9.76% and A2 Clinical Decision F1 +0.40%, but slightly decreases Pathological F1 by 5.64%. Adding RAG, tools, and a Decision Agent yields A2 Instance F1 +19.16%, A2 Pathological F1 +4.64%, and A2 Clinical Decision F1 +11.35%. The full system with RAG, tools, Decision Agent, and Evaluation Agent yields A2 Instance F1 +21.95%, A2 Pathological F1 +10.90%, and A2 Clinical Decision F1 +21.24%. In the robustness study, the multi-agent system’s consistency decreases by only 0.54% after perturbation, and total reward increases from 2465 for Qwen2.5-32B alone to 4154 in the multi-agent setup.

7. Limitations, Extensions, and Position in Benchmark Research

The benchmark identifies several current limitations. It does not yet include some ophthalmic modalities, specifically lens photographs, Scanning Laser Ophthalmoscopy, and Fundus Fluorescein Angiography. The multi-agent system also lacks real-time clinician feedback and does not yet support online adaptation under expert supervision. The authors further note that underlying datasets may contain geographic or device biases, although this issue is not deeply analyzed in the reported study (Pan et al., 24 Jul 2025).

Planned extensions include broader multimodal expansion, more complex cross-modal questions, and expert-in-the-loop learning for retrieval selection, tool calibration, and agent decision policies. The project resources are released at https://github.com/ppxy1/EH-Benchmark, with separate downloads for the A1 and A2 question sets.

Within arXiv usage, the term “EH-Benchmark” is not unique to ophthalmology. In hardware-aware NAS, an “end-to-end hardware benchmark in the loop” is described as an EH-benchmark-like construct grounded in E2E-Perf and differentiable latency prediction (Jiang et al., 2021). In weather-model verification, “Extreme Weather Bench” is explicitly framed as an Extreme/High-impact benchmark for case-based evaluation of high-impact phenomena (McGovern et al., 1 May 2026). This suggests that “EH-Benchmark” functions both as a proper noun for the ophthalmic hallucination benchmark and as a broader acronymic pattern in unrelated benchmark literatures. A plausible methodological implication is that the ophthalmic benchmark’s explicit separation of hallucination classes, task families, and scoring procedures is consistent with the capability-content-evidence distinctions emphasized in benchmark-design frameworks such as Evidence-Centered Benchmark Design (Liu et al., 2024).