EndoAgentBench: Multimodal Endoscopic Evaluation
- EndoAgentBench is a unified benchmark that integrates fine-grained lesion detection with clinical language generation to simulate endoscopic diagnostic workflows.
- It incorporates five tasks—lesion classification, quantification, visual grounding, image captioning, and report generation—to assess both visual perception and decision-making.
- The dataset combines six public and one private clinical dataset into 5,709 image–question–answer pairs, emphasizing realistic pathology and multi-step reasoning.
Searching arXiv for the specified paper to ground the article in the current record. (Tang et al., 10 Aug 2025) EndoAgent EndoAgentBench arXiv search EndoAgentBench is a unified, expert-annotated benchmark for evaluating multimodal endoscopic “vision-to-decision” agents in realistic diagnostic scenarios. Introduced alongside EndoAgent, it is designed to test not only fine-grained visual understanding but also open-ended clinical language generation, with each benchmark item coupling an endoscopic image to a clinical question and requiring the model to transform visual evidence into clinically meaningful output (Tang et al., 10 Aug 2025). Its stated role is to align evaluation with the endoscopic diagnostic workflow: models must recognize findings such as lesion presence, location, and count, and then express findings, clinical significance, and recommendations in text.
1. Conceptual scope and “vision-to-decision” formulation
EndoAgentBench operationalizes endoscopic “vision-to-decision” reasoning through two complementary capability axes. The first is fine-grained visual perception, covering recognition, counting, and localization. The second is open-ended clinical language generation, covering descriptive captioning and structured report writing. This organization is intended to approximate the way endoscopists synthesize image evidence across steps rather than treating perception and reporting as isolated tasks (Tang et al., 10 Aug 2025).
The benchmark maps this formulation onto five subtasks: lesion classification, lesion quantification, visual grounding, image captioning, and report generation. In each item, the model is asked to perceive relevant findings—for example, distinguishing normal tissue from polyp, adenoma, or cancer, or selecting the correct lesion-localizing bounding box—and to convert those findings into actionable clinical statements. The benchmark therefore evaluates whether a system can move from “seeing” to “deciding,” rather than merely recognizing visual patterns.
Its design principles emphasize realistic clinical scenarios and controlled difficulty. The dataset includes a diverse case mix spanning normal, benign, pre-malignant, and malignant findings; it encodes multi-step reasoning prerequisites such as the dependence of quantification on detection and segmentation and the geometric demands of grounding; and it is compatible with memory/reflective agent frameworks while keeping each benchmark item single-image and single-turn for tractable evaluation. Hard distractors are generated for grounding, and stratified sampling is used to cover variable lesion counts.
2. Task taxonomy and dataset composition
EndoAgentBench contains 5,709 image–question–answer pairs. Its task distribution is explicitly divided between visual understanding and language generation, with the former comprising 61.2% of items and the latter 38.8% (Tang et al., 10 Aug 2025).
| Task | Count | Share |
|---|---|---|
| Lesion classification | 884 | 15.5% |
| Lesion quantification | 1,376 | 24.1% |
| Visual grounding | 1,319 | 23.1% |
| Image captioning | 1,064 | 18.6% |
| Report generation | 1,066 | 18.7% |
The visual-understanding subset includes lesion classification, lesion quantification, and visual grounding. Classification concerns lesion description and category assignment; quantification concerns lesion counting; grounding concerns lesion detection and localization through bounding-box-based option selection. The language-generation subset consists of image captioning, which targets diagnostic feature description, and medical report generation, which targets comprehensive follow-up and management expressed through structured reports.
The lesion distribution is also specified. Normal cases number 855 (15.0%), polyps 2,994 (52.4%), adenomas 896 (15.7%), and cancers 964 (16.9%). Abnormal cases therefore comprise over two-thirds of the benchmark, which the authors describe as providing sufficient coverage of clinically significant findings. Tool recommendation and quality assessment are not defined as standalone categories, although recommendations are evaluated within report generation, and visual grounding is described as implicitly reflecting procedural localization competence (Tang et al., 10 Aug 2025).
This composition suggests that the benchmark is deliberately weighted toward pathology-rich decision settings rather than balanced screening prevalence. A plausible implication is that it is optimized for stress-testing diagnostic reasoning under clinically salient abnormal findings rather than for estimating population-level deployment behavior.
3. Data sources, modalities, and annotation protocols
The benchmark aggregates six widely used public endoscopy datasets together with a private clinical dataset annotated by experts. The public datasets and their reported sample counts are CVC-300 (60), CVC-ClinicDB (62), CVC-ColonDB (380), Kvasir/Kvasir-SEG (99), ETIS-LaribPolypDB (196), and SUN-SEG (1,354). The private clinical dataset contributes 3,558 samples. Overall, public data account for 37.7% of the total and the private set 62.3% (Tang et al., 10 Aug 2025).
The sources span colonoscopy image benchmarks and a clinician-annotated private set, combining public research data with real-world clinical data. Specific imaging modalities are not enumerated in the paper. The text notes that most public sets are white-light colonoscopy still frames and that ETIS-LaribPolypDB is cited from work on wireless capsule endoscopy images. Device brands, hospital identities, and population metadata are not disclosed.
Annotation protocols differ by task but follow a common standardization logic. For lesion classification, public labels follow official dataset annotations, while private labels are assigned by board-certified physicians. For visual grounding, existing public bounding boxes are used when available; when only segmentation masks are present, minimum enclosing rectangles of connected components are computed. Private grounding boxes are manually annotated by expert clinicians, and distractor boxes are algorithmically generated to increase difficulty. For lesion quantification, counts are derived from the number of masks or boxes in public datasets or from expert annotators in the private dataset, with stratified sampling used to cover varying lesion counts.
Language-task annotations follow a different pipeline. Image captioning and report generation questions are created from diverse templates intended to reflect real diagnostic queries. Reference answers are automatically generated by Qwen-VL-Plus, guided with prior lesion information to ensure medical adequacy. Reference reports are structured into three sections: Endoscopic Findings, Clinical Significance, and Recommendation. The paper reports standardized protocols and rigorous quality control, but it does not provide inter-annotator agreement statistics. Ambiguity handling is described as implicit in template design and distractor generation, and multi-image or multi-turn instances are excluded to keep evaluation tractable (Tang et al., 10 Aug 2025).
4. Evaluation protocol, answer formats, and metrics
EndoAgentBench is presented primarily as an evaluation benchmark for zero-shot and few-shot agents and models rather than as a training corpus. The paper does not declare official train, validation, and test splits. This choice places emphasis on cross-model comparability in instruction-following settings rather than on supervised optimization against a canonical partition (Tang et al., 10 Aug 2025).
Answer formats differ by capability axis. Visual-understanding tasks use closed-form multiple-choice outputs, enabling exact matching. Language-generation tasks use open-ended text and are scored with an LLM-as-judge protocol. Prompts instruct evaluated models to produce medically accurate outputs based only on visible image evidence and the question context. For reference generation in captioning and report generation, Qwen-VL-Plus is given lesion priors; evaluated models are not given priors beyond the image and the question.
The visual metric is accuracy:
Language generation is evaluated by a relative score using Qwen-VL-Plus across seven clinical dimensions: diagnostic accuracy, clinical structure, medical terminology, detailed description, clinical significance, recommendations, and professional quality. Each dimension is scored from 0 to 10, and the reported metric is the ratio of the model score to the reference score:
The paper further specifies:
For visual grounding case analysis, IoU is used internally by the agent to select among candidate boxes:
However, the official benchmark metric for grounding remains accuracy over answer options rather than direct box overlap.
The benchmark is distributed in standardized JSONL, and visual tasks are additionally preprocessed into VLMEvalKit-compatible TSV with base64-encoded images and fields for question, options, gold answer, and metadata. Language tasks maintain unified JSONL schemas. The repository positions these assets as part of the evaluation release, and the visual-task pipeline is registered for one-command benchmarking through VLMEvalKit (Tang et al., 10 Aug 2025).
5. Comparative performance and what the benchmark reveals
EndoAgentBench is used to compare general multimodal LLMs, medical multimodal LLMs, and the agentic EndoAgent system. The general multimodal baselines are GPT-4o, Gemini 2.5 Pro, Step-1o-Vision-32k, Yi-Vision, and Qwen-VL-Plus. The medical multimodal baselines are LLaVA-Med, HuatuoGPT-Vision-7B, and ColonGPT. These baselines are evaluated in zero-shot or instruction-following mode, and no task-specific fine-tuning on EndoAgentBench is reported (Tang et al., 10 Aug 2025).
On fine-grained visual tasks, EndoAgent attains 88.46 accuracy on lesion classification, 84.16 on lesion quantification, and 83.47 on visual grounding, with an overall visual average of 84.97. The next-best scores are 68.44 for classification by Gemini 2.5 Pro, 81.83 for quantification by Step-1o-Vision-32k, and 76.27 for grounding by Gemini 2.5 Pro; the next-best overall visual average is 74.57 by Gemini 2.5 Pro. The paper summarizes the margin as +20.02 points on classification and +7.20 on grounding versus the second-best model, while noting that quantification exhibits a narrower spread because of the dominance of single-lesion samples.
On open-ended language tasks, captioning is led by Gemini 2.5 Pro at 104.83 relative score, with EndoAgent second at 100.32. In medical report generation, EndoAgent leads at 95.90, with GPT-4o second at 95.79. The language average is 97.83 for EndoAgent, compared with 94.76 for GPT-4o and 94.35 for Gemini 2.5 Pro. These results indicate that EndoAgent is strongest on the benchmark’s integrated visual reasoning and structured-report setting, while captioning remains competitive among general multimodal systems (Tang et al., 10 Aug 2025).
The benchmark also exposes system-level effects through EndoAgent ablations. Without reflection and without dual memory, EndoAgent records 55.50 visual average, 96.96 on captioning, 102.35 on report generation, and 99.79 language average. Reflection only yields 82.00, 100.90, 110.52, and 105.98, respectively. Reflection plus dual memory yields 83.50, 102.26, 115.19, and 109.04. The reported interpretation is that removing both modules causes a large drop of 28.0 points on visual performance and 9.25 points on language performance, that reflection provides the bulk of gains, and that dual memory further improves both axes.
A further analysis varies reflection rounds. Increasing the maximum number of rounds from 1 to 3 improves visual accuracy from 80.0% to 85.0% and language relative score from 105.4% to 116.2%, while 4 rounds slightly degrade performance because of over-reflection and error accumulation. The framework therefore defaults to 3 rounds. Replacing GPT-4o with Gemini 2.5 Pro, Claude 3.7 Sonnet, or Grok-2-Vision reportedly keeps performance stable, with a 6.67-point visual range and a 12.07-point language range across backbones, which the authors present as evidence of framework robustness.
6. Qualitative behavior, limitations, and relation to EndoAgent
The qualitative analyses reported for EndoAgentBench emphasize multi-step correction, geometric validation, tool collaboration, and hallucination control. In lesion quantification, an initial detection may identify one polyp, after which reflection triggers verification through segmentation, revealing two polyps and correcting the count. In visual grounding, the system computes IoU between its detected box and candidate options, selecting the one with IoU approximately 0.93 in the illustrated case. In lesion removal and comprehensive analysis examples, segmentation feeds an editing tool, and sequential invocation of classification, detection, and VQA supports detailed caption or report generation. The paper attributes consistency gains to memory maintaining action traces and reflections, thereby avoiding redundant loops (Tang et al., 10 Aug 2025).
These observations are specifically about how the benchmark surfaces agentic behavior rather than about benchmark items themselves. EndoAgentBench is used strictly for evaluation and is not used to train the agent core. EndoAgent instead relies on external expert tools such as AFACNet, YOLOv8, UniMed, ColonGPT, and Polyp-Gen, trained on separate datasets, while GPT-4o and alternatives provide reasoning. In this sense, the benchmark functions as a probe of coordination, memory use, and iterative refinement over heterogeneous endoscopic tasks.
The benchmark’s limitations are explicit. Domain coverage is concentrated on colonoscopy still images and polyp-centric pathology, with broader gastrointestinal anatomy, additional modalities such as NBI or chromoendoscopy, and video sequences underrepresented. The use of Qwen-VL-Plus both for generating language references and for automated evaluation introduces potential model-specific bias, although the paper notes that pairwise, criteria-based prompts partially mitigate this. Safety constraints remain central: captioning and report generation encourage image-grounded language, but the outputs are not substitutes for clinical judgment, and recommendations may lack patient context such as history, comorbidities, risk factors, bowel preparation quality, or procedural steps. The predominance of private clinical data enhances authenticity but may introduce site or device bias, and demographic and acquisition details are not disclosed (Tang et al., 10 Aug 2025).
The authors suggest future extensions including continual learning, richer self-reflection, expanded task coverage such as step identification and quality control, additional modalities, and multi-image or video cases. This suggests that EndoAgentBench currently occupies a transitional position: it is broader than narrowly perceptual lesion benchmarks, yet still deliberately constrained to single-image, single-turn evaluation so that multimodal clinical reasoning can be benchmarked in a standardized and reproducible manner.