EndoAgent: Memory-Guided Endoscopic AI

Updated 8 July 2026

EndoAgent is a memory-guided reflective agent framework that facilitates multi-step vision-to-decision reasoning in endoscopy by coordinating specialized tools and leveraging a dual-memory design.
The architecture integrates iterative reasoning with both short-term memory for action tracking and long-term reflective memory to enhance clinical decision-making.
Benchmark results show significant improvements in lesion classification, visual grounding, and language generation compared to existing endoscopic analysis models.

Searching arXiv for papers directly about “EndoAgent” and closely related endoscopic agent systems. First, a direct search for the exact term. Searching arXiv for: EndoAgent EndoAgent is a memory-guided reflective agent framework for intelligent endoscopic vision-to-decision reasoning that formulates endoscopic analysis as a multi-step, tool-augmented process rather than a one-shot image-recognition task. In its specific 2025 formulation, it receives an endoscopic image and a clinical query, iteratively selects among expert tools, records action traces in short-term memory, stores reflective feedback in long-term memory, and returns a task-appropriate decision, explanation, or report. More broadly, the term also names an architectural pattern in which endoscopic AI systems couple perception, reasoning, tool use, and feedback across diagnosis, navigation, and intervention (Tang et al., 10 Aug 2025).

1. Definition and conceptual scope

EndoAgent is defined around vision-to-decision reasoning: converting visual endoscopic evidence into clinically useful decisions through multi-round reasoning and tool use. The framework is motivated by the observation that realistic endoscopic workflows do not stop at lesion recognition. They require a sequence of subtasks that may include identifying lesions, localizing them, counting or characterizing them, producing a textual description, and generating a structured report. The paper therefore contrasts EndoAgent with both task-specific endoscopy models, which have limited transferability, and large-scale pretrained foundational or multimodal models, which remain largely limited to single-step visual recognition, task-switching without structured coordination, weak multi-step reasoning, and inability to manage end-to-end diagnostic workflows (Tang et al., 10 Aug 2025).

The 2025 system claims to be the first memory-guided agent for vision-to-decision endoscopic analysis and, more broadly, the first agent-based model for endoscopic analysis. Its core capabilities are iterative reasoning, adaptive tool use, collaboration among specialized modules, reflective behavior, and final decision support. The architecture is therefore modular rather than monolithic: the agent is not itself a single end-to-end vision backbone, but an LLM-centered orchestration layer coordinating visual tools and language generation (Tang et al., 10 Aug 2025).

A broader reading of “EndoAgent” is also present across adjacent literature. In that wider sense, the term denotes endoscopic systems that perceive scene state, decide how to intervene, and use task outcomes as feedback. AgentPolyp, for example, is described as a focused instance of such an endoscopic agent system for polyp segmentation under poor imaging conditions, with a perception–action–feedback loop over image enhancement and segmentation (Wang et al., 15 Apr 2025). SurgVisAgent generalizes the same pattern to multimodal reasoning over low light, overexposure, motion blur, and smoke, routing frames to task-specific enhancement tools (Lei et al., 3 Jul 2025). In bronchoscopy, a different but related formulation appears as hierarchical long-short agents for pure-vision navigation using preoperative CT-derived virtual targets and live endoscopic video (Wu et al., 9 Mar 2026). These systems are not the same model, but they instantiate the same shift from static prediction to structured endoscopic agency.

2. Core architecture and reasoning loop

EndoAgent consists of three tightly coupled modules: Actor, Evaluator, and Self-Reflection. These operate inside an iterative loop with a modular toolset. Given a clinical query $Q$ , an endoscopic image $I$ , a toolset $T$ , and a maximum number of rounds $N$ , the initial context is defined as

$\mathrm{context}_0 = (Q,\,I)$

with empty short-term and long-term memories,

$M_s^0 = \emptyset, \quad M_l^0 = \emptyset.$

At each step $t$ , the Actor selects a tool conditioned on the current context and both memories: $\text{tool}_t = \mathrm{SelectTool}(\mathrm{context}_{t-1},\,M_s^{t-1},\,M_l^{t-1},\,T),$ then invokes it: $\text{output}_t = \text{tool}_t \cdot \mathrm{invoke}(\mathrm{context}_{t-1}).$ The resulting action-output pair is appended to short-term memory, the Self-Reflection module generates reflective feedback,

$\mathrm{reflection}_t = \mathrm{LLMreflection}(\mathrm{context}_{t-1},\,M_s^t,\,M_l^{t-1}),$

and long-term memory is updated accordingly. The context for the next round is then

$I$ 0

with final output

$I$ 1

The stopping rule is heuristic: if the current output contains keywords semantically similar to “finish,” reasoning stops; otherwise the loop continues until the round budget is exhausted. The main method uses a maximum of 3 rounds, and Appendix B.2 confirms inference with up to 3 reflection rounds per case (Tang et al., 10 Aug 2025).

This formalism is intentionally lightweight. The paper does not provide a training loss for the agent loop, a policy optimization objective, a probabilistic stopping rule, a confidence-gating formula, or a memory retrieval similarity function. What is specified is a symbolic control loop over context, action traces, and reflections. A common misunderstanding is to treat EndoAgent as a fully formalized optimization framework; the paper instead specifies an executable reasoning workflow with explicit memory objects and tool calls (Tang et al., 10 Aug 2025).

3. Dual-memory design and expert tool collaboration

The defining architectural feature is the dual-memory design. Short-term memory $I$ 2 stores the current case’s ordered action trace,

$I$ 3

and is used to maintain context, avoid redundant actions, and preserve a traceable reasoning chain. Long-term memory $I$ 4 stores reflective experience from previous rounds of the same case: $I$ 5 where each entry includes round index, error analysis, optimization suggestion, and distilled experience. The paper explicitly states that short-term memory supports logical coherence and traceability, whereas long-term memory provides accumulated experience to refine subsequent tool selection and reasoning strategy. It also explicitly notes a limitation: this is not persistent cross-case continual learning; the “learning” is reflective accumulation across reasoning rounds within the episode (Tang et al., 10 Aug 2025).

The tool layer spans six core tasks: lesion classification, detection, segmentation, image editing, visual question answering, and medical report generation. The default implementation integrates expert-designed tools rather than retraining them end-to-end.

Tool	Purpose	Model
Classification	Lesion category	AFACNet
Detection	Lesion localization	YOLOv8
Segmentation	Pixel-level delineation	UniMed
VQA	Clinical question answering	ColonGPT
Image editing	Synthetic lesion generation/removal	Polyp-Gen
Report generation	Final synthesis	GPT-4o

AFACNet is trained on 4,591 endoscopic images including normal, polyp, adenoma, and cancer. YOLOv8 is fine-tuned on the SUN dataset. UniMed is trained on over 490,000 images. Polyp-Gen is a Stable Diffusion-based model fine-tuned on LDPolypVideo. GPT-4o is both the central LLM backbone and the report-generation engine. The system is implemented with LangChain and LangGraph, and the paper states that GPT-4o can be replaced by Gemini, Claude, or Grok without retraining (Tang et al., 10 Aug 2025).

“Collaboration” in EndoAgent refers primarily to collaboration among specialized tools under one agent’s orchestration, not to a multi-agent society. The paper’s canonical examples are segmentation plus image editing for lesion removal, classification plus detection plus VQA for comprehensive analysis, and detection followed by segmentation for verification. A second common misunderstanding is therefore to read EndoAgent as a collection of autonomous peer agents; in the paper, it is a single reasoning agent coordinating multiple expert modules (Tang et al., 10 Aug 2025).

4. EndoAgentBench and empirical performance

To evaluate unified endoscopic agents, the paper introduces EndoAgentBench, a benchmark of 5,709 visual question-answer pairs designed to assess fine-grained visual understanding and open-ended language generation in realistic endoscopic scenarios. It is built from six public endoscopic datasets—CVC-300, CVC-ClinicDB, CVC-ColonDB, Kvasir-SEG, ETIS-LaribPolypDB, SUN-SEG—plus a private expert-annotated clinical dataset. Source counts are: Private 3558, CVC-300 60, CVC-ClinicDB 62, CVC-ColonDB 380, ETIS-LaribPolypDB 196, Kvasir 99, and SUN-SEG 1354. Public data make up 37.7% and private data 62.3%. The lesion distribution is Normal 855 (15.0%), Polyp 2994 (52.4%), Adenoma 896 (15.7%), and Cancer 964 (16.9%) (Tang et al., 10 Aug 2025).

The benchmark covers five representative subtasks: Lesion Classification, Lesion Quantification, Visual Grounding, Image Caption, and Report Generation. Task counts are Image Caption 1064, Report Generation 1066, Lesion Classification 884, Visual Grounding 1319, and Lesion Quantification 1376. Fine-grained visual understanding accounts for 61.2% of the benchmark and open-ended language generation for 38.8%. Visual grounding annotations use public boxes where available, or minimum enclosing rectangles extracted from connected components when only segmentation masks existed. Report generation examples typically include Endoscopic Findings, Clinical Significance, and Recommendation (Tang et al., 10 Aug 2025).

Visual tasks are evaluated with accuracy. Language tasks are judged by Qwen-VL-Plus across seven clinical dimensions—diagnostic accuracy, clinical structure, medical terminology, detailed description, clinical significance, recommendations, and professional quality—with each dimension scored from 0 to 10 and aggregated as a relative score,

$I$ 6

This metric can exceed 100%, which explains why some ablation language scores are above 100 despite main-table averages being lower (Tang et al., 10 Aug 2025).

Main quantitative results show the following scores: Lesion Classification 88.46, Lesion Quantification 84.16, Visual Grounding 83.47, and Visual Avg. 84.97. On language tasks, the system reports CAP 100.32, MRG 95.90, and Language Avg. 97.83. The strongest visual baseline was Gemini 2.5 Pro at 74.57 visual average, so EndoAgent improves the visual average by 10.40. For lesion classification the second-best baseline was 68.44, yielding an improvement of 20.02. On language tasks, EndoAgent is not best on CAP alone—Gemini 2.5 Pro scores 104.83 there—but it achieves the best overall language average, ahead of GPT-4o at 94.76 (Tang et al., 10 Aug 2025).

The implementation is based on PyTorch, with GPT-4o as the core inference model, temperature 0.7, top-p 0.95, maximum output length 2048 tokens, up to 5 retries, and hardware specified as an NVIDIA GeForce RTX 4090 GPU (49GB VRAM). Appendix B reports up to 3 reflection rounds per case. Reflection and memory are empirically important. Table 2 reports that No reflection, no dual-memory yields visual average 55.50 and language average 99.79; Reflection only yields 82.00 and 105.98; and Reflection + dual-memory yields 83.50 and 109.04. A reflection-round study shows visual performance rising from 80.0% at 1 round to 85.0% at 3 rounds and language score rising from 105.4% to 116.2% at 3 rounds, with 4 rounds slightly hurting due to over-reflection or error accumulation. This is the empirical basis for the default choice of 3 rounds (Tang et al., 10 Aug 2025).

Although EndoAgent is most precisely the memory-guided framework of 2025, adjacent work shows that “EndoAgent” also functions as a broader systems concept spanning perception, navigation, and intervention.

On the perception side, AgentPolyp argues that endoscopic AI should not treat image enhancement as a fixed preprocessing step, but should first perceive the degradation state of the current frame, choose enhancement behavior adaptively, and validate whether that choice helped the downstream segmentation objective. Its CLIP-guided perception, action selection over seven enhancement operations, and reward defined approximately as $I$ 7 make it an early task-specific perception loop for degraded colonoscopy images (Wang et al., 15 Apr 2025). SurgVisAgent generalizes this into an MLLM-based surgical vision agent that dynamically identifies distortion categories and severity levels and routes frames to low-light enhancement, overexposure correction, motion blur removal, or smoke removal. It reports benchmark performance including SSIM 0.9220, PSNR 31.95, and LPIPS 0.0746, and shows how domain priors plus CoT prompting can turn visual restoration into a diagnosis-and-routing problem (Lei et al., 3 Jul 2025).

On the navigation side, Long-Short Term Agents for Pure-Vision Bronchoscopy Robotic Autonomy provides a preclinical realization of embodied endoscopic agency: a short-term reactive agent for low-latency motion control, a long-term strategic agent for ambiguous branch points, and a world-model critic for conflict resolution. It reports that the system reached all planned segmental targets in a high-fidelity airway phantom, maintained 80% success to the eighth generation ex vivo, and achieved 7/7 target-reaching success in a live porcine model, all without external tracking during navigation (Wu et al., 9 Mar 2026). In endovascular robotics, flow-driven navigation and anatomy-specific device preparation broaden the same concept into the vascular domain. A guidewire shaping robot can produce anatomy-specific C, S, Angled, Hook, and 3D helical tip shapes with 0.56 mm RMS error across shapes, while a flow-driven magnetic microcatheter demonstrates access to arteries as small as 180 µm and curvature radius as small as 0.69 mm, with superselective contrast and embolic infusion in vivo (Tamhankar et al., 3 Feb 2026, Pancaldi et al., 7 Aug 2025).

On the intervention side, modular robotic hardware shows what an embodied EndoAgent would require beyond perception and navigation. The tendon-actuated concentric tube endonasal robot supports follow-the-leader motion through a natural nasal corridor and provides a Cosserat-rod model driven by tendon tensions and relative translation, with cadaver demonstration from nostril to sphenoid sinus (Yamamoto et al., 28 Apr 2025). A modular soft robotic catheter integrates sensing, magnetic actuation, anchoring, compliant grasping, and ultrasound-triggered drug release in a 1.47 mm outer-diameter system, and demonstrates 7.5 cm navigation inside the pancreatic duct plus semi-autonomous deployment in vivo (Calmé et al., 21 Jan 2026). These are not EndoAgent in the narrow 2025 software sense, but they define the hardware substrate that a more general endoluminal agent would need.

6. Limitations, misconceptions, and future directions

The strongest limitations of EndoAgent are explicit. Long-term memory is reflective experience within reasoning cycles, not persistent lifelong learning. The stopping criterion is heuristic, based on “finish”-like semantics. The system depends heavily on external tools and on GPT-4o-like LLM backbones. Benchmark reference generation for captioning and report tasks partly relies on Qwen-VL-Plus with lesion prior knowledge, and language evaluation also uses an LLM judge rather than clinician adjudication. The benchmark is 62.3% private clinical data, which improves realism but limits reproducibility beyond released artifacts. Train/validation/test split details are not specified in the paper text. No formal safety validation, prospective clinical study, or regulatory analysis is provided (Tang et al., 10 Aug 2025).

Several common misconceptions follow from these omissions. EndoAgent is not a single end-to-end multimodal vision model; it is a modular orchestration layer. Its “long-term memory” is not cross-case continual learning. Its “collaboration” is not a multi-agent society but collaboration among specialized tools. Its mathematical formalism is workflow-level rather than optimization-level. These clarifications matter because the paper’s strongest contribution is system architecture, not a new foundation model or a new RL algorithm (Tang et al., 10 Aug 2025).

Future directions are implicit both in the EndoAgent paper and in adjacent work. For software agents, they include more advanced self-reflection, continual learning, stronger tool grounding, and more rigorous safety logic. For visual agents, a likely next step is tighter integration between quality assessment, enhancement, and downstream clinical tasks, as suggested by AgentPolyp and SurgVisAgent (Wang et al., 15 Apr 2025, Lei et al., 3 Jul 2025). For embodied systems, the missing elements are broader temporal reasoning, distal sensing, closed-loop control, and anatomical planning, as emphasized by bronchoscopy autonomy, endonasal robotics, and modular soft endoluminal catheters (Wu et al., 9 Mar 2026, Yamamoto et al., 28 Apr 2025, Calmé et al., 21 Jan 2026). In that broader trajectory, EndoAgent marks a shift in endoscopic AI from isolated predictors toward coordinated, interpretable, tool-using systems that move from image to clinically actionable decision.