---
title: 'EndoAgent: Memory-Guided Endoscopic AI'
url: https://www.emergentmind.com/topics/endoagent
type: topic
---

# EndoAgent: Memory-Guided Endoscopic AI

Searching arXiv for recent papers directly about “EndoAgent” and closely related endoscopic agent systems.
First, a direct search for the exact term.
Searching arXiv for: EndoAgent
EndoAgent is a **memory-guided reflective agent framework for intelligent endoscopic vision-to-decision reasoning** that formulates endoscopic analysis as a multi-step, tool-augmented process rather than a one-shot image-recognition task. In its specific 2025 formulation, it receives an endoscopic image and a clinical query, iteratively selects among expert tools, records action traces in short-term memory, stores reflective feedback in long-term memory, and returns a task-appropriate decision, explanation, or report. More broadly, the term also names an architectural pattern in which endoscopic AI systems couple perception, reasoning, tool use, and feedback across diagnosis, navigation, and intervention [2508.07292].

## 1. Definition and conceptual scope

EndoAgent is defined around **vision-to-decision reasoning**: converting visual endoscopic evidence into clinically useful decisions through multi-round reasoning and tool use. The framework is motivated by the observation that realistic endoscopic workflows do not stop at lesion recognition. They require a sequence of subtasks that may include identifying lesions, localizing them, counting or characterizing them, producing a textual description, and generating a structured report. The paper therefore contrasts EndoAgent with both **task-specific endoscopy models**, which have limited transferability, and **large-scale pretrained foundational or multimodal models**, which remain largely limited to single-step visual recognition, task-switching without structured coordination, weak multi-step reasoning, and inability to manage end-to-end diagnostic workflows [2508.07292].

The 2025 system claims to be the **first memory-guided agent for vision-to-decision endoscopic analysis** and, more broadly, the **first agent-based model for endoscopic analysis**. Its core capabilities are iterative reasoning, adaptive tool use, collaboration among specialized modules, reflective behavior, and final decision support. The architecture is therefore modular rather than monolithic: the agent is not itself a single end-to-end vision backbone, but an LLM-centered orchestration layer coordinating visual tools and language generation [2508.07292].

A broader reading of “EndoAgent” is also present across adjacent literature. In that wider sense, the term denotes endoscopic systems that perceive scene state, decide how to intervene, and use task outcomes as feedback. AgentPolyp, for example, is described as a focused instance of such an endoscopic agent system for polyp segmentation under poor imaging conditions, with a perception–action–feedback loop over image enhancement and segmentation [2504.10978]. SurgVisAgent generalizes the same pattern to multimodal reasoning over low light, overexposure, motion blur, and smoke, routing frames to task-specific enhancement tools [2507.02252]. In bronchoscopy, a different but related formulation appears as hierarchical long-short agents for pure-vision navigation using preoperative CT-derived virtual targets and live endoscopic video [2603.07909]. These systems are not the same model, but they instantiate the same shift from static prediction to structured endoscopic agency.

## 2. Core architecture and reasoning loop

EndoAgent consists of three tightly coupled modules: **Actor**, **Evaluator**, and **Self-Reflection**. These operate inside an iterative loop with a modular toolset. Given a clinical query \(Q\), an endoscopic image \(I\), a toolset \(T\), and a maximum number of rounds \(N\), the initial context is defined as
\[
\mathrm{context}_0 = (Q,\,I)
\]
with empty short-term and long-term memories,
\[
M_s^0 = \emptyset, \quad M_l^0 = \emptyset.
\]
At each step \(t\), the Actor selects a tool conditioned on the current context and both memories:
\[
\text{tool}_t = \mathrm{SelectTool}(\mathrm{context}_{t-1},\,M_s^{t-1},\,M_l^{t-1},\,T),
\]
then invokes it:
\[
\text{output}_t = \text{tool}_t \cdot \mathrm{invoke}(\mathrm{context}_{t-1}).
\]
The resulting action-output pair is appended to short-term memory, the Self-Reflection module generates reflective feedback,
\[
\mathrm{reflection}_t = \mathrm{LLMreflection}(\mathrm{context}_{t-1},\,M_s^t,\,M_l^{t-1}),
\]
and long-term memory is updated accordingly. The context for the next round is then
\[
\mathrm{context}_t = \mathrm{UpdateContext}(\mathrm{context}_{t-1},\, \mathrm{output}_t,\, \mathrm{reflection}_t),
\]
with final output
\[
O^* = \mathrm{output}_t.
\]
The stopping rule is heuristic: if the current output contains keywords semantically similar to “finish,” reasoning stops; otherwise the loop continues until the round budget is exhausted. The main method uses a maximum of **3** rounds, and Appendix B.2 confirms inference with **up to 3 reflection rounds per case** [2508.07292].

This formalism is intentionally lightweight. The paper does not provide a training loss for the agent loop, a policy optimization objective, a probabilistic stopping rule, a confidence-gating formula, or a memory retrieval similarity function. What is specified is a symbolic control loop over context, action traces, and reflections. A common misunderstanding is to treat EndoAgent as a fully formalized optimization framework; the paper instead specifies an executable reasoning workflow with explicit memory objects and tool calls [2508.07292].

## 3. Dual-memory design and expert tool collaboration

The defining architectural feature is the **dual-memory design**. Short-term memory \(M_s\) stores the current case’s ordered action trace,
\[
M_s^t = \left[(tool_1, output_1), \ldots, (tool_t,, output_t)\right],
\]
and is used to maintain context, avoid redundant actions, and preserve a traceable reasoning chain. Long-term memory \(M_l\) stores reflective experience from previous rounds of the same case:
\[
M_l^t = \left[ (r_1, e_1, s_1, x_1),\ldots, (r_t, e_t, s_t, x_t) \right],
\]
where each entry includes round index, error analysis, optimization suggestion, and distilled experience. The paper explicitly states that short-term memory supports logical coherence and traceability, whereas long-term memory provides accumulated experience to refine subsequent tool selection and reasoning strategy. It also explicitly notes a limitation: this is not persistent cross-case continual learning; the “learning” is reflective accumulation across reasoning rounds within the episode [2508.07292].

The tool layer spans **six core tasks**: lesion classification, detection, segmentation, image editing, visual question answering, and medical report generation. The default implementation integrates expert-designed tools rather than retraining them end-to-end.

| Tool | Purpose | Model |
|---|---|---|
| Classification | Lesion category | AFACNet |
| Detection | Lesion localization | YOLOv8 |
| Segmentation | Pixel-level delineation | UniMed |
| VQA | Clinical question answering | ColonGPT |
| Image editing | Synthetic lesion generation/removal | Polyp-Gen |
| Report generation | Final synthesis | GPT-4o |

AFACNet is trained on **4,591 endoscopic images** including normal, polyp, adenoma, and cancer. YOLOv8 is fine-tuned on the **SUN dataset**. UniMed is trained on **over 490,000 images**. Polyp-Gen is a Stable Diffusion-based model fine-tuned on LDPolypVideo. GPT-4o is both the central LLM backbone and the report-generation engine. The system is implemented with **LangChain** and **LangGraph**, and the paper states that GPT-4o can be replaced by Gemini, Claude, or Grok without retraining [2508.07292].

“Collaboration” in EndoAgent refers primarily to collaboration among specialized tools under one agent’s orchestration, not to a multi-agent society. The paper’s canonical examples are segmentation plus image editing for lesion removal, classification plus detection plus VQA for comprehensive analysis, and detection followed by segmentation for verification. A second common misunderstanding is therefore to read EndoAgent as a collection of autonomous peer agents; in the paper, it is a **single reasoning agent coordinating multiple expert modules** [2508.07292].

## 4. EndoAgentBench and empirical performance

To evaluate unified endoscopic agents, the paper introduces **EndoAgentBench**, a benchmark of **5,709 visual question-answer pairs** designed to assess **fine-grained visual understanding** and **open-ended language generation** in realistic endoscopic scenarios. It is built from six public endoscopic datasets—**CVC-300, CVC-ClinicDB, CVC-ColonDB, Kvasir-SEG, ETIS-LaribPolypDB, SUN-SEG**—plus a private expert-annotated clinical dataset. Source counts are: Private 3558, CVC-300 60, CVC-ClinicDB 62, CVC-ColonDB 380, ETIS-LaribPolypDB 196, Kvasir 99, and SUN-SEG 1354. Public data make up **37.7%** and private data **62.3%**. The lesion distribution is Normal 855 (**15.0%**), Polyp 2994 (**52.4%**), Adenoma 896 (**15.7%**), and Cancer 964 (**16.9%**) [2508.07292].

The benchmark covers five representative subtasks: **Lesion Classification**, **Lesion Quantification**, **Visual Grounding**, **Image Caption**, and **Report Generation**. Task counts are Image Caption 1064, Report Generation 1066, Lesion Classification 884, Visual Grounding 1319, and Lesion Quantification 1376. Fine-grained visual understanding accounts for **61.2%** of the benchmark and open-ended language generation for **38.8%**. Visual grounding annotations use public boxes where available, or minimum enclosing rectangles extracted from connected components when only segmentation masks existed. Report generation examples typically include **Endoscopic Findings**, **Clinical Significance**, and **Recommendation** [2508.07292].

Visual tasks are evaluated with **accuracy**. Language tasks are judged by **Qwen-VL-Plus** across seven clinical dimensions—diagnostic accuracy, clinical structure, medical terminology, detailed description, clinical significance, recommendations, and professional quality—with each dimension scored from 0 to 10 and aggregated as a relative score,
\[
S_\mathrm{model} / S_\mathrm{reference}(\%).
\]
This metric can exceed 100%, which explains why some ablation language scores are above 100 despite main-table averages being lower [2508.07292].

Main quantitative results show the following scores: **Lesion Classification 88.46**, **Lesion Quantification 84.16**, **Visual Grounding 83.47**, and **Visual Avg. 84.97**. On language tasks, the system reports **CAP 100.32**, **MRG 95.90**, and **Language Avg. 97.83**. The strongest visual baseline was Gemini 2.5 Pro at **74.57** visual average, so EndoAgent improves the visual average by **10.40**. For lesion classification the second-best baseline was **68.44**, yielding an improvement of **20.02**. On language tasks, EndoAgent is not best on CAP alone—Gemini 2.5 Pro scores **104.83** there—but it achieves the best overall language average, ahead of GPT-4o at **94.76** [2508.07292].

The implementation is based on **PyTorch**, with **GPT-4o** as the core inference model, temperature **0.7**, top-p **0.95**, maximum output length **2048 tokens**, **up to 5 retries**, and hardware specified as an **NVIDIA GeForce RTX 4090 GPU (49GB VRAM)**. Appendix B reports **up to 3 reflection rounds** per case. Reflection and memory are empirically important. Table 2 reports that **No reflection, no dual-memory** yields visual average **55.50** and language average **99.79**; **Reflection only** yields **82.00** and **105.98**; and **Reflection + dual-memory** yields **83.50** and **109.04**. A reflection-round study shows visual performance rising from **80.0%** at 1 round to **85.0%** at 3 rounds and language score rising from **105.4%** to **116.2%** at 3 rounds, with 4 rounds slightly hurting due to over-reflection or error accumulation. This is the empirical basis for the default choice of **3 rounds** [2508.07292].

## 5. Broader EndoAgent ecosystem across perception, navigation, and intervention

Although EndoAgent is most precisely the memory-guided framework of 2025, adjacent work shows that “EndoAgent” also functions as a broader systems concept spanning perception, navigation, and intervention.

On the perception side, **AgentPolyp** argues that endoscopic AI should not treat image enhancement as a fixed preprocessing step, but should first perceive the degradation state of the current frame, choose enhancement behavior adaptively, and validate whether that choice helped the downstream segmentation objective. Its CLIP-guided perception, action selection over seven enhancement operations, and reward defined approximately as \(0.9 \cdot \text{Dice} + 0.1 \cdot \mathcal{Q}(I')\) make it an early task-specific perception loop for degraded colonoscopy images [2504.10978]. **SurgVisAgent** generalizes this into an MLLM-based surgical vision agent that dynamically identifies distortion categories and severity levels and routes frames to low-light enhancement, overexposure correction, motion blur removal, or smoke removal. It reports benchmark performance including **SSIM 0.9220**, **PSNR 31.95**, and **LPIPS 0.0746**, and shows how domain priors plus CoT prompting can turn visual restoration into a diagnosis-and-routing problem [2507.02252].

On the navigation side, **Long-Short Term Agents for Pure-Vision Bronchoscopy Robotic Autonomy** provides a preclinical realization of embodied endoscopic agency: a short-term reactive agent for low-latency motion control, a long-term strategic agent for ambiguous branch points, and a world-model critic for conflict resolution. It reports that the system reached all planned segmental targets in a high-fidelity airway phantom, maintained **80% success to the eighth generation ex vivo**, and achieved **7/7** target-reaching success in a live porcine model, all without external tracking during navigation [2603.07909]. In endovascular robotics, flow-driven navigation and anatomy-specific device preparation broaden the same concept into the vascular domain. A guidewire shaping robot can produce anatomy-specific C, S, Angled, Hook, and 3D helical tip shapes with **0.56 mm** RMS error across shapes, while a flow-driven magnetic microcatheter demonstrates access to arteries as small as **180 µm** and curvature radius as small as **0.69 mm**, with superselective contrast and embolic infusion in vivo [2602.04050; 2508.05312].

On the intervention side, modular robotic hardware shows what an embodied EndoAgent would require beyond perception and navigation. The tendon-actuated concentric tube endonasal robot supports follow-the-leader motion through a natural nasal corridor and provides a Cosserat-rod model driven by tendon tensions and relative translation, with cadaver demonstration from nostril to sphenoid sinus [2504.19948]. A modular soft robotic catheter integrates sensing, magnetic actuation, anchoring, compliant grasping, and ultrasound-triggered drug release in a **1.47 mm** outer-diameter system, and demonstrates **7.5 cm** navigation inside the pancreatic duct plus semi-autonomous deployment in vivo [2601.14837]. These are not EndoAgent in the narrow 2025 software sense, but they define the hardware substrate that a more general endoluminal agent would need.

## 6. Limitations, misconceptions, and future directions

The strongest limitations of EndoAgent are explicit. Long-term memory is reflective experience within reasoning cycles, not persistent lifelong learning. The stopping criterion is heuristic, based on “finish”-like semantics. The system depends heavily on external tools and on GPT-4o-like LLM backbones. Benchmark reference generation for captioning and report tasks partly relies on **Qwen-VL-Plus** with lesion prior knowledge, and language evaluation also uses an LLM judge rather than clinician adjudication. The benchmark is **62.3% private clinical data**, which improves realism but limits reproducibility beyond released artifacts. Train/validation/test split details are not specified in the paper text. No formal safety validation, prospective clinical study, or regulatory analysis is provided [2508.07292].

Several common misconceptions follow from these omissions. EndoAgent is not a single end-to-end multimodal vision model; it is a modular orchestration layer. Its “long-term memory” is not cross-case continual learning. Its “collaboration” is not a multi-agent society but collaboration among specialized tools. Its mathematical formalism is workflow-level rather than optimization-level. These clarifications matter because the paper’s strongest contribution is system architecture, not a new foundation model or a new RL algorithm [2508.07292].

Future directions are implicit both in the EndoAgent paper and in adjacent work. For software agents, they include more advanced self-reflection, continual learning, stronger tool grounding, and more rigorous safety logic. For visual agents, a likely next step is tighter integration between quality assessment, enhancement, and downstream clinical tasks, as suggested by AgentPolyp and SurgVisAgent [2504.10978; 2507.02252]. For embodied systems, the missing elements are broader temporal reasoning, distal sensing, closed-loop control, and anatomical planning, as emphasized by bronchoscopy autonomy, endonasal robotics, and modular soft endoluminal catheters [2603.07909; 2504.19948; 2601.14837]. In that broader trajectory, EndoAgent marks a shift in endoscopic AI from isolated predictors toward coordinated, interpretable, tool-using systems that move from image to clinically actionable decision.

Source: https://www.emergentmind.com/topics/endoagent