EndoFinder: Explainable Polyp Retrieval

• EndoFinder is a retrieval-based framework that embeds polyp images into a learned feature space to provide digital twin comparisons and explainable diagnoses.
• The system uses a ViT-based self-supervised encoder with polyp-aware masking and hash-based retrieval to achieve real-time performance and enhanced diagnostic accuracy.
• The 2025 extension incorporates multi-view scene representations via a Scene Representation Transformer that improves lesion re-identification and pathology classification.

EndoFinder is a content-based image retrieval framework for explainable colorectal polyp diagnosis during colonoscopy. Rather than emitting only a supervised class label, it embeds a query polyp into a learned feature space, retrieves visually and semantically similar historical polyps as “digital twins,” and infers the query label from the retrieved cases and their known pathology. The framework was introduced as an online image retrieval system for explainable optical biopsy in 2024 and was later extended in 2025 to online lesion retrieval with latent multi-view scene representations, a Scene Representation Transformer, and hash-based retrieval for near real-time use (Yang et al., 2024, Yang et al., 23 Jul 2025).

1. Clinical rationale and conceptual basis

EndoFinder addresses a specific decision point in colonoscopy: whether a detected polyp should be resected immediately, biopsied, or left under surveillance. Histopathology remains the gold standard, but it is time-consuming and costly, and its results are not available during the procedure. Optical biopsy systems based on deep learning have shown strong performance, but the central critique in the EndoFinder papers is that most such systems behave as supervised black-box classifiers, require large labeled datasets, need retraining as data evolve, and provide limited case-level explanation (Yang et al., 2024, Yang et al., 23 Jul 2025).

The framework therefore reorients diagnosis from direct classification toward retrieval-based decision support. In this formulation, the clinically relevant output is not only a predicted label, but also a ranked set of prior cases judged similar in a learned embedding space. The intended interpretive mechanism is analogical: a clinician can compare the query lesion with retrieved lesions whose histology and prior management are known. The 2024 paper describes this as finding a “digital twin” polyp in a reference database, and the 2025 paper generalizes the same principle to online lesion retrieval based on latent scene representations (Yang et al., 2024, Yang et al., 23 Jul 2025).

This retrieval-centered design also reframes explainability. Saliency methods such as CAM or Grad-CAM indicate influential pixels, but EndoFinder emphasizes case-based reasoning: similar lesions with known outcomes are presented directly to the endoscopist. A plausible implication is that the framework’s notion of explanation is closer to clinical precedent matching than to post hoc visualization.

2. Retrieval pipeline and decision rule

At inference time, the 2024 formulation takes an endoscopic image or crop containing a detected polyp, applies a universal polyp-aware encoder $E_\phi$, and produces an embedding $z = E_\phi(I) \in \mathbb{R}^d$. For fast retrieval, this embedding can be quantized to a binary code $\bar{z} \in \{-1, +1\}^d$. The embedding or hash is then compared against a reference database $S = \{(I_i, y_i)\}_{i=1}^N$, and the $K$ nearest neighbors are returned. Diagnosis is obtained by majority voting over the neighbor labels (Yang et al., 2024).

The majority-vote rule is written as

$$y_i = \underset{c \in \{1,\dots,C\}}{\mathrm{argmax}} \; \sum_{k \in \mathcal{N}(I_i)} \mathbf{1}_{\{y_k = c\}},$$

where $\mathcal{N}(I_i)$ denotes the retrieved neighbors and $\mathbf{1}_{\{y_k = c\}}$ is 1 when neighbor $k$ has class $c$. The database is therefore not only an index structure but also the operational knowledge base of the system (Yang et al., 2024).

The 2025 formulation preserves the same retrieval logic but changes the query representation. Instead of a single image, the input is typically a small set of views of the same polyp,

$z = E_\phi(I) \in \mathbb{R}^d$0

usually 2–4 views from different angles or distances. These views are fused into a latent scene representation

$z = E_\phi(I) \in \mathbb{R}^d$1

which is then binarized by a sign function to obtain a hash code $z = E_\phi(I) \in \mathbb{R}^d$2. Retrieval is carried out with a ball tree index using Hamming distance, and diagnosis again follows $z = E_\phi(I) \in \mathbb{R}^d$3-nearest-neighbor majority voting (Yang et al., 23 Jul 2025).

Formulation	Query unit	Retrieval representation
2024 EndoFinder	Single polyp image or crop	Continuous embedding $z = E_\phi(I) \in \mathbb{R}^d$4 or binary hash $z = E_\phi(I) \in \mathbb{R}^d$5
2025 EndoFinder	Multi-view lesion set	Scene embedding $z = E_\phi(I) \in \mathbb{R}^d$6 or binary hash $z = E_\phi(I) \in \mathbb{R}^d$7

The two versions differ mainly in representational granularity. The 2024 system is a single-view CBIR framework specialized for endoscopy, whereas the 2025 system treats each polyp as a three-dimensional “scene” and retrieves at the lesion level rather than the frame level.

3. Polyp-aware representation learning

The core encoder in the 2024 EndoFinder is a ViT-based model trained self-supervised on Polyp-18k, a dataset of 17,969 polyp images with segmentation masks. Its design combines masked image modeling with contrastive learning, and its principal architectural distinction is a segmentation-mask-guided masking strategy during MAE pretraining. Background patches are masked more heavily, while polyp regions are masked less aggressively, making the encoder polyp-centric rather than uniformly image-centric (Yang et al., 2024).

The single-view embedding is produced from the CLS token by a linear projection followed by $z = E_\phi(I) \in \mathbb{R}^d$8 normalization: $z = E_\phi(I) \in \mathbb{R}^d$9 Pretraining uses the joint loss

$\bar{z} \in \{-1, +1\}^d$0

The MAE term reconstructs masked pixels only on masked regions,

$\bar{z} \in \{-1, +1\}^d$1

while the contrastive branch employs an InfoNCE-style objective with an additional entropy-like regularizer that penalizes overly small distances to near negatives (Yang et al., 2024).

This architecture is explicitly intended to create a generic embedding space that is usable across downstream clinical tasks without task-specific finetuning. In the 2024 experiments, the same encoder supports both polyp re-identification and retrieval-based optical biopsy by changing only the labels attached to the indexed database and the decision rule applied to retrieved neighbors (Yang et al., 2024).

The 2025 paper retains the same broad strategy but specifies the backbone as ViT-L/16 and trains a “Polyp-aware Image Encoder” with a joint contrastive and masked reconstruction objective: $\bar{z} \in \{-1, +1\}^d$2 Its masking policy again preserves polyp regions while masking background more heavily, and it uses SSCD’s “advanced+mixup” augmentation strategy, a masking ratio of 50%, AdamW with learning rate $\bar{z} \in \{-1, +1\}^d$3, weight decay $\bar{z} \in \{-1, +1\}^d$4, cosine annealing, temperature $\bar{z} \in \{-1, +1\}^d$5, effective batch size 64, and 100 epochs on Polyp-18k (Yang et al., 23 Jul 2025).

A consistent methodological point across both versions is that representation learning is largely decoupled from pathology labels. This suggests that EndoFinder is intended to reduce dependence on fully annotated histopathology datasets while still producing semantically useful retrieval spaces.

4. Multi-view scene representations

The major conceptual extension in the 2025 system is the treatment of a polyp as a multi-view scene rather than as an isolated image. The paper argues that polyps are three-dimensional structures seen from different angles, distances, and lighting conditions, and that single frames may be ambiguous or incomplete. To address this, EndoFinder introduces a Scene Representation Transformer that fuses multiple views into a single latent scene embedding (Yang et al., 23 Jul 2025).

The pipeline is staged. First, each view $\bar{z} \in \{-1, +1\}^d$6 is processed by the frozen polyp-aware image encoder $\bar{z} \in \{-1, +1\}^d$7, producing a global feature $\bar{z} \in \{-1, +1\}^d$8. These view embeddings are then passed to a Scene Encoder $\bar{z} \in \{-1, +1\}^d$9, a Transformer with a learnable [Scene] token that aggregates scene-level information. The output is the scene representation $S = \{(I_i, y_i)\}_{i=1}^N$0 (Yang et al., 23 Jul 2025).

Training is again self-supervised, but now at the scene level. For each polyp with $S = \{(I_i, y_i)\}_{i=1}^N$1 views, the method forms $S = \{(I_i, y_i)\}_{i=1}^N$2 combinations in which one view becomes the reconstruction target and the remaining $S = \{(I_i, y_i)\}_{i=1}^N$3 views define the scene embedding. Combinations from the same polyp are treated as mutual positives; combinations from different polyps are negatives. The scene-level InfoNCE term is

$S = \{(I_i, y_i)\}_{i=1}^N$4

with

$S = \{(I_i, y_i)\}_{i=1}^N$5

An entropy term pushes apart nearest non-positive scene embeddings, and a cross-attention reconstruction term reconstructs masked patches in a target view using both the visible tokens of that view and the global scene representation (Yang et al., 23 Jul 2025).

The total scene objective is

$S = \{(I_i, y_i)\}_{i=1}^N$6

The scene transformer is trained on PolypScene-2k, which contains 2,219 polyps with 4 views each. Scene-based evaluation is then performed on PolypScene-250 for re-identification and on PolypScene-80 for pathology classification (Yang et al., 23 Jul 2025).

The paper explicitly notes that this scene encoder learns geometry only implicitly; it does not explicitly reconstruct 3D pose or geometry. A plausible implication is that the framework occupies a middle ground between frame-based CBIR and explicit 3D endoscopic reconstruction.

5. Empirical performance

The 2024 paper evaluates EndoFinder on two downstream tasks: polyp re-identification on Polyp-Twin and optical biopsy on Polyp-Path. On Polyp-Twin, EndoFinder-Raw achieves uAP 0.695, Acc@1 0.693, Recall@90% 0.495, time 0.456 s, and 2.19 FPS, while EndoFinder-Hash achieves uAP 0.693, Acc@1 0.693, Recall@90% 0.524, time 0.009 s, and 108.57 FPS. On Polyp-Path, EndoFinder-Raw reaches ACC 77.24, SEN 81.24, SPE 73.75, and F1 80.45, and EndoFinder-Hash reaches ACC 73.79, SEN 81.92, SPE 63.92, and F1 78.21. The paper emphasizes that EndoFinder-Raw slightly exceeds the best supervised classifier in ACC and F1 on this task, despite using no supervised training on Polyp-Path for the encoder (Yang et al., 2024).

The 2025 paper reports a progression from single-view to multi-view retrieval. On Polyp-Twin, EndoFinder-I achieves $S = \{(I_i, y_i)\}_{i=1}^N$7AP 0.67, Acc@1 0.70, and Recall@P90 0.56. On PolypScene-250, average fusion with EndoFinder-I gives $S = \{(I_i, y_i)\}_{i=1}^N$8AP 0.68, Acc@1 0.70, and Recall@P90 0.58, while transformer fusion with EndoFinder-S improves these to $S = \{(I_i, y_i)\}_{i=1}^N$9AP 0.71, Acc@1 0.74, and Recall@P90 0.59. On PolypScene-80, EndoFinder-S with $K$0-NN yields AUC 85.59, ACC 78.75, and F1 81.00, which the paper reports as the best AUC and F1, with ACC equal to supervised ViT-L fine-tuning (Yang et al., 23 Jul 2025).

Evaluation	System	Reported result
Polyp-Twin re-ID	EndoFinder-Raw	uAP 0.695, Acc@1 0.693, Recall@90% 0.495
Polyp-Twin re-ID	EndoFinder-Hash	uAP 0.693, Acc@1 0.693, Recall@90% 0.524
Polyp-Path optical biopsy	EndoFinder-Raw	ACC 77.24, SEN 81.24, SPE 73.75, F1 80.45
PolypScene-250 re-ID	EndoFinder-S	$K$1AP 0.71, Acc@1 0.74, Recall@P90 0.59
PolypScene-80 pathology classification	EndoFinder-S + k-NN	AUC 85.59, ACC 78.75, F1 81.00

These evaluations support two distinct claims. First, a retrieval-based system can match or exceed supervised black-box classifiers on binary pathology tasks when the embedding is sufficiently specialized. Second, multi-view scene fusion produces measurable gains beyond both single-view retrieval and naive averaging across views.

Hashing is important in both versions. In 2024, the hash variant reduces retrieval time from approximately 0.45 seconds per query to approximately 0.009 seconds per query on a database of 12k+ images with negligible change in retrieval quality. In 2025, post hoc sign quantization plus ball-tree indexing yields about a 4× speed-up on PolypScene-250, with EndoFinder-S losing about 6.7 percentage points in $K$2AP after binarization while still outperforming weaker baselines (Yang et al., 2024, Yang et al., 23 Jul 2025).

6. Explainability, deployment, and relation to adjacent endoscopy systems

EndoFinder’s explainability is operational rather than purely visual. The clinician is presented with a gallery consisting of the query image or query multi-view set and the top retrieved neighbors, together with the neighbors’ pathology labels and, in the 2025 presentation, potentially additional metadata. The system’s logic is therefore inspectable: if the retrieved neighbors are visually convincing and their labels are concordant, the diagnostic suggestion is correspondingly persuasive; if the retrievals appear inconsistent, the clinician can discount the output (Yang et al., 2024, Yang et al., 23 Jul 2025).

The deployment model assumes that detection is already available. The 2024 paper states that polyp detection per se is not the focus and assumes polyp images or patches are provided. The 2025 paper similarly describes online use in which a small set of views is collected around a candidate lesion, encoded on the fly, hashed, and queried against a pre-built database. Both versions emphasize that new confirmed cases can be added to the database without retraining the encoder, and the 2025 paper further notes that deleting a patient’s data removes that knowledge immediately from the system because the retrieval engine is database-centric rather than solely model-centric (Yang et al., 2024, Yang et al., 23 Jul 2025).

Two adjacent endoscopy systems are especially relevant to the broader EndoFinder ecosystem. “EndoBoost: a plug-and-play module for false positive suppression during computer-aided polyp detection in real-world colonoscopy” introduces a post-hoc module for detector proposals based on a ResNet-50 feature extractor and a normalizing flow density estimator. It is designed to filter false positives in open-world settings and is explicitly described as attachable to any polyp detector, which makes it directly compatible with a workflow in which EndoFinder operates downstream of detection (Wang et al., 2022).

“EndoDINO: A Foundation Model for GI Endoscopy” provides a GI-endoscopy-specific DINOv2 ViT family trained on images sampled from 130,037 GI videos and is presented as a frozen feature encoder for multiple downstream tasks, including anatomical landmark classification, polyp segmentation, and Mayo endoscopic scoring. The paper explicitly positions EndoDINO as a representation layer on which search, retrieval, indexing, and multi-task inference can be built. This suggests that future EndoFinder variants could substitute or augment the current polyp-aware encoder with a broader GI foundation model when cross-task or cross-organ generality is required (Dermyer et al., 8 Jan 2025).

A plausible modular architecture therefore consists of detector, false-positive suppression, retrieval encoder, and indexed historical case base. The literature provided does not report a unified system integrating all of these components, but it does provide the constituent mechanisms.

7. Limitations and open directions

The EndoFinder papers are explicit about several limitations. Dataset scope is the most immediate. Polyp-18k is in-house, Polyp-Path contains 147 images, PolypScene-80 contains 80 polyps with binary pathology labels, and the 2025 scene datasets are from a single center or limited centers. Generalization across devices, hospitals, countries, and patient populations is therefore not fully established (Yang et al., 2024, Yang et al., 23 Jul 2025).

The pretraining strategy also depends on segmentation masks. Both versions rely on polyp-aware masking derived from segmentation masks, and the 2025 paper notes directly that segmentation errors can propagate into representation learning and that some settings may lack such annotations. Multi-view performance is strongest when several views are available, but routine practice may yield only one or two usable frames, making robustness to sparse and variable-quality views a continuing issue (Yang et al., 23 Jul 2025).

Task scope remains constrained. The 2024 optical-biopsy experiments are binary benign-versus-malignant classification on Polyp-Path, and the 2025 pathology experiments on PolypScene-80 also use binary labels. More granular histologic subtyping, multiclass pathology, temporal reasoning over video, and formal user studies of trust or workflow impact are not reported. The 2024 paper also notes that rare polyp types and cases with no close twin in the database may lead to mismatched retrievals, although the clinician can inspect and reject such suggestions (Yang et al., 2024).

The scene-based extension improves retrieval and classification, but it does not explicitly model pose or geometry. The 2025 paper states that geometry is learned only implicitly from multi-view consistency and suggests that explicit 3D reconstruction methods could be beneficial for some tasks. This suggests a future bifurcation of retrieval-based CADx: one branch centered on latent scene fusion, and another centered on explicit geometric reconstruction (Yang et al., 23 Jul 2025).

Across both formulations, EndoFinder exemplifies a transductive approach to medical AI. Knowledge is stored not only in parameters but in an indexed archive of historical lesions. That design reduces the need for repeated supervised retraining and provides clinically legible explanations, but it also makes dataset curation, archive diversity, and indexing policy central determinants of system behavior.