Polyp-aware Image Encoder

• Polyp-aware Image Encoder is a module that prioritizes lesion structure in feature learning, ensuring polyp regions are emphasized over generic scene details.
• It integrates contrastive learning with masked reconstruction, using polyp segmentation masks to guide the preservation of critical lesion information.
• The resulting embeddings support diverse tasks including real-time segmentation, retrieval-based diagnosis, and multi-view scene encoding in endoscopic applications.

Polyp-aware Image Encoder denotes an image representation module for endoscopic polyp analysis whose feature space is explicitly biased toward lesion structure rather than generic scene content. In the EndoFinder framework, the term refers to a Vision Transformer ViT-L/16 encoder trained by combining contrastive learning and masked reconstruction, with polyp segmentation masks guiding which patches are preserved or masked, so that the resulting embeddings support re-identification, multi-view scene encoding, hashing, and retrieval-based diagnosis (Yang et al., 2024, Yang et al., 23 Jul 2025). In the broader polyp-analysis literature, closely related designs pursue the same objective through shallow residual hierarchies for real-time segmentation, pyramid transformers with foreground-aware refinement, lesion-aware dynamic kernels, healthy-manifold anomaly modeling, dual encoders, and depth-driven geometric priors (Tomar et al., 2023, Wei et al., 2023, Zhang et al., 2023, Ji et al., 2023, Vazquez et al., 24 Jan 2026).

1. Conceptual scope and defining properties

The term does not denote a single canonical architecture. In current usage, it refers to encoder behavior: lesion structure should dominate representation learning even when the input contains low inter-class variance between polyps and surrounding mucosa, blurred boundaries, specular highlights, variable illumination, bubbles, folds, or motion-related degradation. EndoFinder formulates the problem explicitly: polyps are often visually similar to the surrounding mucosa in color and texture, a single 2D frame is only a partial view of an inherently 3D structure, and reliance on fully supervised pathology labels limits scalability (Yang et al., 23 Jul 2025). Segmentation papers express the same pressure in different terms, emphasizing polyps’ diverse size, texture, color, pattern, and low contrast against background tissue (Patel et al., 2021, Wei et al., 2023).

This suggests that “polyp-aware” is best understood as an inductive bias rather than a fixed module type. Some methods enforce lesion-centricity by preserving polyp regions during masked reconstruction and masking background patches; some by conditioning the segmentation head on lesion features extracted from intermediate predictions; some by modeling healthy colon structure so that polyp tissue becomes out-of-distribution; and some by injecting geometric priors derived from depth maps into encoder skip features (Yang et al., 23 Jul 2025, Zhang et al., 2023, Ji et al., 2023, Vazquez et al., 24 Jan 2026).

A second defining property is task portability. The same lesion-focused representation is used for real-time segmentation, retrieval-based diagnosis, multi-view scene encoding, and wireless capsule endoscopy detection. EndoFinder treats the encoder output as a semantic descriptor for nearest-neighbor search and hashing rather than a direct classifier (Yang et al., 2024). WCE triplet-based detection similarly uses embedding geometry to separate polyp and non-polyp frames under severe imbalance and scarcity of data (Laiz et al., 2019).

2. Self-supervised, contrastive, and anomaly-centered formulations

The most explicit formulation appears in EndoFinder. Its polyp-aware image encoder uses a ViT-L/16 backbone with a learnable [CLS] token, a standard ViT encoder, and an MAE-style decoder. Training combines a SimCLR-like contrastive objective on the [CLS] embedding with masked reconstruction, while polyp segmentation masks guide the masking process so that informative polyp regions are preserved and background is preferentially masked. The total objective is written as

$$\mathcal{L}=\mathcal{L}_{CON}+\gamma\mathcal{L}_{MSE},$$

with $\mathcal{L}_{CON}$ built from InfoNCE plus entropy regularization and $\mathcal{L}_{MSE}$ defined on masked patches only (Yang et al., 23 Jul 2025). The pretraining dataset is Polyp-18k, consisting of 17,969 polyp images with corresponding segmentation masks. On Polyp-Twin, EndoFinder-I reports $\mu AP = 0.67$, Acc@1 $= 0.70$, and Recall@P90 $= 0.56$; on PolypScene-250, EndoFinder-S with a Scene Representation Transformer reaches $\mu AP = 0.71$, Acc@1 $= 0.74$, and Recall@P90 $= 0.59$ (Yang et al., 23 Jul 2025).

The earlier EndoFinder formulation presents the same idea as a content-based image retrieval encoder for explainable colorectal polyp diagnosis. It also combines masked image modeling with contrastive learning, uses adaptive masking biased toward background patches, and treats the normalized CLS projection as the retrieval embedding. In optical biopsy on Polyp-Path, retrieval with EndoFinder-Raw achieves ACC $77.241$, SEN $\mathcal{L}_{CON}$0, SPE $\mathcal{L}_{CON}$1, and F1 $\mathcal{L}_{CON}$2, outperforming the supervised classifiers reported in that study without fine-tuning the encoder on Polyp-Path (Yang et al., 2024).

A related segmentation-oriented contrastive variant is CL-Polyp. Its encoder is ResNet-50, but the polyp-aware behavior is imposed by a contrastive branch operating on deepest encoder features, a momentum encoder, and triplet loss built from positives and negatives defined by polyp number and size categories. The final objective is

$\mathcal{L}_{CON}$3

where $\mathcal{L}_{CON}$4. On Kvasir-SEG and CVC-ClinicDB, the paper reports IoU improvements of $\mathcal{L}_{CON}$5 and $\mathcal{L}_{CON}$6, respectively, over its baseline DeepLabV3+ configuration, and a final Kvasir-SEG result of Dice $\mathcal{L}_{CON}$7 and IoU $\mathcal{L}_{CON}$8 (Li et al., 9 Jul 2025).

A more radical self-supervised formulation abandons explicit lesion supervision altogether. “Rethinking Polyp Segmentation from an Out-of-Distribution Perspective” trains a ViT-Base/16 masked autoencoder only on healthy colonoscopy images, computes healthy latent statistics $\mathcal{L}_{CON}$9, standardizes abnormal latent features,

$\mathcal{L}_{MSE}$0

and then uses reconstruction error as a per-pixel anomaly score (Ji et al., 2023). On Kvasir-SEG it reports Spe $\mathcal{L}_{MSE}$1, $\mathcal{L}_{MSE}$2 $\mathcal{L}_{MSE}$3, and $\mathcal{L}_{MSE}$4 $\mathcal{L}_{MSE}$5, while also showing gains on CVC-ColonDB and ETIS-LaribPolypDB. This suggests that polyp-awareness can be induced by learning a healthy colon manifold and treating lesions as structured deviations rather than by directly encoding polyp masks (Ji et al., 2023).

Triplet-based embedding learning in WCE detection reaches a similar conclusion in a different modality. A ResNet-50 encoder trained with Batch All triplet loss attains AUC $\mathcal{L}_{MSE}$6, sensitivity at $\mathcal{L}_{MSE}$7 specificity of $\mathcal{L}_{MSE}$8, and sensitivity at $\mathcal{L}_{MSE}$9 specificity of $\mu AP = 0.67$0, outperforming both a plain ResNet and Batch Hard triplet mining in that study (Laiz et al., 2019). The paper’s CAM visualizations show that the learned embedding localizes activation on polyp regions or on non-polyp structures that closely mimic polyps, which is consistent with a lesion-sensitive representation (Laiz et al., 2019).

3. Segmentation-oriented backbone design

A lightweight real-time interpretation of a polyp-aware encoder appears in RUPNet. The encoder consists of three encoder blocks, each defined as a residual block followed by $\mu AP = 0.67$1 max-pooling, producing a resolution path $\mu AP = 0.67$2. The design is intentionally shallow relative to typical U-Net variants, on the premise that preserving high-resolution texture and boundary information is critical for small and low-contrast polyps (Tomar et al., 2023). With input size $\mu AP = 0.67$3, RUPNet reports 152.60 FPS, Dice $\mu AP = 0.67$4, mIoU $\mu AP = 0.67$5, sensitivity $\mu AP = 0.67$6, precision $\mu AP = 0.67$7, and F2-score $\mu AP = 0.67$8 (Tomar et al., 2023).

TransRUPNet retains the residual upsampling philosophy but replaces the encoder with a pretrained PVT v2-b2. Three feature maps are extracted, reduced to 64 channels through $\mu AP = 0.67$9 convolution, batch normalization, and ReLU, and then supplied both to the decoder and to direct up blocks. This couples global transformer context with lightweight residual refinement. At $= 0.70$0, TransRUPNet reports 47.07 FPS, Kvasir-SEG mDSC $= 0.70$1, mIoU $= 0.70$2, recall $= 0.70$3, and precision $= 0.70$4; on PolypGen it reports mIoU $= 0.70$5 and mDSC $= 0.70$6, with additional gains on CVC-ClinicDB and BKAI-IGH (Jha et al., 2023).

FLDNet also uses a PVT encoder but argues that long-distance dependency alone is insufficient for polyp segmentation. Its architecture combines a pyramid-based transformer encoder, a Local Context Module built from multi-branch standard and dilated convolutions, and a Foreground-Aware Module that derives a coarse map from high-level features and uses it to construct foreground and background probability maps for feature reweighting (Wei et al., 2023). On CVC-ColonDB, the paper reports mDice $= 0.70$7 and mIoU $= 0.70$8, compared with PraNet’s mDice $= 0.70$9 and mIoU $= 0.56$0; on ETIS, it reports mDice $= 0.56$1 against PraNet’s $= 0.56$2 (Wei et al., 2023).

Other CNN-centered designs reach similar goals through parallel feature diversity rather than transformer context. DUCK-Net introduces a residual downsampling mechanism and a multi-branch DUCK block that combines residual blocks, midscope and widescope atrous branches, and a separated large-kernel approximation, then uses additive skips instead of concatenation. On Kvasir-SEG it reports Dice $= 0.56$3, Jaccard $= 0.56$4, Precision $= 0.56$5, Recall $= 0.56$6, and Accuracy $= 0.56$7 for the 34-filter variant, while outperforming several pre-trained baselines without external pretraining (Dumitru et al., 2023). DPE-Net instead uses two parallel encoder branches, one based on dual convolution blocks and one based on single convolution plus residual addition, with depth-wise concatenation before the decoder; it reports mDice $= 0.56$8 and mIoU $= 0.56$9 on Kvasir, and mDice $\mu AP = 0.71$0 and mIoU $\mu AP = 0.71$1 on CVC-ClinicDB, with a total of 3.4M parameters (Manan et al., 2024).

4. Dynamic, foreground, and geometric conditioning

Lesion-aware dynamic kernels constitute a distinct line of encoder specialization. In LDNet, the deepest Res2Net encoder feature $\mu AP = 0.71$2 is globally pooled and projected to an image-conditioned initial kernel,

$\mu AP = 0.71$3

which is then iteratively updated at each decoder stage using lesion features extracted from intermediate predictions (Zhang et al., 2023). The update is explicitly gated: $\mu AP = 0.71$4 with additional Efficient Self-Attention in the encoder and Lesion-aware Cross-Attention in the decoder (Zhang et al., 2023). LDNet reports best results on Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB, and ETIS among the methods compared in that paper, including Dice $\mu AP = 0.71$5 on Kvasir-SEG and $\mu AP = 0.71$6 on ETIS (Zhang et al., 2023).

A later extension, “Enhancing Polyp Segmentation via Encoder Attention and Dynamic Kernel Update,” moves the conditioning upstream. It aggregates pooled descriptors from all encoder stages using a global Encoder Attention module, produces a global context vector $\mu AP = 0.71$7, and initializes the dynamic kernel as $\mu AP = 0.71$8. Decoder-stage kernels are then updated by lesion-aware aggregation and a gated interpolation between new features and the previous kernel, while Unified Channel Adaptation standardizes all decoder features to $\mu AP = 0.71$9 (Chashmi et al., 27 Sep 2025). The reported results are Dice $= 0.74$0, IoU$= 0.74$1 $= 0.74$2, and mIoU $= 0.74$3 on Kvasir-SEG, and Dice $= 0.74$4, IoU$= 0.74$5 $= 0.74$6, and mIoU $= 0.74$7 on CVC-ClinicDB (Chashmi et al., 27 Sep 2025).

Enhanced U-Net makes the encoder polyp-aware by refining both the deepest semantic representation and the skip pathway. The Semantic Feature Enhancement Module applies patch-wise non-local operations to the last encoder feature map and recalibrates channels with an SE block, while the Adaptive Global Context Module replaces direct skip transfer with decoder-guided cross-attention over encoder features (Patel et al., 2021). On EndoScene the full model reports Dice $= 0.74$8 and IoU $= 0.74$9; on Kvasir-SEG it reports Dice $= 0.59$0 and IoU $= 0.59$1; and on cross-dataset testing it improves over the baseline on ColonDB, ETIS, and CVC-300 (Patel et al., 2021).

Geometric priors extend the notion of polyp-awareness beyond appearance. “Learning with Geometric Priors in U-Net Variants for Polyp Segmentation” fine-tunes VGGT on ColonDepth to estimate depth maps tailored to endoscopy, then injects those maps into each skip connection through a Geometric Prior-guided Module. The Cross-Update Block computes a similarity map

$= 0.59$2

and updates skip features as $= 0.59$3, while the Self-Update Block refines depth using the enhanced features (Vazquez et al., 24 Jan 2026). On five datasets, the method reports consistent improvements over U-Net, U-Net v2, and VM-UNetV2, with especially large gains on ETIS and ColonDB (Vazquez et al., 24 Jan 2026). This challenges the purely appearance-based view of a polyp-aware encoder and suggests that 3D structure, depth discontinuities, and colon wall geometry can act as first-class priors.

5. Dual encoders, two-stage relevance maps, and multi-view scene representations

A separate tradition makes the encoder polyp-aware by architectural redundancy. Y-Net uses two parallel VGG19-like encoders, one pretrained and one randomly initialized with SELU activations, and fuses them through a sum-skip-concatenation mechanism. The pretrained branch is updated with a smaller effective learning rate than the scratch-trained branch, explicitly targeting domain shift between natural-image pretraining and colonoscopy appearance (Mohammed et al., 2018). On the MICCAI 2015 ASU–Mayo dataset, the method reports F1 $= 0.59$4 and F2 $= 0.59$5, outperforming the previous methods listed in that work (Mohammed et al., 2018).

Double encoder-decoder networks push the same idea into sequential refinement. The 2021 formulation defines the model as

$= 0.59$6

where the second encoder-decoder receives the original image concatenated with the first network’s segmentation map, which functions as an attention mechanism enabling the second network to focus on interesting areas (Galdran et al., 2021). Its best DPN68×2 configuration exceeds single encoder-decoder variants on Kvasir-SEG and shows particularly large generalization gains on CVC-ColonDB and ETIS, with ETIS Dice $= 0.59$7 versus PraNet’s $= 0.59$8 in the reported comparison (Galdran et al., 2021). A later double encoder-decoder competition system uses ResNeXt101 plus FPN in both stages, SAM-wrapped Adam, 4-fold rotation, and tempered ensembling; on the hidden MedAI test set it reports Dice $= 0.59$9, Precision $77.241$0, and Recall $77.241$1 for polyps at temperature $77.241$2 (Galdran, 2024).

Multi-view encoding broadens the scope again. In EndoFinder-S, per-view embeddings from the polyp-aware image encoder are frozen and then fused by a one-layer Scene Representation Transformer with a learnable [Scene] token, after which a sign-based quantization produces $77.241$3 hash codes for Hamming-distance retrieval (Yang et al., 23 Jul 2025). This indicates that a polyp-aware image encoder need not terminate in a segmentation decoder; it can also serve as a single-view lesion-centric backbone for scene-level aggregation, explainable retrieval, and diagnosis.

6. Empirical patterns, misconceptions, and future directions

Several empirical regularities recur across the literature. First, polyp-awareness is not synonymous with backbone size. RUPNet shows that a three-stage residual encoder can sustain 152.60 FPS at $77.241$4 while maintaining competitive segmentation quality (Tomar et al., 2023), and DUCK-Net shows that a carefully designed CNN trained from scratch can exceed multiple pre-trained baselines on Kvasir-SEG, CVC-ClinicDB, ETIS-LARIBPOLYPDB, and CVC-ColonDB (Dumitru et al., 2023). Second, pathology labels are not a prerequisite. EndoFinder’s image encoder is self-supervised with respect to pathology or identity labels, using only segmentation masks for mask guidance (Yang et al., 23 Jul 2025), and Polyp-OOD requires no polyp masks at training time because it models only healthy images (Ji et al., 2023). Third, the same lesion-centric representation is useful beyond segmentation, as retrieval-based diagnosis and WCE frame ranking both benefit from metric structure in embedding space (Yang et al., 2024, Laiz et al., 2019).

A common misconception is that polyp-awareness is purely a matter of adding attention modules. The literature suggests a broader picture. Some models become lesion-focused through shallow depth and residual preservation of high-resolution boundaries; some through foreground-aware map generation; some through kernel hypernetworks conditioned on lesion features; some through multi-view fusion and hashing; and some through explicit geometric priors (Tomar et al., 2023, Wei et al., 2023, Zhang et al., 2023, Vazquez et al., 24 Jan 2026). This suggests that the essential issue is not attention per se, but whether the encoder’s representational geometry privileges lesion morphology over nuisance variation.

The limitations are equally consistent. Several methods depend on segmentation masks during pretraining or mask-guided reconstruction, so segmentation quality can bound representation quality (Yang et al., 23 Jul 2025). Dataset-specific distributions remain a persistent concern: Polyp-OOD computes latent statistics from a healthy in-distribution set (Ji et al., 2023), RUPNet reports over-segmentation under lighting conditions (Tomar et al., 2023), EndoFinder notes dependence on particular endoscopic devices and imaging protocols (Yang et al., 23 Jul 2025), and CL-Polyp is strong on Kvasir-SEG and CVC-ClinicDB but not uniformly best on cross-domain datasets (Li et al., 9 Jul 2025). Cross-center robustness, rare lesion coverage, and temporal stability in video remain open problems.

The reported future directions are correspondingly diverse. RUPNet proposes adding transformer blocks in the encoder (Tomar et al., 2023). EndoFinder suggests reducing reliance on detailed masks, pursuing pseudo-masks or weakly supervised localization, extending to temporal modeling and multi-modal fusion, and adapting across institutions or devices (Yang et al., 23 Jul 2025). Geometric-prior work indicates that depth, surface structure, and topology can be injected into U-Net variants without redesigning the full backbone (Vazquez et al., 24 Jan 2026). Taken together, these directions imply that the next generation of polyp-aware image encoders is likely to be hybrid: lesion-centric in supervision, multi-scale in architecture, geometry-aware in prior structure, and transferable across retrieval, diagnosis, and segmentation tasks.