EndoMatcher: Dense Matching in Endoscopy

• EndoMatcher is a dense, detector-free endoscopic image matcher that employs a two-branch Siamese Vision Transformer for robust dense correspondence.
• It utilizes multi-scale feature fusion with Dual Interaction Blocks to address challenges like weak textures, occlusions, and non-rigid deformations.
• Generalization is achieved via large-scale, multi-domain pre-training on Endo-Mix6, yielding state-of-the-art metrics and real-time performance in surgical applications.

Searching arXiv for EndoMatcher and closely related endoscopic matching papers. EndoMatcher is a dense, detector-free image matcher for endoscopic images that was introduced for robot-assisted surgery as a method for generalizable correspondence estimation across organs and imaging conditions. In the cited formulation, it is a “generalizable endoscopic image matcher via large-scale, multi-domain data pre-training,” built around a two-branch Siamese Vision Transformer, Dual Interaction Blocks, a pyramid-fusion decoder, and a multi-scale matching module trained with a robust multi-scale response loss (Yang et al., 7 Aug 2025). The term has also been used more broadly to denote endoscopic matching systems that retrieve visually similar lesions for explainable diagnosis, localize or re-localize endoscopic frames in mapped procedures, or establish feature correspondences for geometric reconstruction. In that broader sense, EndoMatcher denotes a family of matching-oriented systems in endoscopy, with dense correspondence learning (Yang et al., 7 Aug 2025), content-based “digital twin” retrieval for colorectal polyps (Yang et al., 2024), calibrated procedure-scale localization benchmarks (Azagra et al., 2022), and self-supervised embedding adaptation for surgical image matching (Rota et al., 11 Dec 2025) representing adjacent instantiations.

1. Definition and problem setting

EndoMatcher, in its primary sense, addresses dense feature matching in endoscopic images for robot-assisted tasks including 3D reconstruction, navigation, and surgical scene understanding (Yang et al., 7 Aug 2025). The task is to find correspondences between two endoscopic images under conditions that are especially adverse for conventional matching: weak or repetitive textures, large viewpoint changes and rotations, specularities and illumination changes, tissue deformation and motion, and occlusions, fluids, smoke, blood, and blur (Yang et al., 7 Aug 2025). The underlying objective is not merely visual similarity in the generic computer-vision sense, but geometrically consistent correspondence estimation under the optical, anatomical, and procedural constraints of endoscopy.

The broader literature frames closely related problems with different operational emphases. One line of work treats endoscopic matching as content-based image retrieval, in which a query polyp image is embedded and matched to a reference database to retrieve “digital twin” polyps with known pathology; the retrieved cases then support explainable optical biopsy and related decisions (Yang et al., 2024). Another line concerns frame-to-frame and frame-to-map matching for localization, loop closure, and Visual SLAM, for which complete calibrated procedures and same-patient repeat examinations are required; the EndoMapper dataset was introduced to support such spatial AI tasks in endoscopy (Azagra et al., 2022). A further line adapts foundation-model features to endoscopic correspondence estimation through self-supervised contrastive embedding learning driven by novel-view synthesis, explicitly targeting epipolar consistency and geometric precision (Rota et al., 11 Dec 2025).

These formulations are complementary rather than contradictory. A plausible implication is that “EndoMatcher” denotes not a single task but a unifying computational primitive—matching—that underlies explainable diagnosis, retrieval, localization, SLAM, geometric reconstruction, and surgical navigation.

2. Core architecture of the dense matcher

The EndoMatcher architecture proposed for robot-assisted surgery is a two-branch Siamese Vision Transformer with a shared ResNet-50 plus Transformer encoder and a multi-scale decoder (Yang et al., 7 Aug 2025). Given a source image $I_s$ and a target image $I_t$, each image passes through the shared feature extractor: a ResNet-50 backbone produces a 1/16-resolution feature map, this is tokenized and fed into a 12-block ViT, and a pyramid-fusion decoder reassembles tokens and early CNN features into multi-scale feature maps at 1/8, 1/4, 1/2, and full resolution (Yang et al., 7 Aug 2025). The method is detector-free and dense, and it reformulates descriptor learning as a keypoint localization problem: for each source keypoint, the network predicts a heatmap over the target image, and the peak yields the match (Yang et al., 7 Aug 2025).

A distinctive architectural element is the Dual Interaction Block. In the cited design, the first six Transformer blocks are standard self-attention blocks, whereas the last six are replaced by Dual Interaction Blocks that combine intra-image self-attention with gated cross-attention between the source and target branches (Yang et al., 7 Aug 2025). For source tokens $X_s^i$ and target tokens $X_t^i$, the source branch first applies self-attention and an MLP,

$X^{0,i}_s = \mathrm{MHA}(X^i_s) + X^i_s,$

$$X^{1,i}_s = \mathrm{MLP}(\mathrm{LN}(X^{0,i}_s)) + X^{0,i}_s,$$

then performs gated cross-attention,

$$X^{2,i}_s = X^{1,i}_s + G \odot \mathrm{MHA}\big(X^{1,i}_s, X^{1,i}_t, X^{1,i}_t\big),$$

with

$$G = \sigma\left(W_2 \cdot \mathrm{ReLU}(W_1 \cdot X^{1,i}_s)\right),$$

followed by a final MLP,

$$X^{i+1}_s = \mathrm{MLP}(\mathrm{LN}(X^{2,i}_s)) + X^{2,i}_s.$$

The same structure is applied symmetrically to the target branch (Yang et al., 7 Aug 2025). This mechanism permits early cross-view reasoning while using a data-dependent gate to suppress noisy or irrelevant inter-image information. In the cited interpretation, this is particularly relevant for occluded, deformed, or specular regions (Yang et al., 7 Aug 2025).

The decoder follows a DPT-style pyramid-fusion design. It uses early ResNet features and selected Transformer features, including outputs from the third and sixth Dual Interaction Blocks, and progressively fuses them through RefineNet-like modules to produce dense multi-scale representations (Yang et al., 7 Aug 2025). This multi-scale structure is motivated by the need to handle large displacement, weak texture, and non-rigid appearance change simultaneously. The paper reports that removing multi-scale supervision degrades keypoint localization accuracy, which supports the claim that multi-scale structure is not ancillary but central to the method’s performance (Yang et al., 7 Aug 2025).

3. Matching formulation and training objective

EndoMatcher’s dense matching module operates by sampling a source descriptor at a source keypoint and correlating it with the target feature map at multiple scales (Yang et al., 7 Aug 2025). For each scale $1/n$, with source and target features $I_t$0 and $I_t$1, the source descriptor at $I_t$2 is treated as a $I_t$3 convolution kernel and convolved over the target map, producing a similarity heatmap $I_t$4 (Yang et al., 7 Aug 2025). This “Point-of-Interest convolution” turns correspondence estimation into supervised heatmap prediction.

Training uses the Robust Multi-Scale Response loss, which supervises the heatmap to peak at the ground-truth target location $I_t$5 at all scales: $I_t$6 with $I_t$7 and $I_t$8 (Yang et al., 7 Aug 2025). The $I_t$9 weighting biases optimization toward higher-resolution predictions, while the temperature sharpens the response distribution. To mitigate noisy supervision, the training discards the lowest 20% of response errors, referred to as residual clipping (Yang et al., 7 Aug 2025). Ablations reported in the paper indicate that both multi-scale supervision and residual clipping improve Percentage of Correct Keypoints, especially under large viewpoint variation and label noise (Yang et al., 7 Aug 2025).

At inference time, the method predicts dense target heatmaps for arbitrary source points and takes the argmax as the match: $X_s^i$0 A cycle consistency check retains only mutually consistent source-to-target and target-to-source matches within a local neighborhood, and optional geometric verification such as RANSAC can be applied downstream (Yang et al., 7 Aug 2025). This yields a matching front-end intended to be compatible with standard geometric estimators.

A related but distinct self-supervised matching pipeline adapts a frozen DINOv2 backbone with an additional Transformer layer and trains the adaptation head using triplet loss mined from novel-view synthesized correspondences (Rota et al., 11 Dec 2025). In that work, descriptors $X_s^i$1 and $X_s^i$2 are matched by cosine similarity,

$X_s^i$3

with mutual nearest-neighbor filtering and a confidence threshold $X_s^i$4 (Rota et al., 11 Dec 2025). The triplet objective is

$X_s^i$5

with margin $X_s^i$6 and cosine-based distance

$X_s^i$7

This suggests a broader methodological pattern in endoscopic matching: either directly learn a dense heatmap matcher from labeled correspondences (Yang et al., 7 Aug 2025) or adapt semantic foundation-model embeddings into geometrically discriminative descriptors through self-supervised contrastive learning (Rota et al., 11 Dec 2025).

4. Multi-domain pre-training and data regime

A central claim of EndoMatcher is that generalization requires large-scale, multi-domain pre-training rather than single-organ or single-dataset supervision (Yang et al., 7 Aug 2025). To support this, the authors construct Endo-Mix6, described as the first multi-domain dataset for endoscopic matching, consisting of approximately 1.2M real and synthetic image pairs across six domains (Yang et al., 7 Aug 2025). The dataset comprises C3VD, EndoSLAM, SCARED, EndoMapper, a colonoscopic dataset, and Ours-Bronch, spanning synthetic colon and stomach scenes, abdominal laparoscopy, real colonoscopy, lesion videos, and bronchoscopic videos (Yang et al., 7 Aug 2025).

The following table summarizes the six domains exactly as reported.

Dataset	Source type	Pairs
C3VD	Colonoscopy 3D Video Dataset, phantom colon	59,740
EndoSLAM	synthetic colon/stomach etc.	614,720
SCARED	stereo abdominal laparoscopy	251,560
EndoMapper	real colonoscopy	62,450
Colonoscopic dataset	GI lesion classification videos	50,070
Ours-Bronch	new bronchoscopic videos from clinical collaborator	150,380

Synthetic domains use ground-truth depth and camera poses to compute dense correspondences directly, whereas real domains use Structure-from-Motion-based labels and simulated perspective transformations (Yang et al., 7 Aug 2025). For real data, the pipeline reconstructs 3D points through a LoFTR-enhanced SfM procedure, reprojects them into frames, enforces visibility checks, and applies RANSAC with a strict reprojection threshold; to increase density, visibility from neighboring frames is also included (Yang et al., 7 Aug 2025). The alternative labeling route perturbs image corners to simulate realistic perspective transformations and warps sampled source pixels by homography, discarding out-of-bounds or severely distorted matches (Yang et al., 7 Aug 2025).

Because Endo-Mix6 is highly heterogeneous, training is formulated as a multi-objective optimization problem over per-dataset RMSR losses,

$X_s^i$8

and solved approximately with Multiple Gradient Descent Algorithm weighting,

$X_s^i$9

(Yang et al., 7 Aug 2025). The training schedule is progressive: Stage 1 pretrains on synthetic datasets C3VD and EndoSLAM, and Stage 2 fine-tunes on the real datasets SCARED, EndoMapper, Colonoscopic, and Ours-Bronch while freezing the ResNet-50 backbone and optimizing the Transformer and decoder (Yang et al., 7 Aug 2025). This configuration, called Endo-Mix6-PMO, is reported as the strongest among Endo-Syn, Endo-Real, naive Endo-Mix6, Endo-Mix6-MO, and Endo-Mix6-PMO (Yang et al., 7 Aug 2025).

The dataset ecology around EndoMatcher includes EndoMapper, which provides 96 complete clinical procedures, more than 24 hours of calibrated video, same-patient repeated colonoscopies, tool masks, anatomical region labels, COLMAP reconstructions, and simulated sequences with full ground truth (Azagra et al., 2022). EndoMapper is explicitly designed for Visual SLAM, 6-DoF pose estimation, 3D mapping, re-localization, and frame-to-map matching in endoscopy (Azagra et al., 2022). This suggests that EndoMatcher’s use of EndoMapper in Endo-Mix6 is not incidental but tied to an existing benchmark tradition for spatial reasoning in endoscopic data.

5. Empirical performance and zero-shot generalization

EndoMatcher is trained exclusively on Endo-Mix6 and evaluated zero-shot on three datasets not included in training: the Hamlyn Centre Dataset for kidney laparoscopy, a Bladder Tissue Dataset, and the Gastro-Matching Dataset for gastroscopy (Yang et al., 7 Aug 2025). The principal dense-matching metrics are the number of initial matches $X_t^i$0, the number of geometrically verified inlier matches $X_t^i$1, and the Keep Ratio,

$X_t^i$2

The evaluation also uses PCK at 5, 10, and 20 pixels for labeled ablations, and on Gastro-Matching it reports Homography Estimation Accuracy at 3 and 5 pixels and Matching Direction Prediction Accuracy (Yang et al., 7 Aug 2025).

On Hamlyn and Bladder, EndoMatcher is compared with SIFT + FLANN, SuperPoint + SuperGlue, SuperPoint + LightGlue, OmniGlue, LoFTR, and TransMatch (Yang et al., 7 Aug 2025). On Hamlyn, EndoMatcher achieves $X_t^i$3, $X_t^i$4, and $X_t^i$5; on Bladder, it achieves $X_t^i$6, $X_t^i$7, and $X_t^i$8 (Yang et al., 7 Aug 2025). Relative to TransMatch, this corresponds to 140.69% more inliers on Hamlyn and 201.43% more inliers on Bladder, while also maintaining a higher or comparable Keep Ratio (Yang et al., 7 Aug 2025). The speed is reported as 47.38 FPS per 1K matches on an RTX 4090 for 384×384 inputs, faster than LoFTR and orders of magnitude faster than TransMatch (Yang et al., 7 Aug 2025).

On Gastro-Matching, EndoMatcher attains HEA@3px of 88.7, HEA@5px of 95.8, and MDPA of 85.4, improving Matching Direction Prediction Accuracy by 9.40% over TransMatch (Yang et al., 7 Aug 2025). The combination of accuracy on stomach-domain geometric tasks with zero-shot robustness on kidney and bladder scenes is presented as evidence that the model generalizes across unseen organs and luminal environments (Yang et al., 7 Aug 2025).

The self-supervised embedding adaptation approach on SCARED offers a different performance profile. It reports mean symmetric epipolar distance

$X_t^i$9

precision $X^{0,i}_s = \mathrm{MHA}(X^i_s) + X^i_s,$0, normalized Frobenius fundamental-matrix error, and inlier percentage after RANSAC (Rota et al., 11 Dec 2025). In the reported comparison, the adapted matcher achieves epipolar error $X^{0,i}_s = \mathrm{MHA}(X^i_s) + X^i_s,$1 px and precision $X^{0,i}_s = \mathrm{MHA}(X^i_s) + X^i_s,$2, outperforming general matchers in those geometric metrics, though with a more conservative inlier profile (Rota et al., 11 Dec 2025). This does not contradict EndoMatcher’s results, since the tasks, architectures, and evaluation protocols differ; rather, it reinforces the point that endoscopic matching rewards domain adaptation and geometry-aware supervision.

6. Related formulations of matching in endoscopy

A broader encyclopedia treatment of EndoMatcher must include the retrieval-based diagnostic use of matching, because the supplied corpus explicitly equates “EndoMatcher” in one context with systems that find the most similar colorectal polyps in a reference database and infer semantics from their known pathology (Yang et al., 2024). EndoFinder operationalizes this paradigm as a content-based image retrieval framework with a polyp-aware Vision Transformer encoder, self-supervised pre-training on Polyp-18k, optional semantic hashing, and $X^{0,i}_s = \mathrm{MHA}(X^i_s) + X^i_s,$3-nearest-neighbor label inference (Yang et al., 2024).

In EndoFinder, a query image $X^{0,i}_s = \mathrm{MHA}(X^i_s) + X^i_s,$4 is embedded as $X^{0,i}_s = \mathrm{MHA}(X^i_s) + X^i_s,$5, optionally binarized to a semantic code $X^{0,i}_s = \mathrm{MHA}(X^i_s) + X^i_s,$6, and matched against a reference set

$X^{0,i}_s = \mathrm{MHA}(X^i_s) + X^i_s,$7

Similarity in training uses temperature-scaled cosine similarity,

$X^{0,i}_s = \mathrm{MHA}(X^i_s) + X^i_s,$8

and retrieval uses cosine similarity in the continuous space or Hamming distance in the binary space (Yang et al., 2024). Diagnostic inference is by majority voting among the $X^{0,i}_s = \mathrm{MHA}(X^i_s) + X^i_s,$9 nearest neighbors,

$$X^{1,i}_s = \mathrm{MLP}(\mathrm{LN}(X^{0,i}_s)) + X^{0,i}_s,$$0

which turns matching into explainable case-based reasoning (Yang et al., 2024).

The system is validated on polyp re-identification and optical biopsy. On Polyp-Twin, EndoFinder-Hash achieves uAP 0.693, Acc@1 0.693, Recall@90% 0.524, with retrieval time 0.009 s and 108.57 FPS on a 12k+ image database; on Polyp-Path, EndoFinder-Raw reaches ACC 77.24% and F1 80.45%, surpassing the best supervised classifier in accuracy and F1 without downstream fine-tuning (Yang et al., 2024). This suggests that “matching” in endoscopy may denote either dense geometric correspondence or semantic nearest-neighbor retrieval, depending on whether the target application is spatial reconstruction or explainable diagnosis.

A second adjacent formulation appears in metric scale estimation. EndoMetric uses a monocular reconstruction and a calibrated near-light photometric model to estimate the global metric scale $$X^{1,i}_s = \mathrm{MLP}(\mathrm{LN}(X^{0,i}_s)) + X^{0,i}_s,$$1 of endoscopic SLAM outputs (Iranzo et al., 2024). The model assumes

$$X^{1,i}_s = \mathrm{MLP}(\mathrm{LN}(X^{0,i}_s)) + X^{0,i}_s,$$2

and estimates $$X^{1,i}_s = \mathrm{MLP}(\mathrm{LN}(X^{0,i}_s)) + X^{0,i}_s,$$3 jointly with scaled albedos and per-frame gains by minimizing robust photometric error under near-light inverse-square attenuation and Lambertian reflection (Iranzo et al., 2024). A plausible implication is that once an EndoMatcher-style geometric pipeline supplies up-to-scale camera poses and correspondences, EndoMetric can convert the result into a metric device, which is important for measurement and registration tasks downstream of matching.

7. Benchmarking context, limitations, and future directions

The benchmark context for endoscopic matching is unusually challenging. EndoMapper shows that even with accurate fisheye calibration under the Kannala–Brandt model and full-field-of-view features, ORB-SLAM3 localized only about 25% of frames robustly on one evaluated sequence and fragmented the map into 133 sub-maps (Azagra et al., 2022). This quantifies the difficulty of endoscopic localization under non-rigid motion, highlights, texture scarcity, and frequent tracking loss (Azagra et al., 2022). EndoMatcher’s dense correspondence gains should be understood against that backdrop: the task is not merely harder than natural-scene matching in degree, but structurally different in the geometry–appearance relationship.

Several limitations are explicitly stated for EndoMatcher. Although real-time at about 47 FPS per 1K matches on a high-end GPU, further optimization is needed for more latency-critical applications, higher resolutions, or larger numbers of matches (Yang et al., 7 Aug 2025). The Vision-Transformer-based architecture implies non-trivial memory and compute demands, so deployment on constrained hardware may require pruning, quantization, or distillation (Yang et al., 7 Aug 2025). The generalization claim is anchored in the distribution of Endo-Mix6; unseen modalities such as narrow-band imaging, ultrasound endoscopy, or very different camera optics may require further adaptation or extension of the training set (Yang et al., 7 Aug 2025). Heavy bleeding, smoke, extreme blur, or complete occlusion are not deeply analyzed and remain likely failure modes (Yang et al., 7 Aug 2025).

The self-supervised embedding-adaptation line introduces a different set of trade-offs. Its dependence on DPT-based depth quality means that errors in monocular depth prediction can corrupt the pseudo-correspondence signal used for triplet mining (Rota et al., 11 Dec 2025). It also prioritizes precision and low epipolar error over dense match count, which may be advantageous for pose estimation but limiting for applications demanding very dense reconstructions (Rota et al., 11 Dec 2025). This suggests that future systems may combine the dense output profile of EndoMatcher (Yang et al., 7 Aug 2025) with the geometric conservatism of contrastive embedding adaptation (Rota et al., 11 Dec 2025).

The EndoMatcher paper itself outlines future work centered on real-time improvement, expansion of Endo-Mix6 to more organs and imaging modalities, and tighter integration with endoscopic SLAM and depth estimation (Yang et al., 7 Aug 2025). EndoMapper points toward long-term map reuse, same-patient re-localization, and calibrated VSLAM evaluation on full procedures (Azagra et al., 2022). EndoFinder points toward explainable, retrieval-based support for optical biopsy, re-identification, and surveillance planning (Yang et al., 2024). Taken together, these threads suggest that future “EndoMatcher” systems will likely become composite platforms in which dense correspondence, retrieval, metric scale recovery, and temporal map reasoning are interoperable modules rather than isolated methods.