---
title: 'EndoMatcher: Dense Matching in Endoscopy'
url: https://www.emergentmind.com/topics/endomatcher
type: topic
---

# EndoMatcher: Dense Matching in Endoscopy

Searching arXiv for EndoMatcher and closely related endoscopic matching papers.
EndoMatcher is a dense, detector-free image matcher for endoscopic images that was introduced for robot-assisted surgery as a method for generalizable correspondence estimation across organs and imaging conditions. In the cited formulation, it is a “generalizable endoscopic image matcher via large-scale, multi-domain data pre-training,” built around a two-branch Siamese Vision Transformer, Dual Interaction Blocks, a pyramid-fusion decoder, and a multi-scale matching module trained with a robust multi-scale response loss [2508.05205]. The term has also been used more broadly to denote endoscopic matching systems that retrieve visually similar lesions for explainable diagnosis, localize or re-localize endoscopic frames in mapped procedures, or establish feature correspondences for geometric reconstruction. In that broader sense, EndoMatcher denotes a family of matching-oriented systems in endoscopy, with dense correspondence learning [2508.05205], content-based “digital twin” retrieval for colorectal polyps [2407.11401], calibrated procedure-scale localization benchmarks [2204.14240], and self-supervised embedding adaptation for surgical image matching [2512.10379] representing adjacent instantiations.

## 1. Definition and problem setting

EndoMatcher, in its primary sense, addresses dense feature matching in endoscopic images for robot-assisted tasks including 3D reconstruction, navigation, and surgical scene understanding [2508.05205]. The task is to find correspondences between two endoscopic images under conditions that are especially adverse for conventional matching: weak or repetitive textures, large viewpoint changes and rotations, specularities and illumination changes, tissue deformation and motion, and occlusions, fluids, smoke, blood, and blur [2508.05205]. The underlying objective is not merely visual similarity in the generic computer-vision sense, but geometrically consistent correspondence estimation under the optical, anatomical, and procedural constraints of endoscopy.

The broader literature frames closely related problems with different operational emphases. One line of work treats endoscopic matching as content-based image retrieval, in which a query polyp image is embedded and matched to a reference database to retrieve “digital twin” polyps with known pathology; the retrieved cases then support explainable optical biopsy and related decisions [2407.11401]. Another line concerns frame-to-frame and frame-to-map matching for localization, loop closure, and Visual SLAM, for which complete calibrated procedures and same-patient repeat examinations are required; the EndoMapper dataset was introduced to support such spatial AI tasks in endoscopy [2204.14240]. A further line adapts foundation-model features to endoscopic correspondence estimation through self-supervised contrastive embedding learning driven by novel-view synthesis, explicitly targeting epipolar consistency and geometric precision [2512.10379].

These formulations are complementary rather than contradictory. A plausible implication is that “EndoMatcher” denotes not a single task but a unifying computational primitive—matching—that underlies explainable diagnosis, retrieval, localization, SLAM, geometric reconstruction, and surgical navigation.

## 2. Core architecture of the dense matcher

The EndoMatcher architecture proposed for robot-assisted surgery is a two-branch Siamese Vision Transformer with a shared ResNet-50 plus Transformer encoder and a multi-scale decoder [2508.05205]. Given a source image \(I_s\) and a target image \(I_t\), each image passes through the shared feature extractor: a ResNet-50 backbone produces a 1/16-resolution feature map, this is tokenized and fed into a 12-block ViT, and a pyramid-fusion decoder reassembles tokens and early CNN features into multi-scale feature maps at 1/8, 1/4, 1/2, and full resolution [2508.05205]. The method is detector-free and dense, and it reformulates descriptor learning as a keypoint localization problem: for each source keypoint, the network predicts a heatmap over the target image, and the peak yields the match [2508.05205].

A distinctive architectural element is the Dual Interaction Block. In the cited design, the first six Transformer blocks are standard self-attention blocks, whereas the last six are replaced by Dual Interaction Blocks that combine intra-image self-attention with gated cross-attention between the source and target branches [2508.05205]. For source tokens \(X_s^i\) and target tokens \(X_t^i\), the source branch first applies self-attention and an MLP,
\[
X^{0,i}_s = \mathrm{MHA}(X^i_s) + X^i_s,
\]
\[
X^{1,i}_s = \mathrm{MLP}(\mathrm{LN}(X^{0,i}_s)) + X^{0,i}_s,
\]
then performs gated cross-attention,
\[
X^{2,i}_s = X^{1,i}_s + G \odot \mathrm{MHA}\big(X^{1,i}_s, X^{1,i}_t, X^{1,i}_t\big),
\]
with
\[
G = \sigma\left(W_2 \cdot \mathrm{ReLU}(W_1 \cdot X^{1,i}_s)\right),
\]
followed by a final MLP,
\[
X^{i+1}_s = \mathrm{MLP}(\mathrm{LN}(X^{2,i}_s)) + X^{2,i}_s.
\]
The same structure is applied symmetrically to the target branch [2508.05205]. This mechanism permits early cross-view reasoning while using a data-dependent gate to suppress noisy or irrelevant inter-image information. In the cited interpretation, this is particularly relevant for occluded, deformed, or specular regions [2508.05205].

The decoder follows a DPT-style pyramid-fusion design. It uses early ResNet features and selected Transformer features, including outputs from the third and sixth Dual Interaction Blocks, and progressively fuses them through RefineNet-like modules to produce dense multi-scale representations [2508.05205]. This multi-scale structure is motivated by the need to handle large displacement, weak texture, and non-rigid appearance change simultaneously. The paper reports that removing multi-scale supervision degrades keypoint localization accuracy, which supports the claim that multi-scale structure is not ancillary but central to the method’s performance [2508.05205].

## 3. Matching formulation and training objective

EndoMatcher’s dense matching module operates by sampling a source descriptor at a source keypoint and correlating it with the target feature map at multiple scales [2508.05205]. For each scale \(1/n\), with source and target features \(F_{s1/n}\) and \(F_{t1/n}\), the source descriptor at \(P_s\) is treated as a \(1 \times 1\) convolution kernel and convolved over the target map, producing a similarity heatmap \(M_{t1/n}\) [2508.05205]. This “Point-of-Interest convolution” turns correspondence estimation into supervised heatmap prediction.

Training uses the Robust Multi-Scale Response loss, which supervises the heatmap to peak at the ground-truth target location \(P_t\) at all scales:
\[
\mathcal{L}_{\mathrm{rmsr}} = \sum_{n} \frac{1}{n} \left(
-\log
\frac{e^{\sigma M_{t1/n}(P_t)}}{\sum_{P} e^{\sigma M_{t1/n}(P)}}
\right),
\]
with \(n \in \{1, 2, 4, 8\}\) and \(\sigma = 20\) [2508.05205]. The \(1/n\) weighting biases optimization toward higher-resolution predictions, while the temperature sharpens the response distribution. To mitigate noisy supervision, the training discards the lowest 20% of response errors, referred to as residual clipping [2508.05205]. Ablations reported in the paper indicate that both multi-scale supervision and residual clipping improve Percentage of Correct Keypoints, especially under large viewpoint variation and label noise [2508.05205].

At inference time, the method predicts dense target heatmaps for arbitrary source points and takes the argmax as the match:
\[
\hat{P}_t = \arg\max_P M_t(P).
\]
A cycle consistency check retains only mutually consistent source-to-target and target-to-source matches within a local neighborhood, and optional geometric verification such as RANSAC can be applied downstream [2508.05205]. This yields a matching front-end intended to be compatible with standard geometric estimators.

A related but distinct self-supervised matching pipeline adapts a frozen DINOv2 backbone with an additional Transformer layer and trains the adaptation head using triplet loss mined from novel-view synthesized correspondences [2512.10379]. In that work, descriptors \(\mathbf{M}_s\) and \(\mathbf{M}_t\) are matched by cosine similarity,
\[
\mathbf{S}_{ij} = \frac{\mathbf{M}_s^i \cdot \mathbf{M}_t^j}{\|\mathbf{M}_s^i\| \, \|\mathbf{M}_t^j\|},
\]
with mutual nearest-neighbor filtering and a confidence threshold \(s_{ij} > 0.95\) [2512.10379]. The triplet objective is
\[
\mathcal{L}_{\text{triplet}} = \max\left(0,\; d(\mathbf{A}, \mathbf{P}) - d(\mathbf{A}, \mathbf{N}) + m\right),
\]
with margin \(m=1\) and cosine-based distance
\[
d(\mathbf{u}, \mathbf{v}) = \sqrt{2 - 2 \cdot \frac{\mathbf{u}^\top \mathbf{v}}{\|\mathbf{u}\| \cdot \|\mathbf{v}\|}}.
\]
This suggests a broader methodological pattern in endoscopic matching: either directly learn a dense heatmap matcher from labeled correspondences [2508.05205] or adapt semantic foundation-model embeddings into geometrically discriminative descriptors through self-supervised contrastive learning [2512.10379].

## 4. Multi-domain pre-training and data regime

A central claim of EndoMatcher is that generalization requires large-scale, multi-domain pre-training rather than single-organ or single-dataset supervision [2508.05205]. To support this, the authors construct Endo-Mix6, described as the first multi-domain dataset for endoscopic matching, consisting of approximately 1.2M real and synthetic image pairs across six domains [2508.05205]. The dataset comprises C3VD, EndoSLAM, SCARED, EndoMapper, a colonoscopic dataset, and Ours-Bronch, spanning synthetic colon and stomach scenes, abdominal laparoscopy, real colonoscopy, lesion videos, and bronchoscopic videos [2508.05205].

The following table summarizes the six domains exactly as reported.

| Dataset | Source type | Pairs |
|---|---|---:|
| C3VD | Colonoscopy 3D Video Dataset, phantom colon | 59,740 |
| EndoSLAM | synthetic colon/stomach etc. | 614,720 |
| SCARED | stereo abdominal laparoscopy | 251,560 |
| EndoMapper | real colonoscopy | 62,450 |
| Colonoscopic dataset | GI lesion classification videos | 50,070 |
| Ours-Bronch | new bronchoscopic videos from clinical collaborator | 150,380 |

Synthetic domains use ground-truth depth and camera poses to compute dense correspondences directly, whereas real domains use Structure-from-Motion-based labels and simulated perspective transformations [2508.05205]. For real data, the pipeline reconstructs 3D points through a LoFTR-enhanced SfM procedure, reprojects them into frames, enforces visibility checks, and applies RANSAC with a strict reprojection threshold; to increase density, visibility from neighboring frames is also included [2508.05205]. The alternative labeling route perturbs image corners to simulate realistic perspective transformations and warps sampled source pixels by homography, discarding out-of-bounds or severely distorted matches [2508.05205].

Because Endo-Mix6 is highly heterogeneous, training is formulated as a multi-objective optimization problem over per-dataset RMSR losses,
\[
\min_{\boldsymbol{\theta}}
\Big(
\mathcal{L}_{\mathrm{rmsr}}^{1}(\boldsymbol{\theta}),
\ldots,
\mathcal{L}_{\mathrm{rmsr}}^{D}(\boldsymbol{\theta})
\Big)^\top,
\]
and solved approximately with Multiple Gradient Descent Algorithm weighting,
\[
\mathcal{L} = \sum_{d=1}^D w^d \, \mathcal{L}_{\mathrm{rmsr}}^{d}(\boldsymbol{\theta})
\]
[2508.05205]. The training schedule is progressive: Stage 1 pretrains on synthetic datasets C3VD and EndoSLAM, and Stage 2 fine-tunes on the real datasets SCARED, EndoMapper, Colonoscopic, and Ours-Bronch while freezing the ResNet-50 backbone and optimizing the Transformer and decoder [2508.05205]. This configuration, called Endo-Mix6-PMO, is reported as the strongest among Endo-Syn, Endo-Real, naive Endo-Mix6, Endo-Mix6-MO, and Endo-Mix6-PMO [2508.05205].

The dataset ecology around EndoMatcher includes EndoMapper, which provides 96 complete clinical procedures, more than 24 hours of calibrated video, same-patient repeated colonoscopies, tool masks, anatomical region labels, COLMAP reconstructions, and simulated sequences with full ground truth [2204.14240]. EndoMapper is explicitly designed for Visual SLAM, 6-DoF pose estimation, 3D mapping, re-localization, and frame-to-map matching in endoscopy [2204.14240]. This suggests that EndoMatcher’s use of EndoMapper in Endo-Mix6 is not incidental but tied to an existing benchmark tradition for spatial reasoning in endoscopic data.

## 5. Empirical performance and zero-shot generalization

EndoMatcher is trained exclusively on Endo-Mix6 and evaluated zero-shot on three datasets not included in training: the Hamlyn Centre Dataset for kidney laparoscopy, a Bladder Tissue Dataset, and the Gastro-Matching Dataset for gastroscopy [2508.05205]. The principal dense-matching metrics are the number of initial matches \(N_{\text{pt}}\), the number of geometrically verified inlier matches \(N_{\text{inlier}}\), and the Keep Ratio,
\[
\text{KR} = \frac{N_{\text{inlier}}}{N_{\text{pt}}}.
\]
The evaluation also uses PCK at 5, 10, and 20 pixels for labeled ablations, and on Gastro-Matching it reports Homography Estimation Accuracy at 3 and 5 pixels and Matching Direction Prediction Accuracy [2508.05205].

On Hamlyn and Bladder, EndoMatcher is compared with SIFT + FLANN, SuperPoint + SuperGlue, SuperPoint + LightGlue, OmniGlue, LoFTR, and TransMatch [2508.05205]. On Hamlyn, EndoMatcher achieves \(N_{\text{pt}} = 8190.21\), \(N_{\text{inlier}} = 6375.70\), and \(\text{KR} = 73.40\); on Bladder, it achieves \(N_{\text{pt}} = 8346.82\), \(N_{\text{inlier}} = 6116.20\), and \(\text{KR} = 68.75\) [2508.05205]. Relative to TransMatch, this corresponds to 140.69% more inliers on Hamlyn and 201.43% more inliers on Bladder, while also maintaining a higher or comparable Keep Ratio [2508.05205]. The speed is reported as 47.38 FPS per 1K matches on an RTX 4090 for 384×384 inputs, faster than LoFTR and orders of magnitude faster than TransMatch [2508.05205].

On Gastro-Matching, EndoMatcher attains HEA@3px of 88.7, HEA@5px of 95.8, and MDPA of 85.4, improving Matching Direction Prediction Accuracy by 9.40% over TransMatch [2508.05205]. The combination of accuracy on stomach-domain geometric tasks with zero-shot robustness on kidney and bladder scenes is presented as evidence that the model generalizes across unseen organs and luminal environments [2508.05205].

The self-supervised embedding adaptation approach on SCARED offers a different performance profile. It reports mean symmetric epipolar distance
\[
\begin{split}
E_e = \frac{1}{|\mathcal{C}|} \sum_{(p_s^i, p_t^i) \in \mathcal{C}} \Bigg(
&\frac{|(p_t^i)^\top F_{gt} p_s^i|} {\sqrt{(F_{gt}p_s^i)_1^2 + (F_{gt}p_s^i)_2^2}} \\
+&\frac{|(p_s^i)^\top F_{gt}^\top p_t^i|} {\sqrt{(F_{gt}^\top p_t^i)_1^2 + (F_{gt}^\top p_t^i)_2^2}}
\Bigg),
\end{split}
\]
precision \(Pr = \frac{TP}{TP+FP}\), normalized Frobenius fundamental-matrix error, and inlier percentage after RANSAC [2512.10379]. In the reported comparison, the adapted matcher achieves epipolar error \(8.63 \pm 4.14\) px and precision \(14.22 \pm 4.61\%\), outperforming general matchers in those geometric metrics, though with a more conservative inlier profile [2512.10379]. This does not contradict EndoMatcher’s results, since the tasks, architectures, and evaluation protocols differ; rather, it reinforces the point that endoscopic matching rewards domain adaptation and geometry-aware supervision.

## 6. Related formulations of matching in endoscopy

A broader encyclopedia treatment of EndoMatcher must include the retrieval-based diagnostic use of matching, because the supplied corpus explicitly equates “EndoMatcher” in one context with systems that find the most similar colorectal polyps in a reference database and infer semantics from their known pathology [2407.11401]. EndoFinder operationalizes this paradigm as a content-based image retrieval framework with a polyp-aware Vision Transformer encoder, self-supervised pre-training on Polyp-18k, optional semantic hashing, and \(K\)-nearest-neighbor label inference [2407.11401].

In EndoFinder, a query image \(I\) is embedded as \(z = E_\phi(I)\in\mathbb{R}^d\), optionally binarized to a semantic code \(\bar{z}\in\{-1,1\}^d\), and matched against a reference set
\[
S = \{(I_i, y_i)\}_{i=1}^N.
\]
Similarity in training uses temperature-scaled cosine similarity,
\[
s_{i,j} = \frac{z_i^\top z_j}{\|z_i\|\|z_j\|} / \tau,
\]
and retrieval uses cosine similarity in the continuous space or Hamming distance in the binary space [2407.11401]. Diagnostic inference is by majority voting among the \(K\) nearest neighbors,
\[
\hat{y}_i = \underset{c \in \{1,\dots,C\}}{\mathrm{argmax}}
\sum_{k \in \mathcal{N}(I_i)} \mathbf{1}_{\{y_k = c\}},
\]
which turns matching into explainable case-based reasoning [2407.11401].

The system is validated on polyp re-identification and optical biopsy. On Polyp-Twin, EndoFinder-Hash achieves uAP 0.693, Acc@1 0.693, Recall@90% 0.524, with retrieval time 0.009 s and 108.57 FPS on a 12k+ image database; on Polyp-Path, EndoFinder-Raw reaches ACC 77.24% and F1 80.45%, surpassing the best supervised classifier in accuracy and F1 without downstream fine-tuning [2407.11401]. This suggests that “matching” in endoscopy may denote either dense geometric correspondence or semantic nearest-neighbor retrieval, depending on whether the target application is spatial reconstruction or explainable diagnosis.

A second adjacent formulation appears in metric scale estimation. EndoMetric uses a monocular reconstruction and a calibrated near-light photometric model to estimate the global metric scale \(\lambda\) of endoscopic SLAM outputs [2410.15065]. The model assumes
\[
\mathbf{X}_{\text{metric}} = \lambda \, \mathbf{X}_{\text{SLAM}},
\]
and estimates \(\lambda\) jointly with scaled albedos and per-frame gains by minimizing robust photometric error under near-light inverse-square attenuation and Lambertian reflection [2410.15065]. A plausible implication is that once an EndoMatcher-style geometric pipeline supplies up-to-scale camera poses and correspondences, EndoMetric can convert the result into a metric device, which is important for measurement and registration tasks downstream of matching.

## 7. Benchmarking context, limitations, and future directions

The benchmark context for endoscopic matching is unusually challenging. EndoMapper shows that even with accurate fisheye calibration under the Kannala–Brandt model and full-field-of-view features, ORB-SLAM3 localized only about 25% of frames robustly on one evaluated sequence and fragmented the map into 133 sub-maps [2204.14240]. This quantifies the difficulty of endoscopic localization under non-rigid motion, highlights, texture scarcity, and frequent tracking loss [2204.14240]. EndoMatcher’s dense correspondence gains should be understood against that backdrop: the task is not merely harder than natural-scene matching in degree, but structurally different in the geometry–appearance relationship.

Several limitations are explicitly stated for EndoMatcher. Although real-time at about 47 FPS per 1K matches on a high-end GPU, further optimization is needed for more latency-critical applications, higher resolutions, or larger numbers of matches [2508.05205]. The Vision-Transformer-based architecture implies non-trivial memory and compute demands, so deployment on constrained hardware may require pruning, quantization, or distillation [2508.05205]. The generalization claim is anchored in the distribution of Endo-Mix6; unseen modalities such as narrow-band imaging, ultrasound endoscopy, or very different camera optics may require further adaptation or extension of the training set [2508.05205]. Heavy bleeding, smoke, extreme blur, or complete occlusion are not deeply analyzed and remain likely failure modes [2508.05205].

The self-supervised embedding-adaptation line introduces a different set of trade-offs. Its dependence on DPT-based depth quality means that errors in monocular depth prediction can corrupt the pseudo-correspondence signal used for triplet mining [2512.10379]. It also prioritizes precision and low epipolar error over dense match count, which may be advantageous for pose estimation but limiting for applications demanding very dense reconstructions [2512.10379]. This suggests that future systems may combine the dense output profile of EndoMatcher [2508.05205] with the geometric conservatism of contrastive embedding adaptation [2512.10379].

The EndoMatcher paper itself outlines future work centered on real-time improvement, expansion of Endo-Mix6 to more organs and imaging modalities, and tighter integration with endoscopic SLAM and depth estimation [2508.05205]. EndoMapper points toward long-term map reuse, same-patient re-localization, and calibrated VSLAM evaluation on full procedures [2204.14240]. EndoFinder points toward explainable, retrieval-based support for optical biopsy, re-identification, and surveillance planning [2407.11401]. Taken together, these threads suggest that future “EndoMatcher” systems will likely become composite platforms in which dense correspondence, retrieval, metric scale recovery, and temporal map reasoning are interoperable modules rather than isolated methods.

Source: https://www.emergentmind.com/topics/endomatcher