Endo-Mix6: Multi-Domain Endoscopic Matching

• Endo-Mix6 is a multi-domain dataset of approximately 1.2 million real and synthetic endoscopic image pairs designed for dense correspondence matching in clinically challenging conditions.
• It integrates six distinct domains—from synthetic phantoms to real clinical imagery—using geometry-based and SfM labeling along with simulated perspective transforms to provide dense, accurate annotations.
• Within the EndoMatcher training framework, Endo-Mix6 underpins a progressive multi-objective optimization strategy that enables zero-shot generalization and significant performance improvements on external benchmarks.

Endo-Mix6 is a large-scale, multi-domain dataset for endoscopic image matching, constructed to pre-train the EndoMatcher model for dense correspondence learning under clinically difficult visual conditions (Yang et al., 7 Aug 2025). It consists of approximately 1.2 million real and synthetic image pairs drawn from six domains spanning multiple organs and acquisition modalities, with dense or semi-dense correspondence labels generated from simulator geometry, Structure-from-Motion (SfM), and simulated perspective transformations. Within the EndoMatcher framework, Endo-Mix6 functions as the pre-training substrate enabling zero-shot generalization to unseen organs and imaging conditions, including external kidney, bladder, and gastroscopy datasets (Yang et al., 7 Aug 2025).

1. Definition and motivation

Endo-Mix6 was introduced to address two recurrent limitations in endoscopic feature matching: difficult visual conditions and scarcity of annotated correspondence data (Yang et al., 7 Aug 2025). The difficult visual regime includes weak, smooth, or repetitive mucosal textures; large viewpoint variations, including rotations greater than $70^\circ$ and strong forward camera motion in narrow lumens; and strong illumination changes, specularities, and non-rigid motion. In parallel, dense ground-truth correspondences are difficult to acquire in endoscopy because RGB-D capture is impractical and manual annotation is error-prone.

The dataset was therefore designed to provide both scale and diversity. Its approximately 1.2 million image pairs cover six domains, multiple organs, and both synthetic and clinical modalities. Correspondence labels are generated automatically using SfM and synthetic perspective transformations rather than manual labeling. This design supports the broader objective of training a matcher that can generalize in a zero-shot fashion to unseen organs and external datasets such as Hamlyn kidney laparoscopy, bladder imagery, and Gastro-Matching gastroscopy (Yang et al., 7 Aug 2025).

A central claim attached to Endo-Mix6 is that it is the first multi-domain dataset specifically designed for endoscopic matching. Existing resources cited alongside it, including Hamlyn, EndoSLAM, SCARED, EndoMapper, Bladder Tissue, and Gastro-Matching, are characterized in the source paper as limited in domain coverage, label density, or task focus, with several designed primarily for SLAM, depth, or other downstream problems rather than dense pixel-wise matching across organs and clinical settings (Yang et al., 7 Aug 2025).

2. Dataset composition and domain coverage

Endo-Mix6 combines six pre-training datasets, comprising two synthetic domains and four real clinical domains (Yang et al., 7 Aug 2025).

Domain	Type	Organs / Region
C3VD	Synthetic / phantom	Colon phantom
EndoSLAM	Synthetic	Colon, stomach, etc.
SCARED	Real (stereo, clinical)	Abdomen
EndoMapper	Real (monocular)	Colon
Colonoscopic	Real	Colon
Ours-Bronch	Real (monocular)	Airway

The paper reports the following sequence, frame, and pair counts, with pairs computed using a temporal window $r=20$ (Yang et al., 7 Aug 2025).

Domain	Sequences / Frames	Image pairs ($r=20$)
C3VD	22 / 3,218	59,740
EndoSLAM	340 / 34,306	614,720
SCARED	108 / 13,712	251,560
EndoMapper	27 / 3,406	62,450
Colonoscopic	21 / 2,724	50,070
Ours-Bronch	68 / 8,233	150,380

These counts sum to approximately 1,188,920 image pairs, which the paper refers to as approximately 1.2 million. The synthetic domains provide dense, noise-free depth and camera poses per frame and serve as well-controlled sources for stable geometric supervision. The real domains contribute clinical variability, including tissue motion, instruments, smoke, blood, specular highlights, varied lighting, narrow-lumen navigation, large forward motion, and deformation.

Three datasets are explicitly excluded from Endo-Mix6 and reserved for zero-shot evaluation: Hamlyn Centre, Bladder Tissue Dataset, and Gastro-Matching. This separation is important because it makes the reported cross-organ generalization claims genuinely out-of-domain relative to the Endo-Mix6 training pool (Yang et al., 7 Aug 2025).

3. Correspondence generation and label construction

Endo-Mix6 uses two principal label-generation pipelines: direct geometry-based labeling for synthetic data and SfM-based labeling for real clinical videos, supplemented by simulated perspective transformations to densify supervision (Yang et al., 7 Aug 2025).

For EndoSLAM and C3VD, the simulator or phantom setup provides camera intrinsics $\mathbf{K}$, pose $(\mathbf{R}, \mathbf{t})$, and depth $D(\mathbf{x})$. Dense correspondences between source frame $I_s$ and target frame $I_t$ are generated by back-projecting a source pixel $\mathbf{x}_s=(u_s,v_s)$ with depth $z_s$ into 3D,

$r=20$0

transforming it into the target camera,

$r=20$1

and projecting it to the target image,

$r=20$2

A pair $r=20$3 is retained as a ground-truth correspondence when the target location lies inside the image and the depth is valid. The result is high-fidelity dense supervision.

For SCARED, EndoMapper, Colonoscopic, and Ours-Bronch, the paper uses a custom SfM-based pipeline. LoFTR provides initial detector-free matches between frames in short clips. Camera parameters and 3D points are then estimated by minimizing the reprojection error in a COLMAP / Schönberger-Frahm-style bundle adjustment:

$r=20$4

After reconstruction, 3D points visible in two frames are projected back to both images, subject to visibility checks inside the image bounds and with positive, reasonable depth. To increase label density, the pipeline also considers visibility from neighboring frames within the $r=20$5 temporal window. A strict RANSAC-based filtering stage is then applied after back-projection to enforce geometric consistency and remove matches exceeding a reprojection error threshold.

Because SfM labels in endoscopy remain sparse and may be noisy, the dataset further incorporates synthetic dense correspondences via random perspective transformations. For an image $r=20$6, a perturbed quadrilateral is generated, a homography $r=20$7 is computed, and the image is warped to a synthesized target view $r=20$8:

$r=20$9

Sampled pixels in valid regions of $r=20$0 are mapped through $r=20$1 to obtain dense synthetic correspondences, while points falling outside the image or exhibiting excessive local distortion are discarded. This mechanism augments real data with noise-free labels and simulates realistic viewpoint shifts while preserving exact mapping.

4. Role in training and optimization

Endo-Mix6 is tightly coupled to the EndoMatcher training protocol, especially its progressive multi-objective (PMO) schedule and robust multi-scale response loss (Yang et al., 7 Aug 2025). The dataset is not treated as a homogeneous corpus: its synthetic and real subsets are used in different phases because they differ substantially in label fidelity, noise profile, and domain statistics.

The progressive schedule has two stages. Stage 1 pretrains on synthetic data only, specifically EndoSLAM and C3VD, referred to as Endo-Syn. This stage uses AdamW with learning rate $r=20$2, batch size 16, and 10 epochs, training the full network on high-fidelity synthetic labels. Stage 2 fine-tunes on real data only, specifically SCARED, EndoMapper, Colonoscopic, and Ours-Bronch, referred to as Endo-Real. In this stage, the ResNet-50 backbone is frozen, training runs for 30 epochs with batch size 16, and optimization uses a cosine learning rate schedule with a 5-epoch warm-up.

Each dataset in a given phase is treated as a separate task. Rather than minimizing a fixed weighted sum, the method formulates training as a multi-objective optimization problem over per-dataset RMSR losses:

$r=20$3

The paper uses the Multiple Gradient Descent Algorithm (MGDA) to compute non-negative weights $r=20$4 and form a Pareto-descent direction,

$r=20$5

with effective total loss

$r=20$6

This adaptive weighting is explicitly motivated by Endo-Mix6’s dataset-size imbalance and domain shifts, preventing large domains such as EndoSLAM from dominating optimization.

The RMSR loss exploits multi-scale correspondence supervision. Given a source point $r=20$7, target point $r=20$8, and feature maps $r=20$9 for $\mathbf{K}$0, the matching module computes a response map by using the source descriptor as a $\mathbf{K}$1 convolution kernel:

$\mathbf{K}$2

The loss is

$\mathbf{K}$3

with temperature $\mathbf{K}$4. To handle noisy SfM labels and ambiguous samples, the training procedure discards the lowest 20% of response errors, a residual clipping strategy intended to focus optimization on harder samples.

Operationally, all Endo-Mix6 images are resized to $\mathbf{K}$5, and 10 correspondences are randomly sampled from each training pair. Augmentations include random brightness adjustment of $\mathbf{K}$6, random contrast adjustment of $\mathbf{K}$7, motion blur, Gaussian noise, and elastic deformation to simulate local non-rigid tissue motion (Yang et al., 7 Aug 2025).

5. Evaluation protocols and empirical significance

Endo-Mix6 is primarily a pretraining resource rather than the principal evaluation benchmark. Main performance reporting is conducted on external, unseen datasets: Hamlyn and Bladder for dense matching evaluation, and Gastro-Matching for downstream homography and motion-direction tasks (Yang et al., 7 Aug 2025). Within some real domains, including Ours-Bronch and SCARED, manually curated validation and test subsets are used for ablations.

The evaluation protocol includes several metric families. For curated internal ablations, the paper uses Percentage of Correct Keypoints,

$\mathbf{K}$8

For zero-shot dense matching on Hamlyn and Bladder, the reported metrics are the number of initial matches $\mathbf{K}$9, the number of geometrically verified inlier matches $(\mathbf{R}, \mathbf{t})$0, and the keep ratio,

$(\mathbf{R}, \mathbf{t})$1

For Gastro-Matching, the paper reports Homography Estimation Accuracy and Matching Direction Prediction Accuracy (MDPA).

The abstract reports that EndoMatcher, pretrained on Endo-Mix6, increases the number of inlier matches by 140.69% on Hamlyn and 201.43% on Bladder over state-of-the-art methods, and improves MDPA by 9.40% on Gastro-Matching (Yang et al., 7 Aug 2025). In the ablation comparing training regimes, Endo-Mix6-PMO yields the strongest zero-shot results among Endo-Syn, Endo-Real, Endo-Mix6, Endo-Mix6-MO, and Endo-Mix6-PMO. On Hamlyn, KR rises from 68.33 for Endo-Syn to 73.40 for Endo-Mix6-PMO. On Hamlyn, $(\mathbf{R}, \mathbf{t})$2 increases from 5848.57 for Endo-Syn to 6375.70 for Endo-Mix6-PMO. On Bladder, $(\mathbf{R}, \mathbf{t})$3 improves from 5570.94 for Endo-Real to 6116.20 for Endo-Mix6-PMO.

These results support a narrow but important interpretation: the dataset’s diversity is empirically valuable, but the paper argues that it must be paired with careful balancing and progressive scheduling. A plausible implication is that Endo-Mix6 is not merely a larger pool of image pairs; it is also a training regime stress test because of pronounced cross-domain imbalance, label heterogeneity, and synthetic-to-real shift.

6. Position in the literature, practical use, and limitations

Relative to prior endoscopic datasets, Endo-Mix6 is distinguished by integrating multiple organs and modalities into a unified matching-oriented corpus (Yang et al., 7 Aug 2025). Hamlyn is described as a kidney laparoscopy resource without dense correspondence labels. EndoSLAM provides synthetic depth and pose but is not by itself a multi-organ matching corpus. SCARED focuses on stereo laparoscopy and depth-related tasks. EndoMapper is highly useful but primarily colon-specific. Bladder Tissue and Gastro-Matching are task-specific rather than general multi-domain pretraining datasets. Endo-Mix6 differs by combining synthetic depth/pose supervision, SfM-derived correspondences, and simulated perspective transformations under a single correspondence-generation framework.

In practical terms, Endo-Mix6 is not released in the paper as a single packaged dataset with an explicit download link or license. The source paper provides code for EndoMatcher at https://github.com/Beryl2000/EndoMatcher, but the dataset itself is assembled from public datasets—C3VD, EndoSLAM, SCARED, EndoMapper, and Colonoscopic—together with a proprietary clinical bronchoscopy collection, Ours-Bronch. Reproducing Endo-Mix6 therefore entails obtaining the component datasets, acquiring or requesting the bronchoscopic videos, and running the official label-generation pipeline.

The paper also states several limitations. It acknowledges that more organs, modalities, and pathological conditions should be included for more comprehensive generalization. It further notes that latency-critical applications such as real-time SLAM and intraoperative navigation still require additional optimization. These caveats are consistent with the dataset’s stated role: Endo-Mix6 is a pretraining foundation for generalizable endoscopic matching, not a claim of exhaustive clinical coverage or deployment readiness (Yang et al., 7 Aug 2025).

A common terminological ambiguity is that the prefix “endo-” is heavily overloaded across technical literatures. In the present context, Endo-Mix6 denotes an endoscopic image-matching dataset and is unrelated to endomorphism-theoretic uses such as endo-commutative algebras, endo-$(\mathbf{R}, \mathbf{t})$4-permutation modules, or boundedly endo-rigid mixed abelian groups (Takahasi et al., 28 Jul 2025, Lassueur et al., 2017, Asgharzadeh et al., 2022). This suggests that the identifier should be interpreted domain-specifically: within computer vision and robot-assisted surgery, Endo-Mix6 names a multi-domain data resource whose significance lies in correspondence supervision, cross-organ transfer, and multi-objective pretraining rather than in algebraic or representation-theoretic notions.