Geo-RepNet: Geometry-Aware Surgical Phase Recognition

• Geo-RepNet is a geometry-aware convolutional network that fuses RGB images with depth-derived geometric priors via attention mechanisms for precise ESD phase recognition.
• It leverages a re-parameterizable RepVGG backbone augmented by specialized DGPG and GEMA modules to integrate structural guidance with efficient feature learning.
• Empirical results on a nine-phase real-world ESD dataset demonstrate significant improvements with 85.02% accuracy, 81.74% F1, and 93.10% AUC.

Searching arXiv for the target paper and closely related endoscopic depth/representation works. arXiv search query: "Geo-RepNet surgical phase recognition Endoscopic Submucosal Dissection depth" Geo-RepNet is a geometry-aware convolutional network for surgical phase recognition in Endoscopic Submucosal Dissection (ESD). It addresses a central difficulty in ESD video understanding: different operative phases can be visually similar in RGB frames, while the distinctions that matter clinically often depend on subtle spatial configurations such as tool–tissue relationships, flap elevation, and local surface geometry. The model integrates RGB images with depth-derived geometric priors, but it does so in a single-stream formulation rather than as a full dual-modality backbone: depth is distilled into lightweight structural guidance that modulates RGB feature learning through attention mechanisms. In the reported evaluation on a nine-phase real-world ESD dataset, Geo-RepNet achieves 85.02% accuracy, 81.74% F1, and 93.10% AUC (Tang et al., 12 Jul 2025).

1. Clinical setting and problem formulation

Endoscopic Submucosal Dissection is a minimally invasive procedure for removing early-stage gastrointestinal tumors en bloc. Its workflow comprises fine-grained phases such as forceps grasp, forceps lift the mucosa flap, knife lifts the mucosa flap, camera adjustment, lens cleaning, and knife dissection. Automatic recognition of these phases is relevant to online assistance and guidance, workflow analysis and skill assessment, automation or semi-automation of certain steps, and post-operative documentation and quality control (Tang et al., 12 Jul 2025).

The recognition problem is unusually difficult in ESD because phases are both fine-grained and visually similar. Many steps involve the same instruments and comparable tissue appearance, while the decisive information often lies in spatial patterns rather than gross texture or color. The operative field further includes low texture, specular highlights, smoke, blood, occlusions, and non-rigid tissue. In that setting, RGB-only phase recognition suffers from high intra-class variability and high inter-class ambiguity (Tang et al., 12 Jul 2025).

Depth is introduced as a complementary geometric signal. The reported motivation is that depth provides relative distances and 3D layout, tissue shape and curvature, whether the mucosal flap is lifted or flat against the wall, and instrument protrusion or insertion depth. The paper assumes a depth map for each frame, obtained from monocular depth estimation on endoscopic RGB video. It also explicitly notes that raw depth maps are noisy and incomplete, and therefore not reliable as a stand-alone modality. A common misconception is that Geo-RepNet is a conventional RGB–depth dual-stream architecture; in fact, the method uses depth as structural guidance rather than as a full second feature stream (Tang et al., 12 Jul 2025).

2. System architecture and backbone design

Geo-RepNet is organized around a re-parameterizable RepVGG backbone augmented by two specialized modules: Depth-Guided Geometric Prior Generation (DGPG) and Geometry-Enhanced Multi-scale Attention (GEMA). The pipeline begins with an RGB image $\mathbf{x} \in \mathbb{R}^{H \times W \times 3}$ and a depth map $\mathbf{d} \in \mathbb{R}^{H \times W}$. The RGB image is processed by Stage 1 of RepVGG to produce a feature map $\mathbf{f}_1$, while the depth map is processed by DGPG to produce sinusoidal relative positional encodings and a geometry-aware mask at the same spatial resolution. These outputs are then injected into GEMA, after which the enhanced feature map is forwarded through the remaining RepVGG stages, followed by global pooling and a classifier (Tang et al., 12 Jul 2025).

RepVGG provides the appearance backbone. At training time, each block is multi-branch, with a $3 \times 3$ convolution, a $1 \times 1$ convolution, and often an identity branch. At inference time, these branches are algebraically fused into a single $3 \times 3$ convolution, with batch normalization parameters folded into the convolution. The reported rationale is twofold: training-time flexibility and regularization, together with inference-time simplicity and speed. The paper further argues that RepVGG has a strong local inductive bias, which is advantageous in surgical video where the decisive changes are often local and where data availability is limited (Tang et al., 12 Jul 2025).

The architectural choice is therefore deliberately asymmetric. Only one CNN backbone processes appearance features, and depth enters exclusively through DGPG and GEMA. This suggests an attempt to preserve computational efficiency and deployment simplicity while avoiding overfitting to noisy depth estimates.

3. Depth-Guided Geometric Prior Generation

DGPG converts raw depth maps into compact priors used to bias attention. Its input is a batch of depth maps $$\mathbf{d} \in \mathbb{R}^{B \times 1 \times H \times W}$$, and its outputs are relative positional encodings $\text{PE}_{\sin}$ and $\text{PE}_{\cos}$ together with a geometric decay or visibility mask $\mathbf{M}_{\text{geo}}$. These outputs are explicitly described not as full-depth feature maps, but as low-dimensional guides for the attention mechanism (Tang et al., 12 Jul 2025).

The module first interpolates the depth map to the spatial size of the Stage 1 feature map. It then computes pairwise depth differences

$\mathbf{d} \in \mathbb{R}^{H \times W}$0

These differences are modulated through a head-specific exponential decay,

$\mathbf{d} \in \mathbb{R}^{H \times W}$1

where $\mathbf{d} \in \mathbb{R}^{H \times W}$2 indexes the attention head, $\mathbf{d} \in \mathbb{R}^{H \times W}$3 is the number of heads, and $\mathbf{d} \in \mathbb{R}^{H \times W}$4 and $\mathbf{d} \in \mathbb{R}^{H \times W}$5 are learnable parameters. The intended effect is to down-weight pairs with large depth differences more strongly, so that attention preferentially concentrates on geometrically coherent regions (Tang et al., 12 Jul 2025).

DGPG also computes sinusoidal relative positional encodings from relative displacements: $\mathbf{d} \in \mathbb{R}^{H \times W}$6 This is conceptually analogous to transformer sinusoids, but the encoding is relative rather than absolute. Positional and depth-derived masks are then combined as

$\mathbf{d} \in \mathbb{R}^{H \times W}$7

with learnable weights $\mathbf{d} \in \mathbb{R}^{H \times W}$8 and $\mathbf{d} \in \mathbb{R}^{H \times W}$9 (Tang et al., 12 Jul 2025).

A notable design point is the absence of direct supervision for DGPG. Its parameters, including $\mathbf{f}_1$0, $\mathbf{f}_1$1, $\mathbf{f}_1$2, and $\mathbf{f}_1$3, are optimized end-to-end through the classification objective alone. A second common misconception is therefore that Geo-RepNet trains a depth branch with explicit geometric losses; the reported method does not.

4. Geometry-Enhanced Multi-scale Attention

GEMA is the module that injects geometry into representation learning. It consists of geometry-aware cross self-attention and efficient multi-scale attention. The first submodule, described as CrossGSA, incorporates geometric priors into query and key computation. The second submodule, EMA, aggregates context along height and width at different scales and reweights channel and spatial responses (Tang et al., 12 Jul 2025).

Given a feature map $\mathbf{f}_1$4 and geometric prior $\mathbf{f}_1$5, CrossGSA computes

$\mathbf{f}_1$6

where $\mathbf{f}_1$7, $\mathbf{f}_1$8, and $\mathbf{f}_1$9 are $3 \times 3$0 convolutions or linear layers and $3 \times 3$1 is depthwise convolution. Queries and keys are then augmented with sinusoidal geometry encodings, and the attention matrix is computed as

$3 \times 3$2

where $3 \times 3$3 is the key dimension and $3 \times 3$4 is an optional geometry bias map derived from the geometric mask. The output is

$3 \times 3$5

The reported interpretation is that geometry modulates similarity scores directly, so attention is guided toward structurally coherent patterns such as tool–tissue boundaries and flap edges (Tang et al., 12 Jul 2025).

EMA then captures multi-scale context without heavy global attention. It performs directional average pooling along height and width,

$3 \times 3$6

processes the pooled features through convolutional layers and group normalization to obtain a context map $3 \times 3$7, and produces attention weights

$3 \times 3$8

The grouping factor controls channel grouping, and the ablation study reports that a factor of 4 yields the best performance, while the model remains robust across other tested factors (Tang et al., 12 Jul 2025).

The functional role of GEMA is scale-sensitive discrimination. Large-scale context is useful for phases such as camera adjustment, whereas high-resolution local context is needed for thin mucosal flaps and tool tips. This suggests that the model is designed not merely to add depth cues, but to reorganize feature saliency across scales.

5. Dataset, annotation protocol, and optimization setup

The evaluation uses a dedicated ESD phase dataset constructed from real-world ESD surgical videos, with dense frame-level annotations for nine distinct phases (Tang et al., 12 Jul 2025). The reported classes are listed below together with the train/test counts provided in the data.

Phase	Train	Test
Forceps grasp	81	9
Forceps lift the mucosa flap	504	57
Forceps and Knife move to the edge mucosa flap	444	51
Forceps rotate	21	3
Knife lifts the mucosa flap	405	48
Adjust the camera position	333	39
Clean the camera lens	303	36
Knife is dissecting the mucosa flap	2628	294
Knife is ready to dissect the mucosa flap	2361	264

The splits are reported as 7080 training samples and 792 validation/test samples. The dataset is explicitly imbalanced, with phases 8 and 9 dominating the distribution. The text identifies this imbalance as a challenge and does not report an explicit balancing strategy (Tang et al., 12 Jul 2025).

Training is performed for 150 epochs with Adam, an initial learning rate of $3 \times 3$9, cosine annealing, and batch size 16, using PyTorch on an NVIDIA RTX 4090. The learning objective is standard categorical cross-entropy,

$1 \times 1$0

No additional explicit loss is imposed on depth or DGPG, and the paper does not mention temporal smoothing, class weighting, or focal loss. This is relevant because the model is a single-frame classifier rather than a sequence model (Tang et al., 12 Jul 2025).

6. Empirical performance, ablation evidence, and limitations

On the ESD dataset, Geo-RepNet reports 85.02% accuracy, 81.74% F1, and 93.10% AUC. The selected baselines listed in the paper include Res2Net-50 at 75.66% accuracy, 76.29% F1, and 82.53% AUC; ConvNeXt at 74.53%, 76.19%, and 83.67%; CSWinTransformer at 77.01%, 78.56%, and 84.53%; and SwinTransformerV2 at 70.41%, 66.67%, and 81.41% (Tang et al., 12 Jul 2025).

The ablation study isolates the contributions of the depth-guided prior generator, geometry-aware self-attention, and efficient multi-scale attention. The baseline without these components reports 77.53 accuracy, 71.88 F1, and 83.74 AUC. The full model reports 85.02, 81.74, and 93.10. Intermediate configurations show that each component improves performance over baseline, with geometry-aware self-attention contributing strongly to F1 and multi-scale attention producing a large AUC gain. The paper further reports EMA grouping-factor sensitivity: factor 4 yields 85.02 / 81.74 / 93.10, factor 8 yields 84.40 / 80.98 / 90.42, factor 16 yields 84.77 / 78.78 / 91.57, and factor 32 yields 83.90 / 79.84 / 90.82 (Tang et al., 12 Jul 2025).

The qualitative discussion emphasizes that geometry priors help shift attention away from unreliable regions such as blood, smoke, and specular highlights, and toward structurally informative areas including incision boundaries, flap edges, and tool–tissue contacts. This suggests that the main empirical advantage is not simply additional modality information, but improved structural disambiguation in low-texture scenes (Tang et al., 12 Jul 2025).

Several limitations are explicit or implicit. The dataset is modest in size and restricted to ESD, so cross-procedure and cross-institution generalization is not established. Performance depends on the quality of monocular depth estimation. The model is frame-level and does not explicitly impose temporal consistency. Domain shift across endoscopes, hospitals, and anatomies is not addressed with a dedicated adaptation mechanism. The paper identifies plausible future extensions including temporal modeling, improved depth estimation, self-supervised or semi-supervised learning, domain adaptation, and broader surgical workflow analysis (Tang et al., 12 Jul 2025).

Taken together, Geo-RepNet occupies a specific position in surgical video understanding: it is neither a pure RGB classifier nor a heavy RGB–depth multimodal network, but a geometry-aware CNN in which depth acts as a lightweight prior for representation learning. Within that design space, the reported results support the claim that geometric reasoning can materially improve ESD phase recognition under complex and low-texture conditions (Tang et al., 12 Jul 2025).