Geometry-Enhanced Multi-scale Attention (GEMA)

• The paper introduces GEMA as a novel attention mechanism that injects depth-derived geometric priors into RGB feature extraction for surgical phase recognition.
• GEMA strategically integrates lightweight geometry cues via the DGPG module at an intermediate network stage, balancing spatial detail and semantic abstraction.
• GEMA unifies geometric structure with multi-scale aggregation to direct attention toward structurally informative regions while enhancing robustness under challenging visual conditions.

Geometry-Enhanced Multi-scale Attention (GEMA) is an attention module introduced in Geo-RepNet for surgical phase recognition in Endoscopic Submucosal Dissection (ESD). Its defining role is to inject depth-derived geometric structure into RGB feature extraction through geometry-aware cross-attention and efficient multi-scale aggregation. In the paper’s formulation, GEMA is not an auxiliary attention block added for architectural symmetry; it is a targeted mechanism for using geometric priors from depth to guide attention toward structurally informative regions such as tool–tissue interfaces, incision boundaries, and relative spatial layout, while reducing emphasis on visually confusing or texture-deficient regions (Tang et al., 12 Jul 2025).

1. Task setting and motivating problem

GEMA is motivated by the failure modes of RGB-only surgical phase recognition. Geo-RepNet identifies several recurrent difficulties in ESD videos: high inter-phase visual similarity, limited structural cues in RGB, low-texture tissue, occlusion and visibility problems, blood, specular highlights, camera motion, and the need for lightweight, real-time, data-efficient modeling. In that setting, ordinary attention mechanisms are described as structurally uninformed because they compute affinities from appearance features alone, which may be unreliable under smoke, reflections, blur, tissue translucency, or other appearance degradations (Tang et al., 12 Jul 2025).

The module is therefore designed around a specific epistemic claim: depth offers geometric priors that encode relative positions, distance relationships, visibility or occlusion structure, and local shape or layout cues that RGB cannot capture reliably. Geo-RepNet argues that this kind of geometry-aware attention can align feature aggregation with true scene structure rather than superficial appearance similarity. The intended effect is improved discrimination between phases that share similar color and texture statistics but differ in spatial configuration, local boundaries, or instrument–anatomy relations (Tang et al., 12 Jul 2025).

A second point of motivation is efficiency. Geo-RepNet explicitly avoids a heavy dual-stream RGB-depth encoder. Instead, raw depth is converted into lightweight geometric priors by the Depth-Guided Geometric Prior Generation (DGPG) module, and GEMA is then used as the mechanism that injects those priors into RGB feature learning. This design addresses a dual constraint: depth should contribute structural information, but direct depth encoding may be inefficient and may amplify incomplete or noisy depth (Tang et al., 12 Jul 2025).

2. Architectural placement in Geo-RepNet

Geo-RepNet is built on a re-parameterizable RepVGG backbone, and both DGPG and GEMA are inserted after Stage 1 of that backbone. The stated reason for this placement is a representational trade-off. Too early in the network, the features have high spatial resolution but insufficient semantic abstraction; too deep in the network, the features are too downsampled and lose local geometric detail. Stage 1 is presented as the balance point where local boundaries, tool–tissue contact patterns, and sufficient contextual semantics remain simultaneously available (Tang et al., 12 Jul 2025).

At that location, GEMA receives the Stage-1 RGB feature map, written either as $\mathbf{f}_1$ or more generically as $X \in \mathbb{R}^{B \times C \times H \times W}$. Its geometry input comes from DGPG and is described in two equivalent ways. In the architectural overview, DGPG produces sinusoidal and cosine relative positional encodings together with visibility-related geometric guidance at the spatial resolution of the Stage-1 feature map, written as $(\mathbf{r}_{\sin}, \mathbf{r}_{\cos})$ and a visibility term. In the GEMA subsection, the geometry input is written more abstractly as $G \in \mathbb{R}^{H \times W \times 2}$ and decomposed as $G_x, G_y = \mathcal{T}(G)$ (Tang et al., 12 Jul 2025).

The output is a geometry-enhanced feature map derived from the Stage-1 RGB representation and the DGPG priors. This placement makes GEMA an intermediate fusion operator rather than a late decision module or a front-end preprocessor. Architecturally, it is the point at which geometric structure is injected into the main RGB stream before higher-level semantic encoding proceeds (Tang et al., 12 Jul 2025).

3. Geometric priors and attention mechanism

The functional core of GEMA is the use of depth-derived geometric priors to bias attention. Geo-RepNet states that GEMA uses positional encodings to emphasize subtle spatial layout differences and visibility masks to direct attention away from texture-deficient regions toward informative structures. In that sense, geometry is not merely appended as an additional feature tensor; it modulates the attention process itself, shaping which regions are emphasized and which interactions are deemphasized during representation learning (Tang et al., 12 Jul 2025).

This design is tightly linked to the task-specific semantics of surgical scenes. Tool–tissue interfaces, incision boundaries, and relative spatial layout are treated as structurally meaningful regions that RGB-only attention may miss or weight unreliably. GEMA is meant to bias the network toward those regions by using depth-derived priors that preserve spatial relations even when appearance cues are degraded. The paper’s argument is that such priors can suppress responses in unreliable areas while making subtle structural distinctions more salient (Tang et al., 12 Jul 2025).

GEMA also couples geometry-aware attention with efficient multi-scale aggregation. The paper presents this pairing as essential: geometry supplies structural guidance, while multi-scale context modeling helps capture both local detail and broader scene organization. A plausible implication is that GEMA should be understood less as a single attention formula than as a fusion principle in which geometric bias and scale aggregation are co-designed. In Geo-RepNet, that principle is explicitly implemented without resorting to a heavy RGB-depth dual encoder, which is central to its efficiency claim (Tang et al., 12 Jul 2025).

4. Multi-scale role, efficiency, and empirical function

The “multi-scale” component of GEMA in Geo-RepNet is not an incidental descriptor. The paper states that GEMA provides efficient multi-scale aggregation, and it repeatedly links the module to the need for combining local geometric consistency with broader contextual reasoning. In surgical phase recognition, this is consequential because many discriminative signals are small and local, while phase identity also depends on larger scene layout and instrument progression. GEMA is therefore positioned as a representation-learning mechanism that exploits geometric structure without expensive explicit depth encoding and combines geometry-aware attention with multi-scale context modeling (Tang et al., 12 Jul 2025).

Geo-RepNet uses GEMA together with DGPG inside a geometry-aware convolutional framework built on RepVGG. The broader system is evaluated on a nine-phase ESD dataset with dense frame-level annotations from real-world ESD videos, and the paper reports that Geo-RepNet achieves state-of-the-art performance while maintaining robustness and high computational efficiency under complex and low-texture surgical environments. Within that system-level result, GEMA is the module responsible for injecting spatial guidance derived from geometry priors into the RGB feature hierarchy (Tang et al., 12 Jul 2025).

A recurring misunderstanding is to treat GEMA as equivalent to simple RGB-depth feature concatenation. Geo-RepNet explicitly rejects that formulation. Depth is first reduced to lightweight geometric priors by DGPG, then GEMA injects those priors into the RGB stream after Stage 1. This suggests that GEMA is best interpreted as a guided attention-and-aggregation operator rather than a generic multimodal fusion block (Tang et al., 12 Jul 2025).

5. Related formulations across the literature

Although the term “GEMA” is specific to Geo-RepNet, closely related formulations appear across multiple subfields. In image classification, “Explicitly Modeled Attention Maps for Image Classification” replaces learned query–key maps with geometry-prior attention kernels such as Gaussian functions with a learnable radius, arguing that the value of self-attention can in part be captured by explicit spatial geometry (Tan et al., 2020). In multi-view 3D vision, “GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers” encodes geometry as relative transformations acting on features in attention rather than as additive positional embeddings, thereby grounding attention in camera geometry (Miyato et al., 2023).

In self-supervised monocular depth estimation, “Attention meets Geometry” uses a coarse predicted depth map to define geometry-induced spatial attention before temporal attention is applied across consecutive frames, providing a coarse-to-fine example of geometry-guided aggregation (Ruhkamp et al., 2021). In point cloud registration, IGASA introduces a Hierarchical Pyramid Architecture together with Hierarchical Cross-Layer Attention and skip attention mechanisms to align multi-resolution features and enhance local geometric consistency, which is strongly analogous to a geometry-enhanced multi-scale attention design even though the paper does not use the term GEMA (Zhang et al., 13 Mar 2026).

Related patterns also appear outside vision tasks narrowly construed. GeoTransolver replaces standard attention with GALE, coupling physics-aware self-attention with cross-attention to shared geometry, global, and boundary-condition context computed from multi-scale ball queries; this is a geometry-aware and explicitly multi-scale attention formulation for irregular domains (Adams et al., 23 Dec 2025). SemGeo-AttentionNet uses a Point Transformer V3 hierarchy together with asymmetric cross-attention in which geometric features query semantic content, making geometry the retrieval source for another modality in 3D saliency estimation (Pahari et al., 6 Feb 2026).

These works suggest that GEMA is plausibly understood as a broader design family characterized by three recurring commitments: attention should be structurally informed by geometry, feature interaction should account for more than one spatial scale, and geometric information should enter the aggregation mechanism rather than being relegated to late fusion. That generalization is interpretive, but it is strongly supported by the recurring architecture patterns across these papers.

6. Misreadings, boundaries, and conceptual scope

Not every geometry-aware attention mechanism is multi-scale, and not every multi-scale attention mechanism is explicitly geometric. GTA, for example, is strongly geometry-aware and multi-view, but it is only weakly multi-scale in the conventional feature-pyramid sense (Miyato et al., 2023). Local-Global Attention integrates local and global contextual features through multi-scale convolutions and positional encoding, but it does not explicitly model geometry in the stronger sense of relative position bias, box geometry, or deformable geometric relations (Shao, 2024). An attention-driven hierarchical multi-scale representation for visual recognition constructs hierarchical multi-scale region graphs and uses attention-based message passing, yet its geometry remains largely implicit in the predefined hierarchy and graph structure rather than being explicitly encoded in the attention coefficients (Wharton et al., 2021).

A similar distinction appears in geometry-enhanced multimodal systems. GeoVLN learns geometry-enhanced visual representation from RGB, depth, normals, and viewpoint angle features, and it uses local-aware slot attention plus a multiway attention module, but its attention is primarily local spatial and modality-wise rather than explicitly multi-scale in a pyramid sense (Huo et al., 2023). For that reason, GeoVLN is better treated as conceptually adjacent to GEMA than as a direct instance of it.

Against that background, GEMA in Geo-RepNet occupies a comparatively clear position. It is explicitly defined as geometry-enhanced and multi-scale; it is inserted at an intermediate feature stage where spatial detail and semantic abstraction are both present; and it is motivated by a concrete structural deficiency in RGB-only surgical modeling. The module’s encyclopedic significance lies less in a single canonical equation than in this design logic: depth-derived geometry priors are converted into lightweight guidance signals and then used to steer attention and multi-scale aggregation toward structurally informative regions under conditions where appearance is ambiguous or unreliable (Tang et al., 12 Jul 2025).