Hierarchical VQA for Surgical Triplet Segmentation

• The paper introduces triplet segmentation for surgical videos that jointly classifies, detects, and localizes instrument-verb-target pairs with pixel-level precision.
• It leverages a transformer-based TargetFusionNet and gated cross-attention to infuse weak anatomical priors, enhancing segmentation and target accuracy.
• Quantitative and qualitative evaluations demonstrate significant mAP improvements in recognition, detection, and spatial grounding over conventional CAM and frame-level methods.

Hierarchical Visual Question Answering (VQA) in the surgical domain, exemplified by the DermaBench setting, centers on the spatially grounded recognition of <instrument, verb, target> triplets at the instance level in minimally invasive surgery videos. The task, termed "triplet segmentation," aims not only to classify and detect individual surgical actions but also to localize each action spatially—down to the pixel-precise mask of each instrument as it interacts with anatomical structures. This multifaceted formulation establishes a unified benchmark uniting recognition, detection, and fine-grained segmentation, advancing interpretability and robustness in surgical scene understanding (Alabi et al., 1 Nov 2025).

1. Formulation of Triplet Segmentation

Triplet segmentation is operationalized as follows: for every annotated frame, the system outputs, for each instrument instance, (1) a mask $\hat{M}_k$ identifying its spatial extent, (2) the action verb $v_k$ being performed, and (3) the anatomical target $t_k$. Critically, each predicted triplet $(I_k, v_k, t_k)$ is spatially grounded by a binary (or probabilistic) mask. In supervised settings such as CholecTriplet-Seg, annotations provide dense linkage between instance masks, action labels, and anatomical targets, enabling strong supervision at both the semantic and pixel levels. This contrasts sharply with prior frame-level or class activation map (CAM) methods, which lack the granularity to link specific instruments to their actions and targets robustly.

2. Architecture: TargetFusionNet and Target-Aware Fusion

TargetFusionNet, a transformer-based architectural extension of Mask2Former, is developed to address the triplet segmentation task. Its key innovation is the introduction of anatomical priors via a gated cross-attention mechanism—referred to as target-aware fusion—within each transformer decoder layer. The primary inputs and internal representations comprise:

• An RGB input image $x \in \mathbb{R}^{H \times W \times 3}$.
• Weak anatomical logits $L \in \mathbb{R}^{H \times W \times C_a}$ from a pretrained EndoViT tissue-segmentation model ($C_a = 6$ coarse tissue classes).
• Pixel-level feature maps $$F_\ell \in \mathbb{R}^{(H/2^\ell)\times(W/2^\ell)\times d}$$, extracted by a ResNet-50 backbone plus pixel decoder.
• Anatomy feature maps $$A_\ell \in \mathbb{R}^{(H/2^\ell)\times(W/2^\ell)\times d}$$, derived from $L$ via a small convolutional encoder.
• A fixed set of $v_k$0 learnable query embeddings $v_k$1.

In each decoder layer, queries $v_k$2 (from the previous layer) first perform standard Mask2Former cross-attention over $v_k$3. The target fusion module then computes cross-attention over $v_k$4, yielding aggregated anatomical context $v_k$5 for each query. This is followed by a gating operation: a sigmoid-activated scalar $v_k$6 is applied channelwise to $v_k$7, and the resulting gated anatomy feature $v_k$8 (elementwise multiplication) is added to the query, producing the updated $v_k$9. This fusion is mathematically formalized as:

• $t_k$0, where $t_k$1, $t_k$2, $t_k$3 (standard transformer projections).
• $t_k$4, with a learnable $t_k$5 and sigmoid $t_k$6.
• $t_k$7.

This repeated, gated anatomical conditioning is essential for accurate anatomical target prediction and refining query representations with contextually relevant information.

3. Prediction Heads, Loss Functions, and Optimization

At the output of the decoder, each query yields two heads:

• Mask head: Computes mask coefficients which, via dot product with pixel features, yield the instance mask $t_k$8.
• Classification head: Applies a softmax over $t_k$9 triplet classes, each encoding a unique (instrument, verb, target) combination, resulting in $(I_k, v_k, t_k)$0. An optional multi-task extension replaces this with three parallel heads predicting instruments ($(I_k, v_k, t_k)$1), verbs ($(I_k, v_k, t_k)$2), and targets ($(I_k, v_k, t_k)$3).

A one-to-one Hungarian matching $(I_k, v_k, t_k)$4 is used to align predictions with ground-truth triplets during training. The total loss is:

$(I_k, v_k, t_k)$5

where

$(I_k, v_k, t_k)$6

For the multi-task setup, the triplet class cross-entropy is replaced by separate losses on instrument, verb, and target heads with corresponding weights.

4. Training Protocol and Implementation Details

TargetFusionNet is implemented in the MMDetection framework with a ResNet-50 backbone pre-trained on CholecTriplet-Seg instrument-segmentation data. EndoViT's tissue segmentation outputs, used as weak anatomical logits, are fed through a dedicated anatomy encoder (three 3×3 convolutional layers). Data augmentation includes random horizontal flips and crop/scale, with images resized to $(I_k, v_k, t_k)$7. Training is conducted with AdamW (initial learning rate $(I_k, v_k, t_k)$8, weight decay $(I_k, v_k, t_k)$9, betas $x \in \mathbb{R}^{H \times W \times 3}$0, $x \in \mathbb{R}^{H \times W \times 3}$1) for 300k iterations. The best checkpoint is selected by validation mask mean Average Precision ($x \in \mathbb{R}^{H \times W \times 3}$2). Loss weights are set as $x \in \mathbb{R}^{H \times W \times 3}$3 (mask loss splits evenly between Dice and BCE).

5. Quantitative Performance and Ablation Results

TargetFusionNet achieves state-of-the-art triplet segmentation performance on CholecTriplet-Seg, particularly for accurate anatomical target linking. Ablation studies isolate the contribution of the gated cross-attention fusion. The following results (as $x \in \mathbb{R}^{H \times W \times 3}$4 in %) compare fusion strategies for full triplet $x \in \mathbb{R}^{H \times W \times 3}$5instrument+verb+target$x \in \mathbb{R}^{H \times W \times 3}$6 segmentation:

Fusion Strategy	mAP_I	mAP_V	mAP_T	mAP_{IVT}
Early-concat	69.13	50.74	21.52	12.21
Late-concat	59.90	40.65	20.21	10.99
TargetFusionNet	67.19	46.27	21.55	13.47

TargetFusionNet's gated cross-attention achieves the highest full triplet segmentation mAP (13.47%), outperforming both early- and late-concatenation of anatomical features.

Comprehensive end-to-end results demonstrate TargetFusionNet's improved performance across recognition (mAP^{rec}), detection (mAP^{det}), and segmentation (mAP^{seg}) for full triplets:

• $x \in \mathbb{R}^{H \times W \times 3}$7 (vs. $x \in \mathbb{R}^{H \times W \times 3}$8 for Mask2Former-Triplet)
• $x \in \mathbb{R}^{H \times W \times 3}$9 (vs. $L \in \mathbb{R}^{H \times W \times C_a}$0)
• $L \in \mathbb{R}^{H \times W \times C_a}$1 (vs. $L \in \mathbb{R}^{H \times W \times C_a}$2)

Notably, the pronounced gain in target accuracy (mAP_T) underscores the benefit of anatomical context integration.

6. Qualitative Insights and Diagnostic Scenarios

Qualitative analysis of challenging surgical frames reveals that TargetFusionNet robustly disambiguates instances where action-target distinction is subtle (e.g., clipping cystic duct vs. artery, irrigating fluid vs. tissue manipulation). In cases with occlusions, multiple interacting instruments, or rare target classes, TargetFusionNet links each instrument mask to the correct verb and anatomical target more reliably than CAM-based grounding (e.g., RDV-Det) or vanilla Mask2Former-Triplet. The refined semantic alignment produced by gated anatomy fusion is most evident on actions that require nuanced tissue-type discrimination.

7. Context, Significance, and Implications

Triplet segmentation formalizes hierarchical VQA as a spatially explicit, instance-level grounding problem, directly addressing the limitations of frame-level or weakly supervised recognition approaches. The introduction of weak anatomical priors, delivered through TargetFusionNet's gated cross-attention mechanism, contributes significantly to target-level accuracy without sacrificing mask quality or verb/instrument recognition. This unified approach facilitates richer surgical scene interpretation and paves the way for robust, interpretable AI assistance in minimally invasive procedures (Alabi et al., 1 Nov 2025).

A plausible implication is that future extensions of this methodology could generalize to other hierarchical, instance-grounded VQA tasks beyond surgical video analysis, particularly where weak spatial priors can be leveraged to improve fine-grained action recognition.