MedCAGD: Context-Aware Gated Decoder for Efficient Medical Image Segmentation

Abstract: Medical image segmentation relies on the ability of encoder-decoder architectures to translate rich feature representations into accurate pixel-level predictions under challenging conditions such as low contrast, structural ambiguity, and scale variability. While recent advances in large-scale pretraining and transformer-based encoders have substantially improved feature extraction, segmentation accuracy remains constrained by decoder design, particularly in terms of cross-scale alignment, contextual integration, and boundary preservation. In this work, we revisit medical image segmentation from a decoder-centric perspective and propose a context-aware gated decoder that systematically regulates feature fusion and contextual aggregation throughout the decoding process. The proposed decoder integrates lightweight multi-scale channel recalibration, gated skip fusion with spatial competition and a global context aggregation mechanism that injects encoder-wide information into intermediate decoding stages. This design enables effective translation of strong pretrained encoder representations into spatially consistent predictions. Extensive experiments across 11 medical image segmentation benchmarks validate the effectiveness and demonstrate that the proposed approach consistently outperforms strong baselines while remaining computationally practical. Code: https://github.com/saadwazir/MedCAGD

• The paper introduces a novel decoder-centric framework that uses adaptive gating and multi-scale attention to enhance segmentation performance, achieving an average Dice of 88.01.
• It integrates spatially competitive gated skip fusion and context aggregation with residual attention to reduce segmentation errors and ensure better anatomical alignment.
• The framework proves efficient for clinical applications by optimizing segmentation through deep and edge supervision while maintaining moderate parameters (30.60M) and 5.0 GFLOPs.

MedCAGD: A Context-Aware Gated Decoder for Efficient Medical Image Segmentation

Introduction and Motivation

The MedCAGD framework introduces a decisive shift in medical image segmentation by repositioning the decoder as the primary locus of architectural innovation. While transformer-based and foundation model encoders have saturated global feature extraction capacity, segmentation performance remains bounded by the limitations of existing decoder designs—specifically, their inability to effectively realign multi-scale semantics, integrate contextual dependencies, and preserve anatomical boundaries under the constraints of efficiency and parameter budget. MedCAGD proposes a principled decoder-centric approach, integrating adaptive gating, global context aggregation, and fine-grained structural refinement, decoupling advances from dependence on ever-growing encoder complexity.

Figure 1: MedCAGD architecture: multi-scale encoder features (b) are projected and sequentially processed via Bottleneck with ECA-MSP and RA (c, f, e), SCA-Gate (g) for competitive skip fusion, Context Aggregator (h), and Refinement Block (d) at each decoding stage.

Architectural Contributions

Efficient Multi-Scale Channel and Spatial Attention

MedCAGD introduces a decoder pipeline wherein multi-scale encoder features are aligned via $1 \times 1$ convolutions, followed by a sequence of highly structured modules. Efficient Channel Attention with Multi-scale Pooling (ECA-MSP) realizes channel-wise recalibration without the inefficiency of global pooling, constructing descriptive statistics at granularities $\mathcal{S} = \{1, 2, 4\}$ to encode multi-scale context. These are combined via parameterized 1D convolutions and merged through aggregation, enforcing channel selectivity informed both by the global scene and localized structure. This design outperforms classical SE/CBAM/ECA blocks by encoding richer, complementary context, with ablation results confirming notable improvements in Dice and boundary measures.

Spatially Competitive Gated Skip Fusion

Skip connections in classic encoder-decoder networks indiscriminately transmit low-level encoder features, frequently introducing decoder-encoder semantic mismatch that propagates segmentation errors. The proposed Spatially Competitive Attention Gate (SCA-Gate) processes encoder and decoder features through ECA-MSP, projects them into a unified latent space, and formulates skip fusion as a normalized (temperature-controlled softmax) multiplicative agreement between the two streams, modulated by both spatial competition (parallel depthwise convolutions) and global context. The result is a highly selective mechanism excising semantically incompatible skip features and improving anatomical fidelity.

Global Context Aggregation and Residual Attention

MedCAGD systematically injects multi-level encoder context into every decoding stage using a Context Aggregator that aligns, averages, and projects encoder features, subsequently refined with residual attention (RA) that leverages non-local feature integration mediated by efficient fully-connected layers. This alleviates cross-scale misalignment and enhances semantic consistency, particularly vital for structures displaying strong anatomical variability or ambiguous boundaries.

Sequential Refinement and Supervision

After gated fusion, decoder features are refined by a local block combining depthwise separable and pointwise convolutions, group normalization, and SiLU activation, culminating in further channel recalibration by ECA-MSP. Deep Supervision (DS) and Edge Supervision (ES) are applied at intermediate layers, optimizing both primary and auxiliary segmentation and edge outputs. As shown by ablation radar plots, the combination of DS and ES yields substantial and consistent gains in both Dice and HD95, indicating improved interior prediction and sharper contours.

Figure 2: Radar plots: enabling both deep and edge supervision yields maximized Dice (outer rings) and minimized HD95 (center), outperforming all alternatives.

Experimental Results

MedCAGD was benchmarked across 11 datasets covering dermoscopy, endoscopy, ultrasound, retina, microscopy, CT, and MRI. It consistently demonstrates SOTA performance across both binary and multi-class tasks, as evidenced by average Dice, IoU, and HD95 on datasets such as ISIC, ETIS, ColonDB, DRIVE, FIVES, BUSI, ThyroidXL, CellSeg, Synapse, and ACDC.

Quantitatively, MedCAGD achieves an average Dice of 88.01 on a composite of 9 benchmarks—outperforming MCADS (85.14) and EMCAD (84.87)—while retaining a moderate model size (30.60M parameters, 5.0 GFLOPs), thus surpassing pure transformer or Mamba-based encoder-decoder competitors not just in accuracy but also in practical efficiency. On Synapse and ACDC, MedCAGD reports the highest overall Dice (87.00 and 87.54, respectively) and the lowest HD95, manifesting both superior segmentation consistency and sharper anatomical boundaries. Importantly, clinically relevant small or structurally variable regions (e.g., vessels in fundus, neoplasms, cardiac substructures) benefit distinctively from the proposed decoder's cross-scale alignment and competitive gating.

Figure 3: Qualitative results: MedCAGD exhibits fewer segmentation errors (red boxes) and better contour fidelity compared to strong baselines across multiple modalities.

Comparative Analysis and Ablation Evidence

Comparative studies of skip attention designs demonstrate that SCA-Gate delivers the highest mean Dice (87.00 on Synapse, 86.61 on CellSeg), outperforming Attention U-Net, ECA-supplemented gates, and even competing SOTA decoders such as MCADS and EMCAD. Componentwise ablation on decoder elements further shows that the cumulative integration of Bottleneck, Context Aggregation with Residual Attention, Refinement Block, and SCA-Gate is strictly additive: the removal of any component systematically degrades performance.

Resolution and backbone studies confirm generalizability—MedCAGD is encoder-agnostic and benefits when paired with stronger/pretrained backbones, but absolute gains over baseline decoders are preserved, underscoring that performance improvements are fundamentally decoder-driven rather than an artifact of scalable encoders.

Implications and Future Directions

The MedCAGD design formalizes the hypothesis that architectural innovation in the decoder pipeline—rather than chasing greater encoder capacity—now provides the dominant returns in medical image segmentation. This has practical implications for resource-constrained deployment in clinical imaging, where increasing backbone scale is cost- or latency-prohibitive, and accuracy must be coupled with model efficiency and interpretability. The supervised auxiliary outputs lend themselves to greater model transparency for clinical acceptance, especially in tasks sensitive to boundary precision.

On a theoretical level, the work demonstrates the efficacy of competitive multiplicative attention for skip fusion and validates deep, multi-point contextual supervision for dense prediction. Extension to OOD generalization, full 3D segmentation pipelines, and comparative studies with next-generation foundation models (e.g., SAM variants, MedicalSAM3) remains unexplored territory.

Conclusion

MedCAGD demonstrates that decoder-centric reformulation, featuring structured multi-scale channel and spatial attention, competitive gated skip fusion, residual context injection, and holistic supervision, yields consistent and substantial improvements in medical image segmentation accuracy, boundary preservation, and computational efficiency. This work motivates prioritizing decoder architecture advances as the key driver of future segmentation frameworks in both specialist and emerging foundation model-driven settings.