FMD-TransUNet: Hybrid CNN-Transformer Segmentation

Updated 12 July 2026

The paper introduces FMD-TransUNet, a hybrid design that integrates frequency-domain learning via MEWB and dual attention via DA+ to refine CT segmentation.
It combines convolutional downsampling, Transformer encoding, and enhanced skip connections to address challenges like ambiguous boundaries, irregular organ shapes, and overlapping intensities.
Empirical results on the Synapse dataset show improved DSC from 77.48% to 81.32% and reduced HD from 31.69 mm to 16.35 mm, especially benefiting small organ segmentation.

FMD-TransUNet is a hybrid CNN–Transformer segmentation framework for abdominal multi-organ CT segmentation that augments the TransUNet architecture with two modules: the Multi-axis External Weight Block (MEWB) for frequency-domain representation learning and the improved Dual Attention module (DA+) based on depthwise separable convolutions. It is designed to address small organ size, irregular shapes, ambiguous boundaries, and overlapping intensity distributions by complementing spatial-domain features with frequency-domain cues and by reducing the semantic gap between encoder and decoder features (Lu et al., 19 Sep 2025).

1. Position within the TransUNet lineage

FMD-TransUNet extends the 2D TransUNet paradigm rather than replacing it. The original "TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation" uses a hybrid R50-ViT encoder, a U-shaped decoder, and skip connections to combine global self-attention with high-resolution CNN features (Chen et al., 2021). FMD-TransUNet preserves that backbone logic and inserts MEWB and DA+ at selected points where spatial and frequency representations can interact most effectively (Lu et al., 19 Sep 2025).

The architectural inheritance is consequential. In the original TransUNet configuration for Synapse, a 224×224 input and a stride-16 CNN feature map yield a 14×14 token grid, giving 196 tokens for the Transformer encoder (Chen et al., 2021). FMD-TransUNet keeps the R50-ViT configuration with patch size $P=16$ , uses images resized to 224×224, and follows TransUNet defaults for exact Transformer depths, heads, and inherited channel widths where those details are not explicitly re-enumerated (Lu et al., 19 Sep 2025).

Within the broader TransUNet family, "3D TransUNet: Advancing Medical Image Segmentation through Vision Transformers" reports that encoder-side Transformers are especially effective for multi-organ segmentation, whereas decoder-side query refinement is more beneficial for small and challenging segmented targets (Chen et al., 2023). This provides useful context for FMD-TransUNet: its design strengthens encoder-side global modeling through frequency-domain decomposition while also refining decoder fusion for fine anatomical detail (Lu et al., 19 Sep 2025).

2. Problem setting and overall architecture

Abdominal multi-organ CT segmentation is difficult because organ boundaries are often ambiguous due to noise and low contrast; organs such as the pancreas and kidneys have irregular shapes; intensity distributions overlap across adjacent structures such as the liver, spleen, and kidneys; and size disparity is severe, with large organs dominating the field of view while small structures occupy only a tiny fraction (Lu et al., 19 Sep 2025). Pure CNN methods are limited by restricted receptive fields, while Transformer-heavy methods provide global information but lack intrinsic mechanisms tailored to spatial position and channel semantics, and may propagate redundant features through skip connections (Lu et al., 19 Sep 2025).

FMD-TransUNet addresses these issues by extending TransUNet at both the encoder tail and the decoder stages. The encoder begins with three convolutional blocks for progressive downsampling and channel expansion. From 224×224, typical downsampling by stride 2 yields 224→112→56→28. After these convolutional blocks, MEWB is inserted to extract multi-axis frequency-domain features and fuse them with spatial features. DA+ is then applied to refine features prior to the Transformer. Patch embedding converts the refined feature map into ViT tokens with patch size 16, producing a 14×14 grid of 196 tokens with learnable absolute positional embeddings. The Transformer encoder then captures long-range dependencies over this token sequence (Lu et al., 19 Sep 2025).

The decoder uses three upsampling blocks to reconstruct the feature map back to 224×224. Each of the three skip connections from the encoder to the decoder is equipped with DA+ to suppress noise and reduce redundancy. After each upsampling stage, decoder features are fused with the corresponding DA+-refined skip features, and MEWB is applied again to refine the fused representation before the next stage. A final segmentation head produces the per-pixel multi-class prediction (Lu et al., 19 Sep 2025).

This organization makes the feature flow explicit:

Input (224×224) → Conv block 1 → Conv block 2 → Conv block 3.
Apply MEWB → DA+ → Patch embedding → Transformer blocks.
Decoder: upsample stage $k$ , fuse transformer-decoder features with skip features from encoder stage $k$ after DA+ → apply MEWB → proceed to next upsample.
Final segmentation head.

The significance of this arrangement lies in how it divides labor across modules: spatial convolutions preserve local texture, the Transformer supplies global context, MEWB injects frequency-domain structure, and DA+ regulates cross-scale fusion (Lu et al., 19 Sep 2025).

3. Multi-axis External Weight Block

MEWB is the principal mechanism through which FMD-TransUNet performs frequency-domain representation learning. It contains four branches: three frequency branches and one spatial branch. The frequency branches apply 2D discrete Fourier transforms along the axis pairs $\mathcal{F}(H,W)$ , $\mathcal{F}(C,W)$ , and $\mathcal{F}(C,H)$ , while the spatial branch applies depthwise separable convolution to preserve local detail (Lu et al., 19 Sep 2025).

Let $X \in \mathbb{R}^{C \times H \times W}$ denote the input feature map after GroupNorm. For each axis pair $A \in \{(H,W),(C,W),(C,H)\}$ , MEWB computes a 2D DFT, multiplies the resulting spectrum by learnable external weights, and returns to the spatial domain by inverse transform. For the height–width branch, the transformed and reweighted signal is

$\tilde{X}^{(H,W)}_c(u,v) = W^{(H,W)}(u,v)\,\mathcal{F}_{(H,W)}(X)_c(u,v),$

followed by inverse transformation to produce $Y^{(H,W)}_c(x,y)$ . Analogous constructions are used for the $k$ 0 and $k$ 1 branches, and the spatial branch is

$k$ 2

The four outputs are concatenated along the channel dimension and passed through an FFN with residual connections: $k$ 3 GroupNorm stabilizes training, and residual connections are applied at two points within MEWB to preserve original information and stabilize optimization (Lu et al., 19 Sep 2025).

The external weights are learnable per-frequency multipliers applied directly in the frequency domain. Their role is to modulate components according to diagnostic importance, such as emphasizing edges and other high-frequency content or enhancing global low-frequency structures. The intuition given for the module is that low-frequency content across $k$ 4– $k$ 5 captures global anatomical structure, while the channel–spatial transforms provide directional edge and boundary cues not readily separable in the spatial domain alone (Lu et al., 19 Sep 2025).

The computational implications are also explicit. A 2D FFT over $k$ 6 per channel is $k$ 7 per channel. The $k$ 8 and $k$ 9 FFTs are $k$ 0 per height row and $k$ 1 per width column, respectively, and the external weights add $k$ 2 parameters per branch. Despite this cost, MEWB is applied only at limited points—post-encoder and after each decoder stage—and empirically improves boundary precision dramatically, as reflected by the reported Hausdorff distance reductions (Lu et al., 19 Sep 2025).

A common misunderstanding is to view MEWB as a substitute for spatial modeling. In the reported design, it is explicitly complementary: the DWConv branch is retained to preserve local textures, and the concatenation-plus-FFN stage lets the network learn the weighting of spatial and frequency-augmented features jointly (Lu et al., 19 Sep 2025).

4. Improved Dual Attention module

DA+ is the second core addition. It adopts dual attention—spatial attention through PAM and channel attention through CAM—but implements the module with depthwise separable convolutions to reduce parameters and FLOPs and improve fusion (Lu et al., 19 Sep 2025).

The paper contrasts standard and depthwise separable convolutions directly. A standard $k$ 3 convolution uses $k$ 4 parameters, whereas a depthwise separable convolution uses $k$ 5. For $k$ 6 and $k$ 7, the reduction ratio is approximately

$k$ 8

which is near an order-of-magnitude reduction for large $k$ 9 (Lu et al., 19 Sep 2025).

The implemented DA+ equations are

$\mathcal{F}(H,W)$ 0

$\mathcal{F}(H,W)$ 1

$\mathcal{F}(H,W)$ 2

PAM follows DANet-style interactions. For $\mathcal{F}(H,W)$ 3 reshaped to $\mathcal{F}(H,W)$ 4 with $\mathcal{F}(H,W)$ 5, the spatial affinity matrix is computed by

$\mathcal{F}(H,W)$ 6

$\mathcal{F}(H,W)$ 7

$\mathcal{F}(H,W)$ 8

CAM models inter-channel correlations by reshaping to $\mathcal{F}(H,W)$ 9 and forming

$\mathcal{F}(C,W)$ 0

$\mathcal{F}(C,W)$ 1

$\mathcal{F}(C,W)$ 2

The stated effect is that PAM improves long-range spatial dependencies for global context, while CAM selectively amplifies diagnostically relevant channels and suppresses redundancy (Lu et al., 19 Sep 2025).

Placement is integral to the method. DA+ appears once before the Transformer to pre-refine features and three additional times in the skip connections to narrow the semantic gap between high-resolution low-level features and coarse global decoder features. This is not a minor insertion: the reported ablations show cumulative gains when DA+ is used on all three skip connections (Lu et al., 19 Sep 2025).

5. Training protocol, dataset, and empirical performance

Training uses a weighted sum of Cross-Entropy and Dice losses with optimal weights $\mathcal{F}(C,W)$ 3 and $\mathcal{F}(C,W)$ 4: $\mathcal{F}(C,W)$ 5 The paper gives

$\mathcal{F}(C,W)$ 6

$\mathcal{F}(C,W)$ 7

Optimization uses SGD with learning rate 0.01, momentum 0.9, weight decay $\mathcal{F}(C,W)$ 8, 200 epochs, and batch size 6. The backbone is a pre-trained R50-ViT. Data preprocessing resizes images to 224×224, and augmentation includes random rotation and flipping, random noise, and random contrast adjustment. The implementation is in PyTorch and was run on a single NVIDIA GeForce RTX 4090 D with 24 GB (Lu et al., 19 Sep 2025).

Evaluation is performed on the Synapse dataset: 30 CT scans comprising 3779 axial slices, with eight organs— aorta, gallbladder, spleen, left kidney, right kidney, liver, pancreas, and stomach. The split is 2212 training slices and 1567 testing slices. Performance is reported using DSC and HD (Lu et al., 19 Sep 2025).

FMD-TransUNet achieves an average DSC of 81.32% and an average HD of 16.35 mm across the eight abdominal organs. Relative to the TransUNet baseline, the average DSC increases by 3.84%, from 77.48% to 81.32%, and the average HD decreases by 15.34 mm, from 31.69 mm to 16.35 mm (Lu et al., 19 Sep 2025).

Variant	DSC	HD
TransUNet baseline	77.48%	31.69 mm
Only MEWB	79.10%	18.45 mm
Only DA+	80.28%	25.14 mm
FMD-TransUNet	81.32%	16.35 mm

The per-organ DSC values reported for FMD-TransUNet are: pancreas 66.31%, left kidney 85.12%, right kidney 82.12%, aorta 88.76%, liver 94.19%, spleen 89.73%, stomach 79.13%, and gallbladder 65.23% (Lu et al., 19 Sep 2025). The pancreas is highlighted because its DSC improves by 10.45% over TransUNet and by 4.69% over DA-TransUNet. Boundary precision is also emphasized: average HD falls to 16.35 mm, compared with 31.69 mm for TransUNet and 23.48 mm for DA-TransUNet (Lu et al., 19 Sep 2025).

The paper states that FMD-TransUNet surpasses U-Net, U-Net++, Residual U-Net, AttnU-Net, TransUNet, TransNorm, Swin-Unet, DA-TransUNet, and MEW-UNet on average. At the same time, statistical significance tests are not reported, so the empirical comparisons are descriptive rather than supported by formal significance analysis (Lu et al., 19 Sep 2025).

6. Ablations, interpretation, and limitations

The ablation study separates the effects of frequency-domain modeling and dual attention. Using only MEWB yields DSC 79.10% and HD 18.45 mm, with strong boundary refinement and pronounced HD reductions, including a gallbladder HD of 27.59 mm. Using only DA+ yields DSC 80.28% and HD 25.14 mm, indicating stronger regional accuracy through attention-based enhancement. Using both modules produces the best overall result, DSC 81.32% and HD 16.35 mm, which the paper interprets as evidence that global-local frequency cues and dual attention are complementary rather than redundant (Lu et al., 19 Sep 2025).

The skip-connection ablation further specifies the role of DA+. With DA+ in 0, 1, 2, and 3 skip connections, the reported results are DSC 78.59%, 79.43%, 80.02%, and 81.32%, and HD 24.64 mm, 21.00 mm, 17.62 mm, and 16.35 mm, respectively. The reported conclusion is that progressive refinement across scales is beneficial (Lu et al., 19 Sep 2025).

Qualitative observations align with the quantitative results. MEWB yields crisper boundaries and reduces leakage into adjacent tissues, especially for small and irregular organs such as the gallbladder and kidneys. DA+ enhances homogeneous regions and improves separation where intensities overlap, such as for the liver and spleen. Combined, the model more closely matches ground truth with improved boundaries and regional consistency (Lu et al., 19 Sep 2025).

Several limitations are explicit. The model is optimized and evaluated on single-modality CT using Synapse, so generalization to other datasets and modalities remains to be tested. Exact Transformer hyperparameters and some block-level channels are inherited from TransUNet defaults and are not exhaustively enumerated. Training and inference speed, memory footprint, and parameter count are not explicitly reported (Lu et al., 19 Sep 2025).

The future directions identified in the report are multi-modal integration, such as combining CT with MR; domain adaptation; further exploration of frequency-band selection and axis-wise weighting strategies; and lightweight variants that balance MEWB frequency processing cost with deployment constraints (Lu et al., 19 Sep 2025). A plausible implication, in light of the task-specific findings reported for 3D TransUNet, is that future variants may continue to separate encoder-side global anatomical modeling from decoder-side fine-structure refinement while making the frequency-domain pathway more adaptive (Chen et al., 2023).