Local-Global Fusion Adapter in U-DFA

• Local-Global Fusion Adapter (LGFA) is a core component in the U-DFA pipeline that bridges frozen DINOv2 transformers and CNN-based spatial feature extraction.
• It employs a multi-stage dual fusion attention mechanism to iteratively inject precise CNN features into the transformer stream and return enriched global context for robust segmentation.
• Ablation studies show that a 3-module LGFA configuration yields optimal performance, balancing segmentation accuracy with efficient use of trainable parameters.

Searching arXiv for the primary paper and closely related local–global fusion works to ground the article. Local-Global Fusion Adapter (LGFA) is the key design element in U-DFA, a unified DINOv2-Unet encoder-decoder architecture for multi-dataset medical image segmentation. In that setting, LGFA is introduced to address a central limitation of hybrid CNN–Transformer segmentation models: local and global features are often extracted jointly but are not fused in a way that is sufficiently deep, stage-wise, and spatially aware for medical image segmentation. LGFA acts as a bridge between a frozen pretrained DINOv2 transformer and a trainable CNN-based Spatial Pattern Adapter (SPA), injecting spatially precise low-level CNN features into the transformer stream while also feeding transformer-refined global context back into the CNN pathway; the authors designate this repeated bidirectional fusion as dual fusion attention (Sajjad et al., 1 Oct 2025).

1. Architectural setting inside U-DFA

Within U-DFA, LGFA is an encoder-side fusion mechanism embedded in a DINOv2-Unet encoder-decoder pipeline. The overall architecture is organized into three major parts—encoder, bottleneck, and decoder—but the encoder is the principal locus of LGFA. It begins with a head module and then proceeds through $N$ intermediate stages. The input image is

$I \in \mathbb{R}^{H \times W \times C}, \quad C=3,$

and is processed in parallel by the DINOv2 embedding layer and the SPA module, creating a dual-path head in which one branch tokenizes the image for DINOv2 and the other extracts multi-scale CNN feature maps for spatial detail (Sajjad et al., 1 Oct 2025).

The pretrained DINOv2 backbone contains $L$ transformer blocks grouped evenly into $N$ stages, so each stage contains $L/N$ frozen DINOv2 blocks and one trainable LGFA module. LGFA is therefore inserted between groups of frozen transformer blocks rather than only at the encoder entrance or exit. This multi-stage insertion enables repeated local-global interaction across depth. The decoder is a cascade upsampling decoder that uses the encoded features after the final stage together with skip connections from the SPA module at different resolutions. In consequence, SPA supports both LGFA fusion inside the encoder and skip-based localization in the decoder, while LGFA strengthens the encoded representation that reaches the decoder (Sajjad et al., 1 Oct 2025).

2. Local and global feature streams

The two streams coupled by LGFA are structurally distinct. The SPA branch is a ResNet-inspired CNN branch that uses three Conv-BN-ReLU blocks for low-level feature extraction, followed by three convolutional blocks that produce multi-scale feature maps at resolutions

$1/r_1,\; 1/r_2,\; 1/r_3$

relative to the input image. These are projected into a shared embedding dimension $D$ and concatenated into

$$f^{1}_{spa} \in \mathbb{R}^{\left( \frac{HW}{r_1^2} + \frac{HW}{r_2^2} + \frac{HW}{r_3^2} \right)\times D}.$$

This construction encodes multi-resolution spatial information and supplies the local stream used throughout the encoder (Sajjad et al., 1 Oct 2025).

The DINOv2 branch produces token embeddings from patchified image input,

$$I_p \in \mathbb{R}^{K \times (P^2 \cdot C)}, \quad K=\frac{HW}{P^2},$$

which are projected into $D$-dimensional token embeddings and augmented with positional embeddings. The DINOv2 backbone remains frozen during training. Adaptation therefore occurs through LGFA and SPA rather than through full backbone fine-tuning. Inside each stage, SPA interacts with DINOv2 twice—before the transformer blocks in the stage and after the transformer blocks in the stage. LGFA thus wraps around the frozen DINOv2 blocks: it first injects CNN-derived locality into the transformer stream and later collects the transformer output back into the SPA pathway (Sajjad et al., 1 Oct 2025).

3. Dual fusion attention and bidirectional exchange

LGFA performs bidirectional cross-attention-based fusion between local spatial features from SPA and global semantic or token features from DINOv2. At stage $I \in \mathbb{R}^{H \times W \times C}, \quad C=3,$0, the first interaction updates the DINOv2 feature using SPA as key and value: $I \in \mathbb{R}^{H \times W \times C}, \quad C=3,$1 Here, the query is the normalized DINOv2 token sequence $I \in \mathbb{R}^{H \times W \times C}, \quad C=3,$2, while the key and value are the normalized SPA features $I \in \mathbb{R}^{H \times W \times C}, \quad C=3,$3. The result is added in residual fashion to the original transformer representation, and the updated feature $I \in \mathbb{R}^{H \times W \times C}, \quad C=3,$4 is then processed by the stage’s DINOv2 blocks to produce $I \in \mathbb{R}^{H \times W \times C}, \quad C=3,$5 (Sajjad et al., 1 Oct 2025).

The second interaction reverses direction after transformer processing: $I \in \mathbb{R}^{H \times W \times C}, \quad C=3,$6 In this case, SPA features form the query, and the updated DINOv2 output serves as key and value. This returns global semantic context to the CNN branch and refines the spatial stream with long-range structure. The fused content is described explicitly as combining CNN local spatial patterns—edges, textures, boundaries, and multi-scale details—with transformer global semantics—long-range dependencies, object-level context, and shape priors. In medical segmentation, this pairing is particularly relevant because anatomical consistency and accurate boundary delineation must be preserved simultaneously (Sajjad et al., 1 Oct 2025).

The paper does not expand MHCA into a full attention-matrix derivation, but it specifies that MHCA is multi-head cross attention used in residual fashion. The resulting dual-path, multi-stage exchange is more than simple concatenation or addition: it is a staged adapter mechanism that lets each stream condition the update of the other. This suggests a conception of LGFA as a structured local-global mediator rather than a terminal fusion block.

4. Bottleneck, decoder path, and training objective

After the encoder, the feature $I \in \mathbb{R}^{H \times W \times C}, \quad C=3,$7 is reshaped from patch or token form to spatial resolution $I \in \mathbb{R}^{H \times W \times C}, \quad C=3,$8. A $I \in \mathbb{R}^{H \times W \times C}, \quad C=3,$9 convolution then reduces channels to the target number of classes. The decoder upsamples via bilinear interpolation, and each decoder stage uses DoubleConv blocks comprising two $L$0 convolution layers, each followed by BatchNorm and ReLU. Skip connections from SPA are added to preserve localization. Although LGFA is primarily encoder-side, its effect propagates through the bottleneck and decoder because the decoder consumes the encoder output that LGFA repeatedly refines (Sajjad et al., 1 Oct 2025).

The training objective is the sum of Dice loss and cross-entropy loss with equal weight,

$L$1

The DINOv2 backbone remains frozen, and optimization uses Adam with weight decay $L$2. In this formulation, LGFA functions as the principal trainable adaptation route between a fixed foundation-model backbone and a task-specific segmentation decoder (Sajjad et al., 1 Oct 2025).

5. Experimental evidence and ablation behavior

The ablation study directly evaluates the number of LGFA modules on Synapse at input size $L$3. With 2 LGFA modules, the model attains DSC $L$4 and HD $L$5; with 3 LGFA modules, DSC $L$6 and HD $L$7; with 6 LGFA modules, DSC $L$8 and HD $L$9. The paper’s interpretation is explicit: 2 modules correspond to under-utilization, 6 modules indicate potential overfitting or inefficiency, and 3 modules provide the best trade-off. In particular, the 3-LGFA configuration yields the best HD, which the authors associate with improved boundary delineation in segmentation (Sajjad et al., 1 Oct 2025).

On overall benchmarks, U-DFA achieves state-of-the-art performance on Synapse and ACDC while using only 33\% of the trainable model parameters because DINOv2 is frozen. On Synapse, the reported result is 82.25 DSC and 15.27 HD, surpassing RotU-Net at 82.15 DSC, MISSFormer at 81.96 DSC, and DSGA-Net at 81.24 DSC. On ACDC, U-DFA reaches 90.46 DSC. The paper states that these results demonstrate a robust and scalable framework for medical image segmentation across multiple modalities. A plausible implication is that the LGFA design is most consequential not merely because it combines CNN and transformer features, but because it performs that combination repeatedly, bidirectionally, and at multiple encoder depths (Sajjad et al., 1 Oct 2025).

6. Relation to surrounding literature and terminological ambiguity

The term “LGFA” is not used uniformly across the literature. In U-DFA, it denotes a Local-Global Fusion Adapter integrated into a frozen DINOv2-Unet encoder-decoder pipeline for medical image segmentation (Sajjad et al., 1 Oct 2025). In speech emotion recognition, however, “LGFA” refers to “Local to Global Feature Aggregation,” a nested Transformer design in which a Frame Transformer learns local inter-frame correlations and a Segment Transformer aggregates frame-informed segment representations into a global utterance-level emotion descriptor (Lu et al., 2023). The shared acronym therefore does not imply a single standardized architecture.

Several neighboring works exhibit LGFA-like design principles without using the same term. InterFormer for automatic speech recognition employs a parallel convolution–Transformer encoder with a bidirectional feature interaction module (BFIM) and a selective fusion module (SFM), emphasizing that local and global features should interact before fusion rather than be combined by direct addition or concatenation (Lai et al., 2023). In federated learning, LG-FedAvg decomposes the model into a client-side local encoder and a shared global predictor, creating a local-global representation pipeline that is adapter-like in spirit even though it is not a plug-in attention module inside a transformer (Liang et al., 2020). In face recognition, the Local and Global Feature Attention Fusion network uses feature norm as a proxy for feature quality and adaptively weights local and global embeddings by attention-weighted addition (Yu et al., 2024). In SAR-enhanced cloud removal, GLF-CR separates global fusion, which preserves scene-wide structure, from local fusion, which transfers complementary SAR detail with dynamic filtering for speckle suppression (Xu et al., 2022).

These related formulations indicate that “local-global fusion” names a broader design family rather than a single mechanism. What distinguishes the U-DFA LGFA within that family is its explicit placement between groups of frozen DINOv2 blocks, its two-stage multi-head cross-attention exchange at every encoder stage, and its coupling to a CNN-based SPA branch that also supplies decoder skip connections (Sajjad et al., 1 Oct 2025). A common misconception is to treat all local-global fusion modules as equivalent to simple concatenation, additive merging, or late skip fusion. The surrounding literature instead shows multiple distinct strategies—cross-attention, gating, hierarchical aggregation, feature-quality weighting, and client-side representation splitting—whose common aim is to couple locality and long-range context more effectively than shallow or one-way fusion.