Semantics-Detail Dual-Branch Encoder

• Semantics-detail dual-branch encoder is a neural architecture that separates global (semantic) and local (detail) processing to capture comprehensive contextual and fine-grained features.
• It employs distinct branches—using Fourier transforms, Transformers, and deep convolutions—followed by adaptive fusion strategies such as cross-attention and channel alignment.
• Empirical results show improved outcomes in applications like low-light enhancement, segmentation, and super-resolution, with measurable gains in metrics such as PSNR and mIoU.

A semantics-detail dual-branch encoder—also described in the literature as a “dual-branch encoder,” “semantics–detail dual-branch encoder,” or “global-local dual-branch encoder”—is a neural architectural paradigm in which two parallel pathways are constructed to separately model global (semantic, low-frequency, or contextual) and local (detail, high-frequency, or structural) information within an input signal. These branches remain structurally and parametrically distinct, with a subsequent fusion stage to recombine their representations for downstream tasks. This approach drives advancements across a range of vision and multimodal tasks, including low-light enhancement, segmentation, retrieval, super-resolution, face restoration, image fusion, survival prediction, compression, and anomaly detection.

1. Structural Principles and Architectural Abstractions

At the core of semantics-detail dual-branch encoders is the deliberate division of representational responsibilities:

• Semantics branch: Typically targets global context, low-frequency or structural properties, and broader semantics. Architectural realizations span Fourier-domain transforms (Zhuang et al., 2022), Transformers/Swin blocks for long-range context (Wei et al., 18 Apr 2025, Xu et al., 1 Dec 2025), large-kernel convolutions in BEV projections (Kim et al., 2024), and global-modality attention (Yang et al., 23 May 2025, Jiang et al., 2023).
• Detail branch: Models fine-grained structure, texture, high-frequency, or local/categorical features. This branch employs deep convolutions with small kernels (Zhuang et al., 2022, Zhu et al., 2024), multi-scale dilated convolutions (Zhuang et al., 2022), Invertible Neural Networks (INN) (Xu et al., 2024), shortest-path/topological aggregators (Shou et al., 2024), or attribute-centric latent codes (Robert et al., 2019).

Fusion is typically performed at the feature level (e.g., adaptive weighting, concatenation, cross-attention) or via hybrid modules designed to preserve both branches’ complementarity (Wei et al., 18 Apr 2025, Yang et al., 23 May 2025, Xu et al., 1 Dec 2025).

Table: Representative Designs

Domain	Semantic (Global) Branch	Detail (Local) Branch
Image Enhancement	Phase-aware Fourier Conv (Zhuang et al., 2022)	Dilated CNN + multi-scale (Zhuang et al., 2022)
Segmentation	Swin Transformer (Wei et al., 18 Apr 2025)	Depthwise-sep. CNN (Wei et al., 18 Apr 2025)
Super-Resolution	RWKV global path (Zhu et al., 2024)	Conv RDEG (Zhu et al., 2024)
Retrieval	Transformer GM (Yang et al., 23 May 2025)	Q-Former DI (Yang et al., 23 May 2025)
Multimodal Fusion	Restormer (Xu et al., 2024)	INN (Xu et al., 2024)

2. Mathematical Foundations and Branch-specific Operations

Typical dual-branch encoders instantiate mathematically distinct operations in each branch. For example, (Zhuang et al., 2022) applies a phase-aware Fourier convolution:

• Frequency/semantics branch:

$$F(u,v) = \sum_{x,y} I_{low}(x,y) e^{-j2\pi(\frac{ux}{M}+\frac{vy}{N})}$$

followed by learned phase and amplitude convolutions.

• Detail branch:

$(f *_d k)(x) = \sum_{t} f(x - d\,t) k(t)$

with multiple dilation rates for multi-scale edge aggregation.

Other instantiations utilize channel-mixing Transformer feedforward blocks (e.g., KAT in (Xu et al., 1 Dec 2025)), domain-adapted additive invertible blocks (Xu et al., 2024), shortest-path topological aggregators (Shou et al., 2024), and token-wise dynamic modulation (Zhang et al., 1 Jan 2026). Global branches use large receptive fields, low-frequency focus, or advanced attention, while detail branches are confined to local neighborhoods, often operating with restricted kernels or explicit spatial constraints.

3. Fusion and Interaction Strategies

Reintegration of semantic and detail features is decisive for overall representational expressivity. Fusion mechanisms are purpose-designed:

• Adaptive fusion modules: Pixel-wise softmax gating between branches (Zhuang et al., 2022, Zhu et al., 2024).
• Channel or feature alignment modules: Cross-attention (Yang et al., 23 May 2025), cross-branch channel correlation (Xu et al., 1 Dec 2025).
• Spatial enhancement: Geometrically adaptive fusion (SFE-GAF) with learned deformable grids (Xu et al., 1 Dec 2025), spatial affinity and global-context modeling (Zhang et al., 1 Jan 2026).
• Loss-weighted or conditional fusions: Detail branch as conditional prior for semantic branch latent (Fu et al., 2024).
• Domain-aligned fusions: Multi-kernel MMD loss in semantic branch, invertible reconstruction in detail branch for invariant and lossless information transfer (Xu et al., 2024).

Empirical ablations repeatedly confirm that adaptive and domain-aware fusion yields superior results over naive summation or concatenation, especially in multi-modal fusion and compositional reasoning (Yang et al., 23 May 2025, Xu et al., 2024).

4. Supervision, Losses, and Optimization

Dual-branch encoders are commonly trained under composite or “committee” losses that reflect their hybrid representational aim:

• Pixel/structure-level: $\mathcal{L}_{SSIM}$, $\mathcal{L}_{MSE}$ or $\mathcal{L}_1$ (Zhuang et al., 2022, Zhu et al., 2024, Xu et al., 2024).
• Global context/frequency: Fourier or wavelet-domain loss terms to enforce alignment in phase, amplitude, or subband structure (Zhuang et al., 2022, Zhu et al., 2024).
• Perceptual/semantic losses: VGG or Transformer-based $\mathcal{L}_{p}$ (Zhuang et al., 2022, Tsai et al., 2023), cross-entropy on classifier heads (Wei et al., 18 Apr 2025, Jiang et al., 2023).
• Domain adaptation or alignment: Multi-Kernel MMD, correlation penalties, InfoNCE contrastive objectives (Xu et al., 2024, Shou et al., 2024).
• Adversarial or disentanglement: Adversarially trained disentanglers (Robert et al., 2019), cross-branch patch-level association (Tsai et al., 2023).
• Retrieval/composition: Atomic detail contrastive loss (Yang et al., 23 May 2025).

This multi-headed supervision enables refined optimization of both semantic and detailed cues, and is critical for preventing branch collapse or over-dominance in multi-modal learning.

5. Application Domains and Empirical Impact

Semantics-detail dual-branch encoders have demonstrated marked empirical advantages across diverse domains:

• Low-light image enhancement (Zhuang et al., 2022): Dual-branch FFT + dilated CNN yields superior PSNR/SSIM, sharp textures, and improved structure over baselines.
• RGB-D semantic segmentation (Wei et al., 18 Apr 2025): Dual RGB branch + lightweight depth encoder achieve mIoU/SOTA gains with orders of magnitude lower FLOPs.
• Remote sensing super-resolution (Zhu et al., 2024): RWKV + deep CNN dual path recovers global context and subpixel structure, outperforming quadratic attention.
• Image retrieval (Yang et al., 23 May 2025): Composed query dual-fusion improves fine-grained detail-aware retrieval, especially in confusable or compositional datasets.
• Retinal vessel segmentation (Xu et al., 1 Dec 2025): CNN+Transformer/KAN dual path, CCI, and geometric fusion achieve leading performance on vessel-specific segmentation.
• Face restoration (Tsai et al., 2023): Dual-branch association achieves SOTA FID/LPIPS via codebook-aligned semantic and LQ detail encoding.
• Infrared-visible fusion (Xu et al., 2024): Ensures modal alignment at the semantic level while preserving lossless texture via INN, outperforming alternatives.
• Graph-based survival prediction (Shou et al., 2024): GCN and shortest-path branches capture semantic and fine topological features for robust domain adaptation.

Model performance improvements frequently manifest as increases in both quantitative metrics (mIoU, PSNR, Recall@K) and qualitative fidelity (texture, object boundary, anomaly localization).

6. Design Considerations, Limitations, and Variants

Critical design decisions in semantics–detail dual-branch encoders include:

• Branch symmetry/asymmetry: Some domains (e.g., RGB vs. Depth (Wei et al., 18 Apr 2025), 3D voxel vs. BEV (Kim et al., 2024)) require heterogeneous branch complexity to match input signal structure.
• Fusion depth: Early/late fusion, single or repeated cross-branch updates, and iterative refinement.
• Orthogonality and disentanglement: Explicit adversarial objectives (Robert et al., 2019) or association training (Tsai et al., 2023) help maintain branch independence.
• Modality adaptation: Domain-specific detail branches (e.g., INN for texture, shortest-path for topology, conditional entropy models for redundancies (Fu et al., 2024)).
• Computation-accuracy trade-off: Dual-branch structures can yield efficiency (e.g., LDFormer (Wei et al., 18 Apr 2025), BEV large kernels (Kim et al., 2024)), but unbalanced fusion or excessive redundancy can degrade scalability.

A plausible implication is that, while dual-branch paradigms are highly flexible, their fusion and supervision schema must be tightly designed to avoid performance collapse of either branch or adverse redundancy.

7. Extensions and Future Directions

Several trends suggest the expansion and refinement of semantics–detail dual-branch encoders:

• Non-vision modalities: Extension to graph, NLP, sound, and cross-modal tasks (Shou et al., 2024, Jiang et al., 2023).
• Learned frequency or spectral splits: Soft-gate/adaptive frequency cutoffs and data-driven mask learning (Zhang et al., 1 Jan 2026).
• Dynamic fusion and query-based interaction: Cross-attentional and token-adaptive blending (Yang et al., 23 May 2025).
• Deeper disentanglement: Adversarial and linearization-driven attribute separation (Robert et al., 2019).
• Continual/online adaptation: Robustness to domain shift via feature/category-level alignment (Shou et al., 2024, Xu et al., 2024).
• Scalability and hardware deployment: Leveraging parallelism and model size reduction with architectural heterogeneity (Wei et al., 18 Apr 2025, Zhu et al., 2024).

This suggests that semantics–detail dual-branch architectures constitute not a static model family but a design principle adaptable to future advances in signal processing, multimodal fusion, and domain adaptation.

---

References

• DPFNet: A Dual-branch Dilated Network with Phase-aware Fourier Convolution for Low-light Image Enhancement (Zhuang et al., 2022)
• HDBFormer: Efficient RGB-D Semantic Segmentation with A Heterogeneous Dual-Branch Framework (Wei et al., 18 Apr 2025)
• DetailFusion: A Dual-branch Framework with Detail Enhancement for Composed Image Retrieval (Yang et al., 23 May 2025)
• DB-KAUNet: An Adaptive Dual Branch Kolmogorov-Arnold UNet for Retinal Vessel Segmentation (Xu et al., 1 Dec 2025)
• GDSR: Global-Detail Integration through Dual-Branch Network with Wavelet Losses for Remote Sensing Image Super-Resolution (Zhu et al., 2024)
• ProtoOcc: Accurate, Efficient 3D Occupancy Prediction Using Dual Branch Encoder-Prototype Query Decoder (Kim et al., 2024)
• Dual Associated Encoder for Face Restoration (Tsai et al., 2023)
• DAF-Net: A Dual-Branch Feature Decomposition Fusion Network with Domain Adaptive for Infrared and Visible Image Fusion (Xu et al., 2024)
• Graph Domain Adaptation with Dual-branch Encoder and Two-level Alignment for Whole Slide Image-based Survival Prediction (Shou et al., 2024)
• DualDis: Dual-Branch Disentangling with Adversarial Learning (Robert et al., 2019)
• Learned Image Compression with Dual-Branch Encoder and Conditional Information Coding (Fu et al., 2024)
• HarmoniAD: Harmonizing Local Structures and Global Semantics for Anomaly Detection (Zhang et al., 1 Jan 2026)
• A semantically enhanced dual encoder for aspect sentiment triplet extraction (Jiang et al., 2023)