Semantic-Structure Fusion Module (SSFM)

• SSFM is a neural network component that fuses complementary structural and semantic features from different modalities using explicit alignment and attention mechanisms.
• It employs specialized blocks like Adaptive Compression–Expansion (ACE) and Structure-Aware Multi-Context (SAMC) to align and merge heterogeneous feature maps efficiently.
• Empirical studies in ultrasound imaging, 3D detection, and image fusion demonstrate that SSFM enhances accuracy, mAP, and segmentation performance through fine detail preservation.

A Semantic-Structure Fusion Module (SSFM) is a neural network component designed to integrate complementary semantic and structural information from one or more modalities or network layers. SSFMs are architected to align feature maps differing in spatial resolution and channel dimension, preserving low-level structural cues and high-level semantic representations while enhancing their fusion through explicit mechanisms such as attention, multi-scale processing, or learned cross-modal aggregation. The core goal is to generate fused feature representations that are highly sensitive to fine-grained structural details and rich semantic content, thereby facilitating improved performance in downstream discriminative or generative tasks.

1. Motivations and Design Objectives

In domains such as ultrasound plane recognition, 3D object detection, and multi-modal image fusion, a persistent challenge is that superficial network layers encode strong structural patterns but weak semantic context, while deeper layers encode semantics yet lose spatial precision. SSFMs are developed to:

• Exploit the complementary strengths of shallow (structural) and deep (semantic) representations, especially in applications where boundaries are ambiguous or signal contrast is low.
• Efficiently align and merge heterogeneous feature maps—typically of differing resolutions and channels—into a unified, task-optimal representation.
• Preserve contextually relevant anatomical, geometric, or object boundary information, which standard fusion or augmentation-based contrastive methods may attenuate (Cai et al., 16 Nov 2025, Gao et al., 2023, Yang et al., 2023).

A plausible implication is that SSFMs enable better localization and classification in settings where either semantic abstraction or structural fidelity alone would be inadequate.

2. Canonical Architectures and Fusion Mechanisms

Ultrasound SSFM: Adaptive Compression–Expansion and Structure-Aware Multi-Context

In SEMC for ultrasound standard plane recognition, each SSFM comprises two main blocks per expert branch (Cai et al., 16 Nov 2025):

• Adaptive Compression–Expansion (ACE) Block: Aligns spatial resolution and channel counts between shallow features $F_i$ (from early ResNet layers) and deep features $D_i$ (from expert-specific deep layers) using:
  • Multi-stage strided depthwise convolution (downsampling) and pointwise convolution (channel adjustment), typically doubling channels per stage.
  • Final element-wise addition of projected shallow and deep features—ensuring structural and semantic fusion without redundant concatenation.
• Structure-Aware Multi-Context (SAMC) Block: Refines merged features via:
  • Channel Attention: Weighted via global pooling, FC layers, and sigmoid gating.
  • Spatial Attention: Computed from channel-weighted feature maps by mean/max aggregation and large-kernel convolution.
  • Multi-Scale Context Fusion: Parallel convolutions (e.g., kernels $1$, $3$, $5$), concatenation, channel shuffle, and linear projection.

Output: Three fused feature maps ($O_1,O_2,O_3$) per expert, feeding both contrastive and classification branches.

Transformer-Based SSFM for Multimodal Fusion

In 3D object detection pipelines with camera-LiDAR data, the SSFM operates within a Transformer backbone (Gao et al., 2023):

• Structure Branch (Sparse Fusion): For each LiDAR voxel, cross-attends to camera features projected to corresponding image regions via deformable attention, generating structure-preserving updates to voxel embeddings.
• Semantic Branch (Dense Fusion): BEV (bird’s eye view) queries cross-attend to temporal, LiDAR, and multi-view camera feature spaces over several Transformer layers, accumulating semantic context.
• Fusion Head: Combines structural ($F_{\rm struct}$) and semantic ($F_{\rm sem}$) representations, typically by weighted channel-wise summation or concatenation with a linear projection to produce unified BEV features.

Edge-Guided and Mask-Based SSFM Variants

In semantic structure-preserving fusion for infrared and visible image fusion, the multi-scale SPF module acts as a specialized SSFM (Yang et al., 2023):

• Structural features and binary edge/structure maps are extracted from each modality at every scale.
• "Unique" edge masks identify modality-specific contours; these masks gate per-branch feature enhancement.
• Fusion is accomplished by explicit mask-based aggregation (add-and-mask), guided by Sobel- or threshold-derived structure maps, rather than solely by learned attention weights.

3. Formal Mathematical Formulation

Overview Table

Subsystem	Core Operation	Key Formulae/Steps
ACE Block	Align, Compress	$X_{i+1}' = \mathrm{DWConv}^{s=2}_{3\times 3}(X_i)$<br>$$X_{i+1} = \mathrm{BN}(\mathrm{Conv}_{1\times1}(\mathrm{ReLU}(\mathrm{BN}(X_{i+1}'))))$$
ACE Fusion	Elem.-wise Add	$M_i = F'_i + D_i$
SAMC Block	Attention/Fusion	$$C_i = \sigma(W_2\delta(W_1\operatorname{AvgPool}(M_i)) + W_2\delta(W_1\operatorname{MaxPool}(M_i)))$$<br>$$S_i = \sigma(\mathrm{Conv}_{k\times k}([\text{Mean}(C_i\odot M_i),\text{Max}(C_i\odot M_i)]))$$<br>$$O_i = \mathrm{Conv}_{1\times1}(\mathrm{Shuffle}(\mathrm{Concat}\{F_k^{(i)}\}_{k=1}^K))$$
Transformer SSFM	Cross-modal Attention	$$F_{\text{struct}} = \mathrm{DeformableAttn}(Q, K, V)$$<br>$$F_{\text{out}} = \alpha \odot F_{\text{struct}} + (1-\alpha)\odot F_{\text{sem}}$$

SSFMs consistently implement fused feature computation as a series of operations that: (i) align feature shapes, (ii) aggregate using either learned attention or explicit edge guidance, and (iii) project to modality/task-agnostic spaces for downstream processing.

4. Training Objectives and Loss Integration

SSFMs are typically trained end-to-end as blocks within larger architectures, with no dedicated module-specific losses. In SEMC (Cai et al., 16 Nov 2025), the overall framework is optimized under a combination of:

• Mixture-of-Experts contrastive loss $(\mathcal{L}_{mc})$—encourages discriminative representation by leveraging multi-expert features.
• Classification loss $(\mathcal{L}_{moe})$—supervises label prediction performance.

For image fusion, explicit self-supervision is imposed on structure maps alongside perceptual and gradient-based losses to enforce edge and structure preservation (Yang et al., 2023). In 3D object detection, objectives generally include focal classification loss, bounding-box regression loss, and IoU loss, with no auxiliary losses inside the SSFM block (Gao et al., 2023).

5. Empirical Impact and Ablation Evidence

Quantitative ablation studies consistently verify that both semantic and structural branches of SSFMs contribute additively and complementarily:

• In ultrasound plane recognition, ACE alone improves accuracy by 1.12 points; adding SAMC increases the margin to +1.25 accuracy and +0.93 F1-score compared to the baseline. The full SSFM with Mixture-of-Experts contrastive learning yields an accuracy of 82.30 and F1-score of 79.32 (Cai et al., 16 Nov 2025).
• In 3D detection, adding the structure branch to a LiDAR-only baseline increases mAP from 65.5 to 67.9. Adding the semantic branch results in 67.2 mAP. Using both in SSFM yields 68.8 mAP and 72.0 NDS; further temporal fusion pushes these to 69.2 and 72.4, respectively. Structural fusion boosts performance on small/movable categories (e.g., pedestrians +3.5 mAP), while semantic fusion benefits large/static classes (cars +2.8 mAP) (Gao et al., 2023).
• For image fusion, ablations on the MSRS dataset demonstrate that SPF (an SSFM instance) combined with SFE yields the highest MI and VIF metrics versus its ablated forms. Downstream, fused images generated by SSFM-equipped models achieve the top mIoU score in semantic segmentation (Yang et al., 2023).

6. Comparative Analysis and Context

SSFMs can be viewed as a unifying framework for multi-scale, multi-modal, or multi-level feature fusion, generalizing classical skip connections and learned attention mechanisms by explicitly structuring the fusion of structural and semantic cues. Compared to concatenation or single-branch methods, SSFMs avoid redundancy, improve efficiency, and demonstrably enhance fine-detail preservation.

In single-modal applications (e.g., ultrasound), SSFMs prioritize layer-wise (shallow-to-deep) alignment and attention. For multimodal fusion (e.g., camera-LiDAR or infrared-visible), SSFMs orchestrate cross-modal context aggregation via attention or mask-based selection, with explicit constraints to preserve modality-unique edges, geometry, or semantics.

A plausible implication is that, as tasks grow in complexity and require precise discrimination between subtle classes or fine structural details, SSFMs provide a scalable and generalizable mechanism to maintain high performance without sacrificing efficiency or relying solely on brute-force data augmentation.

7. Related Modules and Future Directions

Related approaches include Structure-Preserving Fusion (SPF) modules for semantic edge consistency (Yang et al., 2023), Sparse-Dense Fusion (SDF) modules for transformer-based multi-modal feature merging (Gao et al., 2023), and Mixture-of-Experts (MoE) architectures for representation diversification (Cai et al., 16 Nov 2025). Contemporary trends indicate interest in:

• Dynamic fusion schedules or learnable weighting between semantic and structural pathways.
• Incorporation of semantic segmentation maps or higher-order region descriptors into SSFMs for explicit semantic guidance.
• Greater integration of SSFM variants within transformer backbones and multi-stage fusion frameworks.

This suggests that future research may focus on self-supervised or weakly supervised objectives tailored for semantic-structure consistency, and on the architectural search for further parameter and compute efficiency as SSFM-like modules are embedded in large-scale recognition and generation pipelines.