Deformable Semantic Fusion (DSF)

• Deformable Semantic Fusion (DSF) is a family of modules that align and fuse semantically rich data streams by learning continuous, content-adaptive spatial offsets.
• DSF enables precise integration in applications like 3D multimodal medical image fusion and semantic-guided video frame interpolation, achieving notable improvements in metrics such as PSNR, SSIM, LPIPS, and FID.
• The approach combines offset prediction, deformable sampling, and cross-stream fusion within multi-scale architectures, paving the way for future extensions despite current computational limitations.

Deformable Semantic Fusion (DSF) is a family of architectural modules designed to resolve misalignments and enhance the integration of semantically meaningful information across disparate data streams, particularly in scenarios where rigid spatial correspondence is unreliable due to local deformations, complex motion, or inter-modality differences. The DSF paradigm is realized in both 3D multimodal medical image fusion and high-fidelity real-time video frame interpolation, leveraging learned, data-dependent spatial offsets to align and fuse feature representations in a content-adaptive and semantically consistent manner. Two leading instantiations are the Deformable Cross Feature Blend (DCFB) module within DC2Fusion for MRI–PET volumetric fusion (Liu et al., 2023) and the DSF module in the semantic-guided frame interpolation framework SG-RIFE (Wong et al., 20 Dec 2025).

1. Architectural Principles and Conceptual Foundations

Deformable Semantic Fusion addresses the limitations of rigid or naive fusion mechanisms in deep architectures by learning continuous-valued offsets that specify content-aware sampling locations for semantic features. In DC2Fusion (Liu et al., 2023), the DSF concept appears as DCFB blocks within a symmetric U-shaped, dual-branch encoder–decoder network, where 3D MRI and PET volumes are processed in parallel and fused via deformable cross-attention at each downsampling scale. In SG-RIFE (Wong et al., 20 Dec 2025), DSF modules operate within the bottleneck of a flow-based frame interpolation pipeline, aligning semantic priors extracted from a vision transformer (DINOv3) with pixel-level context produced by RIFE’s IFNet via modulated deformable convolution (DCNv2).

In both contexts, DSF’s deformable sampling learns to compensate for misregistrations—whether due to imperfect medical image alignment, modality-induced distortions, or optical flow errors in dynamic scenes. This mechanism enables finer semantic consistency, prevents loss of critical structures, and corrects for spatial discontinuities that arise from conventional warping or concatenation-based methods.

2. Mathematical Formulation and Fusion Workflow

Across implementations, DSF operates by first estimating spatial offsets, then sampling and aligning feature maps, followed by content-aware fusion. A generalized abstraction of the process can be summarized as follows:

1. Offset and Modulation Prediction: Given a context feature tensor $F_{\mathrm{ctx}}$ and a semantic stream (e.g., $\hat{D}_{t\leftarrow i}$ in SG-RIFE or $F_{I_B}$ in DC2Fusion), both are projected into shared embedding spaces via $1 \times 1$ convolutions:

$$Q(p) = \phi_q(F_{\mathrm{ctx}}(p)), \quad K_i(p) = \phi_k(\hat{D}_{t\leftarrow i}(p))$$

The concatenated embeddings $[Q, K_i]$ are passed to a convolutional predictor $\Psi_{\mathrm{off}}$ for per-location offsets and, if present, modulation scalars (SG-RIFE):

$$\{\Delta p_k^{(i)}(p),\,\Delta m_k^{(i)}(p)\}_{k=1}^K = \Psi_{\mathrm{off}}([Q(p), K_i(p)])$$

2. Deformable Sampling: Semantic features are sampled from locations $p + p_k + \Delta p_k^{(i)}(p)$ with learned filter weights and modulations (as in DCNv2):

$$F_{\mathrm{aligned}}^{(i),g}(p) = \gamma \sum_{k=1}^{K} W_k^{(g)} \cdot V_i^{(g)}(p + p_k + \Delta p_k^{(i)}(p)) \cdot \Delta m_k^{(i)}(p)$$

In medical fusion, trilinear interpolation achieves sub-voxel precision when aligning 3D volumes.

3. Bi-directional or Cross-stream Fusion: Features aligned from both streams are fused—by simple summation, $1 \times 1$ convolution, or attention (windowed cross-attention in DC2Fusion):

$$F_{\mathrm{sem}}(p) = F_{\mathrm{aligned}}^{(0)}(p) + F_{\mathrm{aligned}}^{(1)}(p)$$

The result is residually injected into the main processing stream for texture or structure enhancement:

$$F_{\mathrm{fused}}(p) = F_{\mathrm{ctx}}(p) + F_{\mathrm{sem}}(p)$$

3. Application Domains

3.1. 3D Multimodal Medical Image Fusion

In "Three-Dimensional Medical Image Fusion with Deformable Cross-Attention" (Liu et al., 2023), DSF (as DCFB) integrates 3D MRI and PET scans by explicitly modeling inter-modality geometric misalignment and learning semantic correspondence during unsupervised fusion. Because rigid registration rarely yields perfect voxel-wise alignment, especially after region-of-interest cropping or when modalities capture distinct biological phenomena, deformable fusion is essential.

Each DCFB block receives paired modalities $(I_A, I_B)$ and outputs spatially reconciled feature maps at each stage of the U-Net encoder, preserving local details such as gyri thickness and PET activity foci. The design is fully 3D, addressing the contextual discontinuity in 2D slice-based approaches.

3.2. Semantic-guided Video Frame Interpolation

In "SG-RIFE: Semantic-Guided Real-Time Intermediate Flow Estimation with Diffusion-Competitive Perceptual Quality" (Wong et al., 20 Dec 2025), DSF addresses misalignment when static semantic priors from DINOv3 are injected into a flow-based frame interpolation network. Optical flow warping alone leaves high-level semantic features inconsistent with dynamic context, resulting in ghosting especially near occlusions or rapid motion.

DSF employs modulated deformable convolutions to learn residual spatial adjustments of these priors, aligning them with contextually relevant pixel features. The bi-directionally aligned guidance is then fused and injected at multiple scales, enabling the network to hallucinate coherent textures and maintain semantic consistency with the ground-truth target frame.

4. Comparative Analysis with Conventional Strategies

Conventional fusion pipelines typically rely on direct concatenation, rigid warping, or local cross-attention mechanisms that cannot adapt to content-dependent misalignments. In medical imaging, these approaches either overlook inter-modality disparities or introduce averaging artifacts. In frame interpolation, fixed warping of semantic maps leads to significant artifacts at motion/occlusion boundaries.

The table summarizes performance metrics from representative DSF and comparable methods:

Method	Domain	PSNR ↑ / SSIM ↑ (Image Fusion)	LPIPS ↓ / FID ↓ (VFI)
SwinFuse	MRI–PET Fusion (2D)	14.102 / 0.623	–
MATR	MRI–PET Fusion (2D)	15.997 / 0.658	–
DILRAN	MRI–PET Fusion (2D)	19.028 / 0.690	–
DC2Fusion	MRI–PET Fusion (3D)	20.714 / 0.718	–
RIFE	Frame Interp. (Baseline)	–	0.066 / 23.320
SG-RIFE (DSF)	Frame Interp. (Ours)	–	0.047 / 17.896

DC2Fusion outperforms all 2D fusion baselines across PSNR/SSIM, while SG-RIFE’s DSF yields a 28.8% improvement in LPIPS and a 23.3% reduction in FID compared to standard RIFE. In both domains, learned deformable alignment corrects for local spatial inconsistencies that traditional methods cannot address (Liu et al., 2023, Wong et al., 20 Dec 2025).

5. Implementation and Training Protocols

DSF integration imposes distinct computational and memory constraints:

• Medical Image Fusion (DC2Fusion) (Liu et al., 2023):

PyTorch implementation on TITAN RTX/A6000 GPUs, 3D patch embedding (no overlap) due to memory usage, with window size $2\times2\times2$ and channels doubled at each downsampling. Training employs random 3D augmentation, Adam optimizer, and multi-objective fusion loss incorporating SSIM, NCC, $L_1$, and pairwise SSIM-balancing.

• Video Frame Interpolation (SG-RIFE) (Wong et al., 20 Dec 2025):

DSF operates at scales $S_2$/$S_3$ within FusionNet, with DINOv3/flow backbones frozen. Two-stage fine-tuning: DSF/Split-FAPM are “warmed up” for 5 epochs, followed by 25 epochs with joint FusionNet optimization. Offset regularization ($L_1$) constrains spatial drift, and semantic consistency loss ensures DSF-aligned output semantics.

A plausible implication is that, despite hardware limitations enforcing small windows and patching, DSF modules provide significant gains provided the fusion bottleneck can propagate local semantic corrections effectively.

6. Limitations, Impact, and Future Directions

The primary limitation of DSF, as deployed in current frameworks, is effective only within a local spatial window; large-scale nonrigid deformations remain challenging, partly due to computational overhead and stability constraints. In medical imaging, GPU-memory restrictions necessitate reduced resolution and aggressive downsampling, modestly compromising structural detail relative to full-resolution models (Liu et al., 2023). In video interpolation, DSF’s improvement is contingent on the expressiveness of the injected semantic priors and the reliability of upstream flow estimation (Wong et al., 20 Dec 2025).

Potential future directions include:

• Extending DSF to volumetric data beyond medical imaging, such as multimodal geospatial or remote sensing fusion.
• Incorporating anatomical or scene priors (e.g., segmentation masks in medical scenarios, object proposals in video) to inform and regularize offset predictions.
• Adopting sparsified attention or model parallelism to alleviate memory bottlenecks and enable application to larger volumes or frame sizes.
• Applying DSF concepts to additional cross-modal (e.g., CT–MRI, PET–CT) or temporal fusion tasks, and exploring hierarchical offset learning for global spatial consistency.

7. Broader Significance and Cross-domain Transferability

Deformable Semantic Fusion represents a principled mechanism for reconciling the rigid, grid-based inductive biases of convolutional or attention-based networks with the nonrigid, context-dependent nature of real-world correspondence. The concept’s provenance in both biomedical imaging (Liu et al., 2023) and real-time video synthesis (Wong et al., 20 Dec 2025) suggests broad applicability for any task in which semantically meaningful but spatially misaligned features must be jointly reasoned about at inference time. The reliance on learned local offsets—rather than fixed warping or pooling—enables finer-grained integration of disparate modalities or temporal contexts, consolidating DSF as a critical advancement in feature fusion architectures.