Deformable Fusion: Adaptive Feature Integration

• Deformable Fusion is an adaptive design principle that uses dynamic offsets and deformable sampling to fuse heterogeneous evidence when rigid alignment fails.
• It incorporates mechanisms like deformable convolutions, cross-attention, gating, and warp-then-fuse strategies to enhance multi-modal data fusion.
• Applications span tracking, video restoration, medical imaging, and autonomous driving, with empirical studies showing improved alignment and robustness.

Searching arXiv for the cited papers to ground the article in current preprints. Deformable Fusion denotes a family of fusion strategies in which aggregation is made adaptive to spatial displacement, temporal motion, modality mismatch, or structural variability by means of learned offsets, deformable sampling, warping, cross-attention, or learned gates. In the literature, the term spans deformable detector fusion for multi-person tracking, deformable convolution plus gated feature fusion in tracking, feature-level alignment and attention in video restoration, query-space camera–LiDAR fusion, multimodal medical image fusion, automatic fusion for deformable registration, and adaptive state fusion in vision state space models (Ma et al., 2015, Liu et al., 2018, Wang et al., 2019, Meng et al., 2023, Liu et al., 2023, Ke et al., 20 May 2026). This suggests that deformable fusion is best understood not as a single operator, but as a design principle for fusing heterogeneous evidence when rigid alignment or fixed receptive fields are inadequate.

1. Historical development and scope

An early formulation appears in multi-person tracking, where multiple detectors for different regions of interest were fused by grouping detection outputs by bounding box location and depth information, while learning a deformable spatial relationship between detectors to handle significant pose variations (Ma et al., 2015). In tracking-by-detection, a later formulation introduced a deformable convolution layer to enrich target appearance representations and a gated fusion scheme to control how the variations captured by the deformable convolution affect the original appearance (Liu et al., 2018). In video restoration, EDVR combined feature-level deformable alignment with temporal and spatial attention fusion, explicitly addressing the joint problem of alignment under large motion and fusion under diverse motion and blur (Wang et al., 2019).

Recent work broadened the term substantially. In deformable medical image registration, AutoFuse reframed fusion as a learnable, data-driven process by exposing many potential fusion locations and learning them through Fusion Gate modules (Meng et al., 2023). In 3D multimodal medical image fusion, the Deformable Cross Feature Blend module used explicit 3D positional relationship estimation followed by windowed cross-attention to align and blend features from MRI and PET (Liu et al., 2023). In autonomous driving, FusionFormer sampled directly from 2D image and 3D voxel features without explicit transformation to bird’s-eye view during the feature concatenation process, while Li-ViP3D++ introduced Query-Gated Deformable Fusion in query space and Deformba introduced Context-Adaptive State Fusion to make state-space-model readout deformable and query-like (Hu et al., 2023, Halinkovic et al., 28 Jan 2026, Ke et al., 20 May 2026).

The scope also includes task couplings in which fusion is explicitly structured around an auxiliary branch. DAFF-Net, for example, is a multi-task learning framework based on multi-scale dual attention frequency fusion that simultaneously achieves the segmentation masks and dense deformation fields in a single-step estimation; its DAFF module fuses registration and segmentation features at different scales and incorporates both high-frequency and low-frequency information during local weighting (Zhou et al., 2024). Comparable shared-encoder and auxiliary-decoder patterns appear in unsupervised image registration with multi-scale feature fusion from shared encoder, auxiliary and pyramid decoders (Zhou et al., 2024).

2. Core operators and mathematical forms

The most common primitive is deformable convolution. In the standard formulation, at output location $p_0$ one replaces a rigid grid with learned offsets, so that

$y(p_0) = \sum_{k} w_k\, x(p_0 + p_k + \Delta p_k),$

and in the modulated form

$$y(p_0) = \sum_{k} m_k\, w_k\, x(p_0 + p_k + \Delta p_k).$$

This operator is used directly in tracking, EDVR, cardiac cine MRI reconstruction, CoMiX’s X branch, DFAR, and several registration systems (Liu et al., 2018, Wang et al., 2019, Han et al., 2023, Zhang et al., 2024, Luo et al., 2024).

A second major primitive is deformable attention. In MFDS-DETR, a standard multi-scale deformable attention form is

$$DA(q) = \sum_{l=1}^{L} W_l \sum_{k=1}^{K} A_{lk}(q)\cdot x^{l}(p_q + \Delta p_{lk}(q)),$$

with sparse learned sampling around normalized reference points; FusionFormer and MMF-BEV adopt closely related formulations in BEV-space fusion, while Li-ViP3D++ specializes the same principle to fully differentiable BEV sampling with learned per-query offsets (Chen et al., 2024, Hu et al., 2023, Mayank et al., 6 Apr 2026, Halinkovic et al., 28 Jan 2026). These formulations preserve sparse, geometry-aware sampling while avoiding dense all-to-all interaction.

A third pattern is warp-then-fuse. In “Three-Dimensional Medical Image Fusion with Deformable Cross-Attention,” positional relationship estimation predicts a dense 3D offset field, resamples one modality by trilinear interpolation, and only then performs cross-attention with queries from the other modality (Liu et al., 2023). In cardiac cine MRI reconstruction, Motion-Guided Deformable Alignment aligns adjacent frames by optical-flow-guided deformable sampling before Multi-Resolution Fusion corrects blur and artifacts generated from alignment operation (Han et al., 2023). EDVR and DFAR follow the same broad logic: alignment first, then attention-guided fusion (Wang et al., 2019, Luo et al., 2024).

A fourth pattern is gating. The fusion rule in deformable object tracking,

$Y = \sigma \odot X' + (1-\sigma)\odot X,$

makes deformable features and original features compete through a learned spatial gate (Liu et al., 2018). AutoFuse generalizes this idea across many candidate fusion locations with softmax competition between inter-image fused features and current fused-stream features (Meng et al., 2023). Li-ViP3D++ applies query-conditioned gating to adaptively weight visual and geometric cues per agent, while Deformba uses learned offsets and fusion weights in Context-Adaptive State Fusion after a single SSM write pass (Halinkovic et al., 28 Jan 2026, Ke et al., 20 May 2026).

A common misconception is that deformable fusion is synonymous with explicit offset sampling. The literature is more heterogeneous. 3D Brainformer’s Infinite Deformable Fusion Transformer Module realizes deformability by content-adaptive volumetric positional modulation of attention logits on 3D feature maps, and the paper explicitly states that explicit offset sampling is not employed in the reported implementation (Nian et al., 2023). AutoFuse, likewise, is deformable in the sense of adaptive fusion placement and weighting rather than geometric offset prediction (Meng et al., 2023).

3. Recurrent architectural motifs

Across domains, several architectural motifs recur. The first is coarse-to-fine alignment followed by fusion. EDVR uses Pyramid, Cascading and Deformable alignment and then Temporal and Spatial Attention fusion (Wang et al., 2019). DFAR uses Temporal Deformable Alignment followed by Attention-guided Deformable Fusion (Luo et al., 2024). Cardiac cine MRI reconstruction uses Motion-Guided Deformable Alignment followed by Multi-Resolution Fusion (Han et al., 2023). In all three cases, deformable alignment reduces residual motion before the fusion stage attempts aggregation.

The second motif is multi-scale fusion with an auxiliary or parallel branch. DAFF-Net consists of a global encoder, a segmentation decoder, and a coarse-to-fine pyramid registration decoder, with dual attention frequency feature fusion during registration decoding (Zhou et al., 2024). The shared-encoder registration network of (Zhou et al., 2024) uses a shared auxiliary decoder that provides multi-scale high-level feature information from unblended image pairs, then combines deformable alignment with channel and spatial attention inside a multi-scale feature fusion block (Zhou et al., 2024). AutoFuse extends the same intuition by placing Fusion Gate modules at many potential locations in encoder, decoder, and skip pathways, allowing the network to automatically optimize its fusion strategy during training (Meng et al., 2023).

The third motif is bidirectional cross-modal exchange. CoMiX uses CMFeX to recalibrate modality-specific and modality-shared features and to adaptively exchange complementary information before a Feature Fusion Module integrates them (Zhang et al., 2024). MMF-BEV uses Deformable Self-Attention within each modality and bidirectional Deformable Cross-Attention between camera and radar in a shared BEV grid (Mayank et al., 6 Apr 2026). “Three-Dimensional Medical Image Fusion with Deformable Cross-Attention” applies DCFB twice per scale in symmetric fashion, one pass for each modality as query stream (Liu et al., 2023). Li-ViP3D++ similarly aggregates image evidence and LiDAR context per query and then fuses them with query-conditioned gating (Halinkovic et al., 28 Jan 2026).

The fourth motif is token- or state-based multimodal fusion. UAVD-Mamba forms deformable tokens by adding adaptive patches from deformable convolutions to normal patches from normal convolutions, processes them with modality-specific Vision Mamba blocks, and then fuses modalities through a Fusion Mamba block whose state-space projections are conditioned by the other modality (Li et al., 1 Jul 2025). Deformba, in turn, writes a state memory through a single SSM scan and performs deformable, context-adaptive readout from the resulting 2D state map, allowing SSMs to support multi-modal fusion like cross attention while maintaining linear complexity (Ke et al., 20 May 2026).

Architectural motif	Representative mechanism	Representative papers
Alignment then fusion	PCD + TSA; TDA + AGDF; MGDA + MRF	(Wang et al., 2019, Luo et al., 2024, Han et al., 2023)
Multi-scale auxiliary fusion	DAFF module; MSFB; Fusion Gate	(Zhou et al., 2024, Zhou et al., 2024, Meng et al., 2023)
Bidirectional cross-modal exchange	CMFeX + FFM; DCFB; bidirectional DCA	(Zhang et al., 2024, Liu et al., 2023, Mayank et al., 6 Apr 2026)
Query/state-space fusion	QGDF; CASF; Fusion Mamba	(Halinkovic et al., 28 Jan 2026, Ke et al., 20 May 2026, Li et al., 1 Jul 2025)

4. Domain-specific realizations

In medical imaging, deformable fusion is tightly coupled to registration, reconstruction, and multimodal image synthesis. AutoFuse reports higher DSC and drastically lower NJD than several unsupervised and semi-supervised registration baselines across OASIS, Mindboggle, Buckner, and ACDC, while preserving diffeomorphic regularity through a stationary velocity field and Jacobian penalty (Meng et al., 2023). The shared-encoder registration model of (Zhou et al., 2024) combines deformable convolution inside its multi-scale feature fusion block with channel and spatial attention and reports Dice $0.7727 \pm 0.0276$, TRE $2.4663$, NDV $0.0130$, and HdDist95 $3.3197$ on the Learn2Reg LUMIR validation set (Zhou et al., 2024). In cine MRI reconstruction, MGDA + MRF achieves SSIM $89.40\% \pm 4.13\%$, PSNR $y(p_0) = \sum_{k} w_k\, x(p_0 + p_k + \Delta p_k),$0 dB, and NMSE $y(p_0) = \sum_{k} w_k\, x(p_0 + p_k + \Delta p_k),$1 on ACDC at $y(p_0) = \sum_{k} w_k\, x(p_0 + p_k + \Delta p_k),$2 acceleration, and the ablation results show complementary benefits of deformable alignment and multi-resolution fusion (Han et al., 2023).

In multimodal 3D medical image fusion, the DCFB mechanism estimates offsets, resamples one modality, and then performs cross-attention within 3D windows; on ADNI-2, DC2Fusion reports PSNR $y(p_0) = \sum_{k} w_k\, x(p_0 + p_k + \Delta p_k),$3 and SSIM $y(p_0) = \sum_{k} w_k\, x(p_0 + p_k + \Delta p_k),$4, improving over the 2D methods SwinFuse, MATR, and DILRAN in both PSNR and SSIM (Liu et al., 2023). In brain tumor segmentation, 3D Brainformer uses Fusion-Head Self-Attention in the encoder and the Infinite Deformable Fusion Transformer Module in the decoder, reaching Dice $y(p_0) = \sum_{k} w_k\, x(p_0 + p_k + \Delta p_k),$5 for WT/TC/ET and HD95 $y(p_0) = \sum_{k} w_k\, x(p_0 + p_k + \Delta p_k),$6 (Nian et al., 2023).

In remote sensing and multimodal semantic segmentation, CoMiX addresses spatial misalignment, geometric variation, and spatial–spectral coupling through 2D deformable convolutions in the X branch, EPA-based “3D DCN blocks” in the HSI branch, and CMFeX plus FFM for exchange and integration (Zhang et al., 2024). On Houston2013 it reports OA $y(p_0) = \sum_{k} w_k\, x(p_0 + p_k + \Delta p_k),$7, AA $y(p_0) = \sum_{k} w_k\, x(p_0 + p_k + \Delta p_k),$8, and $y(p_0) = \sum_{k} w_k\, x(p_0 + p_k + \Delta p_k),$9, and the ablation progression from baseline to final model explicitly quantifies the incremental benefit of 2D DCN blocks, 3D spatial–spectral aggregation, CMFeX, and FFM (Zhang et al., 2024).

In autonomous driving, deformable fusion is used to avoid rigid BEV projection bottlenecks and to preserve vertical or geometric structure. FusionFormer samples from multi-view image feature maps and 3D LiDAR voxel features directly, reaching $$y(p_0) = \sum_{k} m_k\, w_k\, x(p_0 + p_k + \Delta p_k).$$0 mAP and $$y(p_0) = \sum_{k} m_k\, w_k\, x(p_0 + p_k + \Delta p_k).$$1 NDS on nuScenes without test time augmentation (Hu et al., 2023). Li-ViP3D++ performs query-space fusion with masked image attention, deformable BEV sampling, and query-conditioned gating, reporting EPA $$y(p_0) = \sum_{k} m_k\, w_k\, x(p_0 + p_k + \Delta p_k).$$2, mAP $$y(p_0) = \sum_{k} m_k\, w_k\, x(p_0 + p_k + \Delta p_k).$$3, FP ratio $$y(p_0) = \sum_{k} m_k\, w_k\, x(p_0 + p_k + \Delta p_k).$$4, and runtime $$y(p_0) = \sum_{k} m_k\, w_k\, x(p_0 + p_k + \Delta p_k).$$5 ms (Halinkovic et al., 28 Jan 2026). MMF-BEV applies deformable self-attention in camera and radar branches and bidirectional deformable cross-attention in BEV, with DCA fusion reaching mAP $$y(p_0) = \sum_{k} m_k\, w_k\, x(p_0 + p_k + \Delta p_k).$$6 in the full annotated area and mAP $$y(p_0) = \sum_{k} m_k\, w_k\, x(p_0 + p_k + \Delta p_k).$$7 in the Driving Corridor ROI on VoD validation (Mayank et al., 6 Apr 2026). Deformba further shows that deformable fusion can be embedded into vision SSMs, with Deformba-B reaching $$y(p_0) = \sum_{k} m_k\, w_k\, x(p_0 + p_k + \Delta p_k).$$8 Top-1 on ImageNet-1K and Deformba-S reporting NDS $$y(p_0) = \sum_{k} m_k\, w_k\, x(p_0 + p_k + \Delta p_k).$$9 and mAP $$DA(q) = \sum_{l=1}^{L} W_l \sum_{k=1}^{K} A_{lk}(q)\cdot x^{l}(p_q + \Delta p_{lk}(q)),$$0 for BEV 3D detection on nuScenes with 4 frames (Ke et al., 20 May 2026).

In detection, tracking, and small-object recognition, the same principle appears at smaller scales. MFDS-DETR combines HS-FPN with multi-scale deformable self-attention and cross-deformable attention, reporting AP $$DA(q) = \sum_{l=1}^{L} W_l \sum_{k=1}^{K} A_{lk}(q)\cdot x^{l}(p_q + \Delta p_{lk}(q)),$$1, AP50 $$DA(q) = \sum_{l=1}^{L} W_l \sum_{k=1}^{K} A_{lk}(q)\cdot x^{l}(p_q + \Delta p_{lk}(q)),$$2, and AP75 $$DA(q) = \sum_{l=1}^{L} W_l \sum_{k=1}^{K} A_{lk}(q)\cdot x^{l}(p_q + \Delta p_{lk}(q)),$$3 on WBCDD (Chen et al., 2024). DFAR uses TDA and AGDF for moving infrared dim-small target detection and reports mAP$$DA(q) = \sum_{l=1}^{L} W_l \sum_{k=1}^{K} A_{lk}(q)\cdot x^{l}(p_q + \Delta p_{lk}(q)),$$4 $$DA(q) = \sum_{l=1}^{L} W_l \sum_{k=1}^{K} A_{lk}(q)\cdot x^{l}(p_q + \Delta p_{lk}(q)),$$5 and F1 $$DA(q) = \sum_{l=1}^{L} W_l \sum_{k=1}^{K} A_{lk}(q)\cdot x^{l}(p_q + \Delta p_{lk}(q)),$$6 on DAUB, and mAP$$DA(q) = \sum_{l=1}^{L} W_l \sum_{k=1}^{K} A_{lk}(q)\cdot x^{l}(p_q + \Delta p_{lk}(q)),$$7 $$DA(q) = \sum_{l=1}^{L} W_l \sum_{k=1}^{K} A_{lk}(q)\cdot x^{l}(p_q + \Delta p_{lk}(q)),$$8 and F1 $$DA(q) = \sum_{l=1}^{L} W_l \sum_{k=1}^{K} A_{lk}(q)\cdot x^{l}(p_q + \Delta p_{lk}(q)),$$9 on IRDST (Luo et al., 2024). In tracking, the gated deformable tracker improved OTB-2013 AUC from $Y = \sigma \odot X' + (1-\sigma)\odot X,$0 to $Y = \sigma \odot X' + (1-\sigma)\odot X,$1 and retained the deformation-subset gain from deformable sampling (Liu et al., 2018). The tracking literature had already anticipated one enduring theme: simple fusion is insufficient when pose, occlusion, or viewpoint shift changes the underlying spatial relationship between evidence sources (Ma et al., 2015, Liu et al., 2018).

5. Empirical record

The empirical record is heterogeneous, but a consistent pattern emerges: gains are strongest when fusion must absorb misregistration, large motion, scale disparity, or nontrivial modality complementarity.

Area	Representative result	Paper
Brain registration	Dice $Y = \sigma \odot X' + (1-\sigma)\odot X,$2, TRE $Y = \sigma \odot X' + (1-\sigma)\odot X,$3, NDV $Y = \sigma \odot X' + (1-\sigma)\odot X,$4, HdDist95 $Y = \sigma \odot X' + (1-\sigma)\odot X,$5	(Zhou et al., 2024)
Automatic registration fusion	AutoFuse unsupervised brain: OASIS/Mindboggle/Buckner DSC/NJD $Y = \sigma \odot X' + (1-\sigma)\odot X,$6, $Y = \sigma \odot X' + (1-\sigma)\odot X,$7, $Y = \sigma \odot X' + (1-\sigma)\odot X,$8	(Meng et al., 2023)
Cardiac cine MRI reconstruction	ACDC at $Y = \sigma \odot X' + (1-\sigma)\odot X,$9: SSIM $0.7727 \pm 0.0276$0, PSNR $0.7727 \pm 0.0276$1 dB, NMSE $0.7727 \pm 0.0276$2	(Han et al., 2023)
HSI-X semantic segmentation	Houston2013: OA $0.7727 \pm 0.0276$3, AA $0.7727 \pm 0.0276$4, $0.7727 \pm 0.0276$5	(Zhang et al., 2024)
3D object detection	FusionFormer: $0.7727 \pm 0.0276$6 mAP and $0.7727 \pm 0.0276$7 NDS	(Hu et al., 2023)
End-to-end perception and prediction	Li-ViP3D++: EPA $0.7727 \pm 0.0276$8, mAP $0.7727 \pm 0.0276$9, FP ratio $2.4663$0	(Halinkovic et al., 28 Jan 2026)
Infrared dim-small target detection	DAUB: mAP$2.4663$1 $2.4663$2, F1 $2.4663$3; IRDST: mAP$2.4663$4 $2.4663$5, F1 $2.4663$6	(Luo et al., 2024)
Brain tumor segmentation	WT/TC/ET Dice $2.4663$7	(Nian et al., 2023)
Visual–tactile grasp assessment	Classification accuracy $2.4663$8	(Cui et al., 2020)

Several ablation studies make the same point more directly. On Houston2013, CoMiX improves OA from $2.4663$9 for the baseline to $0.0130$0 after adding 2D DCN blocks, the EPA-based HSI block, CMFeX, and FFM (Zhang et al., 2024). On ACDC, second-order bidirectional propagation improves PSNR from $0.0130$1 to $0.0130$2 and SSIM from $0.0130$3 to $0.0130$4 at $0.0130$5, and from $0.0130$6 to $0.0130$7 and $0.0130$8 to $0.0130$9 at $3.3197$0 (Han et al., 2023). On IRDST, the combination of TDA, feature refinement, and motion compensation loss raises mAP$3.3197$1 from $3.3197$2 to $3.3197$3 (Luo et al., 2024). This suggests that deformable fusion is especially effective when the fusion stage is itself asked to recover spatial correspondence rather than merely average already aligned features.

6. Misconceptions, limitations, and research directions

A recurring misconception is that deformable fusion eliminates the need for geometry, regularization, or carefully structured training. The literature argues otherwise. FusionFormer is sensitive to calibration, and dynamic scenes and fast motion still leave residual misalignments even after ego-pose compensation (Hu et al., 2023). Li-ViP3D++ improves EPA and false-positive behavior, but its forecasting displacement errors are slightly worse than Li-ViP3D, indicating that better detection calibration does not automatically dominate every downstream metric (Halinkovic et al., 28 Jan 2026). MMF-BEV explicitly relies on a two-stage training strategy because decoupling depth learning from cross-modal alignment stabilizes learning (Mayank et al., 6 Apr 2026). DFAR reports that training deformable offsets is unstable without the motion compensation loss, and the paper attributes part of its gains to offset stabilization rather than fusion alone (Luo et al., 2024).

A second misconception is that deformability always improves robustness without trade-offs. In the gated deformable tracker, the method achieves top success AUC on OTB-2013 and strong deformation handling, yet the results also indicate a minor robustness cost relative to some alternatives on distance precision or robustness metrics (Liu et al., 2018). EDVR reduces residual flow magnitude through PCD alignment, but extremely large motions, heavy occlusions, and severe blur can still challenge offset prediction and may require two-stage restoration or broader temporal context (Wang et al., 2019). In CoMiX, extreme misregistration or large inter-sensor parallax may exceed the corrective capacity of learned 2D offsets and CMFeX attention (Zhang et al., 2024). In deformable registration, the shared-encoder auxiliary-decoder model reports very low NDV without explicit diffeomorphic constraints, but the manuscript also states that diffeomorphism is not enforced in the loss (Zhou et al., 2024).

The open problems are correspondingly concrete. Multiple papers propose offset regularization or prior constraints to stabilize learned offsets, explicit geometric alignment cues such as coarse flow or disparity priors, hybrid deformable attention to unify offset-based and attention-based adaptability, confidence-aware attention, modality-aware gates, Bayesian gates with uncertainty-aware fusion, multi-scale positional relationship estimation, and differentiable architecture search for fusion placement or operator selection (Zhang et al., 2024, Liu et al., 2023, Mayank et al., 6 Apr 2026, Meng et al., 2023). Deformba frames the issue at the sequence-modeling level: fixed scanning orders impose predefined geometric structures, and query-based interactions remain difficult for causal vision SSMs unless one explicitly decouples write and deformable read stages (Ke et al., 20 May 2026). The field therefore points toward a broader synthesis in which deformable fusion is not an isolated module, but part of a larger program of adaptive correspondence, sparse geometric interaction, and task-aware fusion control across images, volumes, videos, queries, and states (Zhou et al., 2024, Li et al., 1 Jul 2025, Zhang et al., 2022).