MMIF-AMIN: Adaptive Multimodal Image Fusion

• The paper introduces MMIF-AMIN, which uses an adaptive mixture of instance and group normalization (AMIN) to balance fusion across natural and medical modalities.
• It incorporates large-kernel convolution designs and a multipath adaptive fusion module to refine feature aggregation and improve downstream tasks like detection and segmentation.
• Empirical results demonstrate state-of-the-art performance with improved structural, intensity, and gradient fidelity across various benchmarks.

MMIF-AMIN denotes an AMIN-based multimodal image fusion framework in which the core design question is not only how to combine modalities, but also how normalization strategy and convolutional kernel scale affect fusion behavior across substantially different regimes such as natural and medical image fusion. In the formulation introduced in "Rethinking Normalization Strategies and Convolutional Kernels for Multimodal Image Fusion" (He et al., 2024), AMIN refers to an adaptive mixture of instance normalization and group normalization, and the complete implementation is instantiated as LKC-FUNet: an encoder–decoder architecture that combines an IN-fronted initialization block, GN-based deeper blocks, large-kernel convolutions, and a multipath adaptive fusion module. The stated motivation is that existing methods often emphasize complementary information and training strategy while under-addressing the distinction between natural and medical image fusion, as well as the influence of underlying architectural components (He et al., 2024).

1. Conceptual basis and problem setting

Multimodal image fusion aims to integrate information from different modalities into a single image that is more comprehensive for downstream use. The motivating examples in the source material include medical fusion pairs such as MRI–CT, MRI–PET, and MRI–SPECT, and infrared–visible fusion benchmarks such as MSRS, M³FD, and TNO (He et al., 2024). The framework is explicitly positioned against the tendency of prior methods to prioritize natural image fusion and to concentrate on information complementarity and training strategy while ignoring differences in fusion goals, statistical properties, and data distribution between natural and medical settings (He et al., 2024).

Within this framing, AMIN is not presented merely as a normalization trick. It is embedded in a broader rethinking of end-to-end MMIF, in which sample independence, intrinsic feature correlation, receptive field size, and skip-connection recalibration are treated as interacting design variables. This suggests that the method views normalization and kernel geometry as first-class determinants of fusion quality rather than secondary implementation choices.

The paper’s abstract states that the proposed method exhibits state-of-the-art performance in multiple fusion tasks and significantly improves downstream applications (He et al., 2024). In the detailed summary, those downstream applications include object detection with YOLOv5 on M³FD and semantic segmentation with DeepLabV3 on MSRS (He et al., 2024).

2. AMIN normalization: formulation and architectural realization

The mathematical formulation starts from a minibatch of feature maps $X \in \mathbb{R}^{N \times C \times H \times W}$. The source defines instance normalization, $\mathrm{IN}(X)$, through per-sample, per-channel statistics, and group normalization, $\mathrm{GN}(X)$, through statistics computed over channel groups within each sample (He et al., 2024). The simplest adaptive mixture is written as

$$\mathrm{AMIN}(X) = \alpha \cdot \mathrm{IN}(X) + (1-\alpha)\cdot \mathrm{GN}(X),$$

where $\alpha \in [0,1]$ may be learned as a scalar or channel-wise gating coefficient (He et al., 2024).

The implemented architecture does not use a fully dynamic per-layer $\alpha$. Instead, it adopts what the source calls a “hybrid” approach: the first block, denoted InitBlock, applies Instance Normalization, whereas all deeper encoder and decoder blocks use Group Normalization (He et al., 2024). The InitBlock applies IN–15×15 large-kernel convolution–ReLU, and the deeper blocks use GN + ReLU before each convolution (He et al., 2024).

This fixed split has a specific rationale. IN at the front “equalize[s] sample statistics early,” and GN in deeper layers is used “to maintain inter-feature correlation and stable gradients” (He et al., 2024). Because IN is applied only at the very front, samples are never batch-pooled across one another, while GN keeps feature-group statistics tight (He et al., 2024). The source explicitly interprets the result as preserving per-sample independence while still allowing deeper layers to exploit feature covariance.

A recurrent misconception would be to treat AMIN here as a fully learned interpolation between IN and GN throughout the network. The paper summary states the opposite: the present AMIN scheme is “a fixed IN→GN split rather than a dynamically learned blend,” and future work might explore per-layer or per-channel learnable $\alpha$ coefficients or alternative normalization families such as switchable normalization (He et al., 2024).

3. Large-kernel convolutional design

The convolutional design comprises two block types: LKCBlock and LKDCBlock (He et al., 2024). The LKCBlock is described as a small residual stack of the form GN → ReLU → optional Dropout → Conv($k \times k$) → GN → ReLU → Conv($k \times k$) + residual. Kernel sizes vary with resolution, with examples given as $k=15$ at the highest resolution, followed by $\mathrm{IN}(X)$0, $\mathrm{IN}(X)$1, and $\mathrm{IN}(X)$2 in deeper stages (He et al., 2024).

The rationale is resolution-dependent. Larger kernels at early high-resolution stages expand the effective receptive field to capture global structure, while smaller kernels at deeper low-resolution stages focus on local details (He et al., 2024). The convolution operation is written explicitly as

$\mathrm{IN}(X)$3

The second block, LKDCBlock, uses the RepLKNet reparameterized large-kernel convolution with kernels up to $\mathrm{IN}(X)$4 in a depthwise or group-wise manner (He et al., 2024). In this block, GN is set with $\mathrm{IN}(X)$5, described as “layer-normalization-style,” to avoid disconnecting channels before a depthwise convolution (He et al., 2024). The stated effect is to push the effective receptive field further without a huge parameter blowup.

These large-kernel components are therefore not uniform replacements for all convolutions. They are stage-sensitive and integrated with the normalization policy. A plausible implication is that the paper treats receptive-field allocation and normalization granularity as coupled mechanisms for controlling the trade-off between global structural aggregation and local detail retention.

4. Multipath adaptive fusion and training objective

The Multipath Adaptive Fusion Module, or MPAFM, is inserted into each skip connection between encoder and decoder features of the same spatial size (He et al., 2024). Its operation is described in three steps.

First, it forms an attention feature map from encoder feature $\mathrm{IN}(X)$6 and decoder feature $\mathrm{IN}(X)$7. Channel attention is applied to $\mathrm{IN}(X)$8, spatial attention to $\mathrm{IN}(X)$9, and the two are fused as $\mathrm{GN}(X)$0. A small conv-ReLU-conv module then doubles the channels and splits them into two masks $\mathrm{GN}(X)$1 and $\mathrm{GN}(X)$2, which are applied to the encoder and decoder features respectively, producing $\mathrm{GN}(X)$3 and $\mathrm{GN}(X)$4 (He et al., 2024).

Second, the module performs bidirectional interaction:

$\mathrm{GN}(X)$5

so that each path modulates the other and information flows in both directions (He et al., 2024).

Third, a final small convolutional block produces a pixel-attention map used to weight the decoder input (He et al., 2024). The source characterizes this mechanism as recalibrating the decoder input with features of various scales and receptive fields, thereby ensuring transmission of crucial information (He et al., 2024).

Training uses Adam with initial learning rate $\mathrm{GN}(X)$6, 1000 epochs for medical fusion, crop size $\mathrm{GN}(X)$7, and batch size 32 (He et al., 2024). The total loss is the sum of structural, intensity, and gradient terms:

$\mathrm{GN}(X)$8

The structural-level loss is

$\mathrm{GN}(X)$9

where the three-image SSIM compares the fused image $$\mathrm{AMIN}(X) = \alpha \cdot \mathrm{IN}(X) + (1-\alpha)\cdot \mathrm{GN}(X),$$0 to the stronger luminance of $$\mathrm{AMIN}(X) = \alpha \cdot \mathrm{IN}(X) + (1-\alpha)\cdot \mathrm{GN}(X),$$1 and $$\mathrm{AMIN}(X) = \alpha \cdot \mathrm{IN}(X) + (1-\alpha)\cdot \mathrm{GN}(X),$$2 (He et al., 2024). The intensity-level loss emphasizes high-brightness preservation via an $$\mathrm{AMIN}(X) = \alpha \cdot \mathrm{IN}(X) + (1-\alpha)\cdot \mathrm{GN}(X),$$3 distance to $$\mathrm{AMIN}(X) = \alpha \cdot \mathrm{IN}(X) + (1-\alpha)\cdot \mathrm{GN}(X),$$4, and the gradient-level loss emphasizes edge and detail consistency via an $$\mathrm{AMIN}(X) = \alpha \cdot \mathrm{IN}(X) + (1-\alpha)\cdot \mathrm{GN}(X),$$5 distance to $$\mathrm{AMIN}(X) = \alpha \cdot \mathrm{IN}(X) + (1-\alpha)\cdot \mathrm{GN}(X),$$6 (He et al., 2024).

The loss design therefore combines structural fidelity, bright-region preservation, and edge transfer in a single objective. This suggests a deliberately balanced optimization target rather than a purely perceptual or purely reconstruction-based criterion.

5. Empirical performance and downstream effects

The reported experiments cover medical image fusion, infrared–visible fusion, ablations, and downstream tasks (He et al., 2024). The medical evaluation uses 50 test pairs for each of MRI–CT, MRI–PET, and MRI–SPECT, with metrics SD, AG, SF, SCD, VIFF, and SSIM, where higher values are better (He et al., 2024). For MRI–CT, the method reports SD = 94.03, AG = 8.24, SF = 33.84, SCD = 1.576, VIFF = 0.456, and SSIM = 0.731, with SD described as “↑ +7.5 vs second-best” and all listed metrics marked best for this example (He et al., 2024). On MRI–PET and MRI–SPECT, the summary states that the method is consistently top or second-best in all six metrics (He et al., 2024).

For infrared–visible fusion, the paper evaluates on MSRS, M³FD, and TNO (He et al., 2024). On MSRS, which contains 361 pairs, the method reports SD = 46.57, AG = 4.94, SF = 15.38, SCD = 1.682, VIFF = 0.883, and SSIM = 0.680, and the summary states that it generalizes without fine-tuning to M³FD and TNO while outperforming seven SOTA methods (He et al., 2024).

The following benchmark highlights are explicitly given in the source.

Setting	Result	Source
MRI–CT	SD 94.03, AG 8.24, SF 33.84, SCD 1.576, VIFF 0.456, SSIM 0.731	(He et al., 2024)
MSRS	SD 46.57, AG 4.94, SF 15.38, SCD 1.682, VIFF 0.883, SSIM 0.680	(He et al., 2024)
M³FD detection	mAP@.5 = 87.2% with YOLOv5	(He et al., 2024)
MSRS segmentation	mIoU = 67.3% with DeepLabV3	(He et al., 2024)

For downstream detection on M³FD using YOLOv5, the reported [email protected] values are 83.8% for IR alone, 83.2% for visible alone, 85.8% for CDDFuse, 86.4% for EMMA, and 87.2% for the proposed method (He et al., 2024). For semantic segmentation on MSRS using DeepLabV3, the reported mIoU values are 59.2% for IR, 62.6% for visible, 66.0% for CDDFuse, 66.8% for MMDRFuse, and 67.3% for the proposed method (He et al., 2024).

These results situate MMIF-AMIN not only as a fusion architecture but also as a representation-learning front end for downstream vision pipelines. The comparison against EMMA in detection is especially notable because EMMA is itself a method designed to improve downstream tasks through a self-supervised equivariant paradigm (Zhao et al., 2023).

6. Ablation findings, relation to adjacent MMIF paradigms, and limitations

The ablation study on MRI–CT directly isolates the effects of normalization, kernel size, and skip-fusion design (He et al., 2024). Configuration I, using all BN and $$\mathrm{AMIN}(X) = \alpha \cdot \mathrm{IN}(X) + (1-\alpha)\cdot \mathrm{GN}(X),$$7 convolutions, reports SD = 76.55, AG = 8.15, SF = 28.28, and SSIM = 0.736. Configuration II, all BN with LKC, reports SD = 73.99 and SSIM = 0.748. Configuration III, BN+GN+LKC, reports SD = 78.51 and SSIM = 0.746. Configuration IV, IN+GN+$$\mathrm{AMIN}(X) = \alpha \cdot \mathrm{IN}(X) + (1-\alpha)\cdot \mathrm{GN}(X),$$8, reports SD = 80.00 and SSIM = 0.735. Configuration V, with no normalization, reports SD = 78.41 and SSIM = 0.747. Configuration VI, without MPAFM, reports SD = 81.20 and SSIM = 0.725. The full method, IN+GN+LKC+MPAFM, reports SD = 83.29, AG = 9.13, SF = 33.30, SCD = 1.652, VIFF = 0.578, and SSIM = 0.727 (He et al., 2024).

The source interpretation is explicit: replacing BN with IN+GN raises the upper bound; adding LKC further boosts fine detail; and MPAFM provides another gain in structure and detail flow (He et al., 2024). This interpretation should be read with care, because different metrics do not peak in exactly the same configuration. The ablation therefore supports a multi-objective view of performance rather than a single-metric optimization.

Relative to adjacent MMIF paradigms, MMIF-AMIN differs from EMMA in both supervision and inductive bias. EMMA is a fully self-supervised framework organized around a fusion module, a pseudo-sensing module, and an equivariant fusion module, with the prior that the sensing–fusion chain is equivariant to transformations such as rotations, flips, and shifts (Zhao et al., 2023). By contrast, the AMIN-based method centers normalization strategy, large-kernel receptive-field engineering, and multipath skip recalibration (He et al., 2024). Both frameworks are reported to work across infrared–visible and medical settings, but they attack the cross-domain generalization problem through different mechanisms.

The limitations stated for MMIF-AMIN are threefold. First, although LKC-FUNet unifies IVIF and MIF in one encoder–decoder, training solely on IR–VIS pairs or solely on medical pairs still biases detail and noise behavior; the source states that achieving a truly task-agnostic, one-shot fusion model across all modalities may require a more sophisticated domain-balancing or meta-normalization strategy (He et al., 2024). Second, inference speed is competitive but not the fastest, and further reparameterization or pruning of the large kernels and attention paths could improve real-time applicability (He et al., 2024). Third, as noted above, the present AMIN scheme is fixed rather than dynamically learned, leaving open the possibility of learnable $$\mathrm{AMIN}(X) = \alpha \cdot \mathrm{IN}(X) + (1-\alpha)\cdot \mathrm{GN}(X),$$9 coefficients or alternative normalization families (He et al., 2024).

Taken together, these limitations clarify the scope of MMIF-AMIN. It is not presented as a final universal fusion model, but as a component-level reconsideration of MMIF architecture in which normalization placement, kernel scale, and skip-path interaction materially change fusion behavior across modalities and downstream tasks (He et al., 2024).