Adaptive Fusion Modulation (AFM)

• Adaptive Fusion Modulation (AFM) is a data-driven fusion method that dynamically weights multimodal inputs, such as RGB and thermal images, to enhance reliability under varying conditions.
• It employs global gating and finer granularity mechanisms to adaptively balance modality contributions based on scene context and representation quality.
• Empirical studies show that AFM reduces error rates in tasks like crowd counting and facial expression recognition, demonstrating practical improvements across domains.

Searching arXiv for the cited paper and closely related AFM usages to ground the article. Adaptive Fusion Modulation (AFM) denotes a class of multimodal mechanisms in which fusion is not fixed by concatenation, uniform summation, or a single static rule, but is modulated by data-dependent weights, masks, affine transforms, or related control signals. In the RGB–thermal crowd-counting setting, AFM is a late-fusion module that computes a global scalar $w \in [0,1]$ from joint RGB–thermal content and uses that scalar to blend modality-specific feature maps. Across the broader literature, the same general idea appears as channel-wise importance weighting, spatially guided modulation, affine conditioning, confidence-ratio masking, graph-aware attention, and depth-adaptive optimization, all aimed at making multimodal fusion conditional on reliability, complementarity, or task context rather than treating modalities uniformly (Feng et al., 21 Sep 2025, Sui et al., 2022, Ni et al., 9 Sep 2025, Shen et al., 27 Aug 2025).

1. Definition and motivation

In RGB–T crowd counting, AFM is introduced to address a concrete asymmetry: RGB and thermal imagery do not remain equally reliable across scenes. Bright, well-lit scenes favor RGB because it carries rich texture and color; low-light, nighttime, or heavily occluded scenes often favor thermal because it captures human body heat and is less affected by illumination; noisy thermal scenes can again shift preference toward RGB. The motivation is therefore not merely cross-modal interaction at the feature level, but a macro-level, scene-aware mechanism that decides which modality should dominate overall (Feng et al., 21 Sep 2025).

More generally, the literature frames adaptive fusion modulation as the replacement of fixed fusion with dynamic, learned modulation and weighting. In 2D+3D facial expression recognition, the term is used for feature-map-level modulation based on learned importance and spatial salience rather than direct concatenation after fully connected layers. That formulation emphasizes that multimodal fusion should be data-dependent, position-dependent, and channel-dependent when the modalities contribute unequally across conditions (Sui et al., 2022).

A related motivation appears in multimodal object detection and medical image–tabular fusion. LASFNet attributes inefficiency in earlier detectors to repeated multi-stage fusion and instead proposes a single feature-level fusion unit with attention-guided self-modulation, while AMF-MedIT focuses on dimension discrepancies, modality-contribution imbalance, and noise from high-dimensional tabular inputs. Taken together, these works suggest that AFM is typically motivated by one or more of three recurring problems: modality reliability shifts, representational mismatch, and over- or under-allocation of capacity to particular modalities (Hao et al., 26 Jun 2025, Yu et al., 24 Jun 2025).

2. Specific formulation in RGB–T crowd counting

A particularly explicit formulation appears in the RGB–T crowd-counting framework that couples AFM with Spatially Modulated Attention (SMA). A shared VGG-19 backbone produces $F_r$ and $F_t$ from RGB and thermal inputs, modality-specific Transformer encoders with SMA produce $F'_r$ and $F'_t$, AFM fuses them once per forward pass, and a regression head maps the fused feature to a density map. AFM first forms a joint summary feature

$F_{\text{sum}} = F'_r + F'_t,$

then applies global average pooling and a two-layer MLP realized as $1 \times 1$ convolutions,

$$w = \sigma\!\left(\text{Conv}_{2,1\times1}\!\left(\text{ReLU}\!\left(\text{Conv}_{1,1\times1}\!\left(\text{AvgPool}(F_{\text{sum}})\right)\right)\right)\right),$$

and finally computes the fused map

$F_{\text{fused}} = w \cdot F'_r + (1-w)\cdot F'_t.$

The paper explicitly states that $w$ is a scalar between 0 and 1 representing a global, scene-level assessment of RGB reliability, with $F_r$0 representing the complementary importance of thermal (Feng et al., 21 Sep 2025).

This design fixes the operational granularity of AFM very precisely. Gating is scene-level because the module outputs one global scalar; fusion is feature-map-level because the scalar is applied element-wise across all channels and spatial locations. No per-token, per-channel, or per-pixel gate is used. The stated rationale is that a global scalar weight is preferred over a spatially varying weight map “to enhance adaptability and prevent overfitting,” thereby encouraging a scene-level fusion strategy instead of noisy pixel-wise decisions.

The relation between SMA and AFM is sequential and complementary. SMA addresses where to look by introducing a learnable Spatial Decay Mask that penalizes attention between distant tokens and suppresses background spread; AFM addresses which modality to trust by weighting RGB and thermal after SMA has already sharpened the modality-specific feature maps. This division of labor is central to the “Dual Modulation Framework”: one modulation is spatial, the other is cross-modal (Feng et al., 21 Sep 2025).

3. Modulation granularity and mechanism

The cited literature suggests a useful typology of AFM mechanisms by granularity. At one end is the global scalar gate of RGB–T crowd counting. At finer granularity, AFNet-M combines mask-guided spatial modulation and channel-wise importance weighting. Its Mask Attention module learns $F_r$1 and $F_r$2 from facial masks and applies

$F_r$3

while its Importance Weights Computing module derives channel-wise modality weights from average and max pooling before weighted convolutional fusion at Layer3 and Layer4. LASFNet goes further toward mixed granularity: its ASFF module applies modality-wise channel attention, positional attention on the fused feature, and then global and local self-modulation branches before channel shuffle; its FATM module adds lightweight channel and positional attention at the neck. LEAF-Mamba shifts the mechanism into state-space modeling: Cross-modality Second-order Pooling produces similarity maps $F_r$4 and distance maps $F_r$5, which then modulate SSM output matrices $F_r$6 in a Selective Interaction Module and a Selective Enhancement Module for complementary interaction and reliable fusion (Sui et al., 2022, Hao et al., 26 Jun 2025, Wu et al., 23 Sep 2025).

Other AFM realizations use conditioning or allocation rather than direct attention maps. AFN-HearNet generates per-frequency and per-channel affine parameters $F_r$7, $F_r$8, and $F_r$9 from audiogram embeddings and applies them before Conformer and MLP stages in both frequency-wise and temporal-wise fusion. AMF-MedIT uses a modality confidence ratio $F_t$0 to allocate retained dimensions $F_t$1 and $F_t$2 inside a fixed output dimension, then enforces that allocation with binary masks together with density and leakage losses. ADM-DP computes modality weights with

$F_t$3

and fuses vision, tactile, and graph features as

$F_t$4

so that modality contribution varies with task phase and context. In wireless signal analysis, ModFus-DM extracts multiple diffusion-model block features, concatenates selected blocks, and learns a discriminative projection $F_t$5; the paper explicitly interprets that projection as an adaptive weighting or gating over multi-scale diffusion features (Ni et al., 9 Sep 2025, Yu et al., 24 Jun 2025, Wang et al., 25 Feb 2026, Tan et al., 3 Aug 2025).

4. Empirical evidence

In RGB–T crowd counting, the ablation on RGBT-CC isolates AFM’s contribution cleanly. The baseline without SMA and without AFM, using simple summation, reports GAME(0) $F_t$6 and RMSE $F_t$7. Adding SMA alone reduces these to GAME(0) $F_t$8 and RMSE $F_t$9. Replacing simple summation with AFM yields GAME(0) $F'_r$0 and RMSE $F'_r$1, with corresponding improvements on GAME(1), GAME(2), and GAME(3). The full model also reports, on RGBT-CC, GAME(0) $F'_r$2, GAME(1) $F'_r$3, GAME(2) $F'_r$4, GAME(3) $F'_r$5, and RMSE $F'_r$6, and on DroneRGBT, GAME(0) $F'_r$7 and RMSE $F'_r$8 (Feng et al., 21 Sep 2025).

Comparable empirical patterns appear in other domains. In AFNet-M, convolutional fusion without IWC yields $F'_r$9 and $F'_t$0 on BU-3DFE and Bosphorus, while convolutional fusion with IWC reaches $F'_t$1 and $F'_t$2, and adding mask-based modulation raises performance further to $F'_t$3 and $F'_t$4. In LASFNet on LLVIP, the single-feature fusion baseline reports mAP $F'_t$5, the full ASFF module reaches $F'_t$6, and ASFF with FATM reaches $F'_t$7, while retaining markedly lower parameter count and GFLOPs than heavier baselines such as GM-DETR or Fusion-Mamba (Sui et al., 2022, Hao et al., 26 Jun 2025).

In personalized speech enhancement, AFN-HearNet reports that the AMFT-Conformer variant improves over its FT-Conformer baseline on the DNS test set from HASQI $F'_t$8 to $F'_t$9, WB-PESQ $F_{\text{sum}} = F'_r + F'_t,$0 to $F_{\text{sum}} = F'_r + F'_t,$1, NB-PESQ $F_{\text{sum}} = F'_r + F'_t,$2 to $F_{\text{sum}} = F'_r + F'_t,$3, SDR $F_{\text{sum}} = F'_r + F'_t,$4 to $F_{\text{sum}} = F'_r + F'_t,$5, SI-SNR $F_{\text{sum}} = F'_r + F'_t,$6 to $F_{\text{sum}} = F'_r + F'_t,$7, and STOI $F_{\text{sum}} = F'_r + F'_t,$8 to $F_{\text{sum}} = F'_r + F'_t,$9. In medical image–tabular fusion, AMF-MedIT with FT-Mamba reports $1 \times 1$0 AUC with frozen encoders and $1 \times 1$1 with trainable encoders on OL3I, outperforming concatenation and cross-attention baselines under the reported settings. In multi-agent manipulation, ADM-DP reports an overall average success rate of $1 \times 1$2 versus $1 \times 1$3 for Flow Policy across seven tasks, and its ablation without adaptive modality attention falls to $1 \times 1$4 (Ni et al., 9 Sep 2025, Yu et al., 24 Jun 2025, Wang et al., 25 Feb 2026).

5. Optimization-level modulation and signal-domain variants

AFM is not restricted to explicit feature-fusion blocks. In automatic modulation classification, AFNet performs multi-scale feature extraction with two branches and fuses them channel-wise through a $1 \times 1$5-Softmax rule,

$1 \times 1$6

then adds a confidence-weighted loss in a two-stage learning scheme. With $1 \times 1$7 in its top-$1 \times 1$8 entropy weighting, the reported average accuracy over $1 \times 1$9 to $$w = \sigma\!\left(\text{Conv}_{2,1\times1}\!\left(\text{ReLU}\!\left(\text{Conv}_{1,1\times1}\!\left(\text{AvgPool}(F_{\text{sum}})\right)\right)\right)\right),$$0 dB is $$w = \sigma\!\left(\text{Conv}_{2,1\times1}\!\left(\text{ReLU}\!\left(\text{Conv}_{1,1\times1}\!\left(\text{AvgPool}(F_{\text{sum}})\right)\right)\right)\right),$$1, and the same confidence-weighting idea improves several baseline AMC models as well (Shi et al., 2022).

A broader optimization view appears in Adaptive Gradient Modulation and Adaptive Intra-Network Modulation. AGM computes Shapley-style mono-modal responses $$w = \sigma\!\left(\text{Conv}_{2,1\times1}\!\left(\text{ReLU}\!\left(\text{Conv}_{1,1\times1}\!\left(\text{AvgPool}(F_{\text{sum}})\right)\right)\right)\right),$$2, derives mono-modal cross-entropies $$w = \sigma\!\left(\text{Conv}_{2,1\times1}\!\left(\text{ReLU}\!\left(\text{Conv}_{1,1\times1}\!\left(\text{AvgPool}(F_{\text{sum}})\right)\right)\right)\right),$$3, constructs discrepancy ratios $$w = \sigma\!\left(\text{Conv}_{2,1\times1}\!\left(\text{ReLU}\!\left(\text{Conv}_{1,1\times1}\!\left(\text{AvgPool}(F_{\text{sum}})\right)\right)\right)\right),$$4 and running references $$w = \sigma\!\left(\text{Conv}_{2,1\times1}\!\left(\text{ReLU}\!\left(\text{Conv}_{1,1\times1}\!\left(\text{AvgPool}(F_{\text{sum}})\right)\right)\right)\right),$$5, and scales gradients with

$$w = \sigma\!\left(\text{Conv}_{2,1\times1}\!\left(\text{ReLU}\!\left(\text{Conv}_{1,1\times1}\!\left(\text{AvgPool}(F_{\text{sum}})\right)\right)\right)\right),$$6

AIM moves the modulation signal inside the network by partitioning encoders into depth blocks, constructing depth-aware prototypes, decoupling well-optimized and under-optimized parameters into full and Auxiliary Blocks, and weighting depth-wise modulation losses by imbalance coefficients $$w = \sigma\!\left(\text{Conv}_{2,1\times1}\!\left(\text{ReLU}\!\left(\text{Conv}_{1,1\times1}\!\left(\text{AvgPool}(F_{\text{sum}})\right)\right)\right)\right),$$7. Taken together, these studies suggest that AFM can be understood not only as adaptive feature aggregation but also as adaptive control over how modalities compete during optimization (Li et al., 2023, Shen et al., 27 Aug 2025).

6. Terminology, limitations, and extensions

The term AFM is not semantically stable across arXiv usage. In diffusion-model research, AFM can denote “Attention Frequency Modulation,” a training-free method that edits token-wise cross-attention logits in the Fourier domain with low- and high-frequency reweighting schedules. That method is explicitly not “Adaptive Fusion Modulation,” even though it also involves adaptive weighting and modulation (Oh et al., 30 Mar 2026).

Within adaptive fusion work proper, limitations recur. The RGB–T crowd-counting AFM uses a single scalar gate, so it cannot model scenes where one region is better sensed by thermal and another by RGB; it also assumes aligned RGB and thermal maps and does not use explicit modality-specific quality statistics. LEAF-Mamba’s AFM is efficient because it relies on SSMs and downsampled second-order pooling, but it still adds second-order computation and is formulated for two modalities. AMF-MedIT explicitly notes that $$w = \sigma\!\left(\text{Conv}_{2,1\times1}\!\left(\text{ReLU}\!\left(\text{Conv}_{1,1\times1}\!\left(\text{AvgPool}(F_{\text{sum}})\right)\right)\right)\right),$$8 is manually tuned and that the reported study covers two modalities and classification tasks. ADM-DP reports simulation results and notes future work around real-robot deployment, more advanced tactile sensing, and more sophisticated attention architectures (Feng et al., 21 Sep 2025, Wu et al., 23 Sep 2025, Yu et al., 24 Jun 2025, Wang et al., 25 Feb 2026).

Several papers also indicate clear extension paths. The RGB–T crowd-counting work describes AFM as a “macro-level, scene-aware fusion strategy” and mentions natural extensions such as multi-head fusion, channel-wise or region-wise gating, and transfer to other RGB–T tasks. Training-free multi-style fusion in diffusion models uses semantic token decomposition and similarity-aware re-weighting to balance multiple style references, showing that adaptive modulation can be generalized beyond dual-modality sensing to multi-reference generative control (Feng et al., 21 Sep 2025, Liu et al., 23 Sep 2025).

Taken together, the literature suggests that AFM is best understood as a design principle rather than a single architecture. Its central commitment is that fusion should be conditioned on modality quality, scene context, representational scale, or optimization state. The principal axes of variation are the granularity of modulation—scalar, channel, spatial, temporal, graph-structured, or depth-wise—the signal used to drive it, and the stage at which it acts, whether feature extraction, fusion, or training dynamics.