Intermediate-Modality Fusion

• Intermediate-modality fusion is a multimodal learning approach that combines learned hidden representations from modality-specific modules before final prediction, bridging early and late fusion.
• Architectural designs typically use separate encoders with dedicated fusion modules employing attention, gating, or aggregation to align and blend modality-specific features.
• The approach is widely applied in fields like medical segmentation, remote sensing, and object detection, enhancing interpretability, robustness, and prediction accuracy.

Searching arXiv for recent and foundational papers on intermediate-modality fusion. Intermediate-modality fusion is a multimodal learning strategy in which each modality is first transformed by a dedicated unimodal module into a learned representation, these hidden representations are fused inside the network, and a multimodal module maps the fused state to the output. In the notation used in the biomedical systematic review, $$h_1, h_2, \ldots, h_n = f_1(x_1), f_2(x_2), \ldots, f_n(x_n)$$, then $h = \mathscr{F}(h_1, h_2, \ldots, h_n)$, and finally $y = f(h)$ (Guarrasi et al., 2024). This places intermediate fusion between input-level or early fusion, which combines raw modalities first, and decision-level or late fusion, which combines outputs only at the end (Zhou et al., 2020). Across the literature, the term covers layer-level fusion in medical segmentation, multi-layer fusion in multimodal classification, multi-scale decoder fusion in image fusion, query exchange in DETR-like detectors, and bottleneck fusion in diffusion transformers, but the common property is that cross-modal interaction occurs after some unimodal feature formation and before final prediction or synthesis (Vielzeuf et al., 2018).

1. Conceptual definition and taxonomic position

The core distinction among multimodal fusion regimes is the location of the fusion operator. Early fusion combines raw inputs or very low-level features; late fusion keeps modalities separate until the decision stage; intermediate fusion processes each modality sufficiently to form modality-aware hidden states and then combines those states inside the model (Guarrasi et al., 2024). In medical segmentation, the corresponding terminology is input-level fusion, layer-level fusion, and decision-level fusion, with layer-level fusion being the closest direct match to intermediate-modality fusion (Zhou et al., 2020).

Regime	Combination point	Formalization
Early fusion	Raw inputs or low-level features	$x=\mathscr{F}(x_1,\ldots,x_n)$, then $y=f(x)$
Intermediate fusion	Learned hidden representations	$h_i=f_i(x_i)$, then $h=\mathscr{F}(h_1,\ldots,h_n)$, then $y=f(h)$
Late fusion	Decision stage	$y_i=f_i(x_i)$, then $y=\mathscr{F}(y_1,\ldots,y_n)$

The systematic review of biomedical applications further refines intermediate fusion by asking four questions: what is fused, how many times fusion is applied, when fusion occurs, and how fusion is performed (Guarrasi et al., 2024). This yields distinctions such as single versus multiple fusion, sudden versus gradual versus multi-flow fusion, and synchronous versus asynchronous fusion. Sudden fusion merges all modalities at once; gradual fusion integrates them progressively; multi-flow designs build several parallel fusion flows. Synchronous fusion aligns modalities after the same number of layers, whereas asynchronous fusion fuses them after different depths, which is especially common when modalities are heterogeneous (Guarrasi et al., 2024).

This taxonomic structure is not merely terminological. It expresses a recurrent design problem: different modalities may require different amounts of modality-specific processing before cross-modal interaction becomes useful. CentralNet makes this point explicitly by moving beyond the “early fusion vs. late fusion” dichotomy and learning fusion at multiple intermediate layers of modality-specific deep networks (Vielzeuf et al., 2018).

2. Architectural forms and representational alignment

A canonical intermediate-fusion architecture uses one encoder per modality and one or more internal fusion modules. CentralNet exemplifies this pattern. Each modality is processed by its own deep network, each branch can make its own prediction, and a central network links the modality-specific networks throughout the hierarchy. The central network progressively aggregates modality information, provides a common feature embedding, and regularizes the modality-specific networks through a multitask objective (Vielzeuf et al., 2018). Because the fusion is repeated layer by layer, the model can continuously interpolate between early, intermediate, and late fusion rather than committing to a single fusion point.

A second architectural family emphasizes alignment of semantic depth rather than simple same-layer correspondence. MMA-UNet argues that infrared and visible features do not “enter” deep semantic space at the same rate, and its CKA analysis reports that visible features reach deeper semantic space roughly 10 layers earlier than infrared in the same UNet-style framework (Huang et al., 2024). On that basis, MMA-UNet separately trains specialized feature encoders for different modalities and fuses the first four layers of VI-UNet with the last four layers of IR-UNet. This cross-scale strategy is meant to place features in the same representation space before fusion, rather than forcing same-depth layers to interact when their abstraction levels differ (Huang et al., 2024).

A third family makes the fusion structure itself relational. The Interconnected Fusion framework for HSI-LiDAR classification introduces three encoded inputs—HSI input, replicated center HSI patch, and LiDAR input—and constructs a $h = \mathscr{F}(h_1, h_2, \ldots, h_n)$0 fusion matrix of nine views using self-attention and cross-attention (Huo et al., 2023). The nine elements represent HSI self-relationship, center-patch self-relationship, LiDAR self-relationship, center-to-neighborhood HSI relations, HSI-LiDAR relations, and center-HSI-to-LiDAR relations. The paper’s argument is that these interrelated views complement each other and eliminate bias more effectively than concatenation, summation, or limited encoder-decoder interaction (Huo et al., 2023).

These architectures share a common premise: intermediate fusion is as much about representational compatibility as about combination. The systematic review’s distinction between synchronous and asynchronous fusion formalizes this point at the survey level, while MMA-UNet and CentralNet implement it concretely through asymmetric layer pairing and learned layer-wise aggregation (Guarrasi et al., 2024).

3. Fusion operators, gating mechanisms, and training objectives

The simplest intermediate-fusion operators remain concatenation and tensor operations, and the biomedical review finds concatenation to be by far the most common method in the surveyed literature (Guarrasi et al., 2024). However, more recent work places increasing emphasis on attention, calibration, and learned gating, reflecting a shift from static fusion rules to adaptive modulation.

CentralNet uses trainable scalar weights to combine the central hidden representation and the modality-specific hidden representations at each depth, and it trains the full system with a multitask loss,

$h = \mathscr{F}(h_1, h_2, \ldots, h_n)$1

The paper states that including unimodal losses is crucial because it preserves unimodal performance, stabilizes training, and acts as a regularizer in the spirit of multitask learning (Vielzeuf et al., 2018).

In infrared-visible image fusion, FusionNet makes the two-stage structure of intermediate fusion especially explicit. Separate IR and VIS encoders produce modality-specific feature maps, a modality-aware attention mask performs feature-level fusion,

$h = \mathscr{F}(h_1, h_2, \ldots, h_n)$2

and a learned pixel-wise alpha map then controls the final grayscale image as a convex combination of infrared intensity and visible luminance (Sun et al., 14 Sep 2025). The same model adds a target-aware loss with weak ROI supervision,

$h = \mathscr{F}(h_1, h_2, \ldots, h_n)$3

with reported weights $h = \mathscr{F}(h_1, h_2, \ldots, h_n)$4, $h = \mathscr{F}(h_1, h_2, \ldots, h_n)$5, and $h = \mathscr{F}(h_1, h_2, \ldots, h_n)$6, thereby making the intermediate fusion process explicitly sensitive to pedestrians, vehicles, and other annotated targets (Sun et al., 14 Sep 2025).

A more robustness-oriented formulation appears in the Causal Feature Integrator. There, fusion occurs in a U-Net-like Siamese encoder-decoder at intermediate decoder scales through bidirectional cross-modal attention, aggregation of complementary and local features, and adaptive invariance gating (Wang et al., 24 Mar 2026). The gate emphasizes regions that remain important under complementary masking, identical-region masking, and modality dropout. This moves the fusion objective away from raw statistical correlation and toward intervention-stable dependencies (Wang et al., 24 Mar 2026).

Query-level fusion extends the same idea from feature maps to object hypotheses. In the Modality-Decoupled RGB-T detector, two independent DETR-like branches exchange high-confidence queries and proposals after each decoder stage. Query selection uses top-$h = \mathscr{F}(h_1, h_2, \ldots, h_n)$7 confidence filtering, and query adaptation uses stage-specific MLP adapters so that imported RGB and TIR queries can be reused without destroying branch independence (Tian et al., 13 Jan 2026). Intermediate fusion, in this setting, occurs neither at the pixel level nor only at the final output, but in the internal proposal-refinement loop.

More abstractly, Meta Fusion treats conventional intermediate fusion as one member of a larger cohort of models built from different combinations of latent representations across modalities (Liang et al., 27 Jul 2025). For two modalities, each student takes $h = \mathscr{F}(h_1, h_2, \ldots, h_n)$8, and “true” intermediate-fusion constructions are precisely those in which both inputs are learned latent representations rather than raw features or null mappings. Its student loss,

$h = \mathscr{F}(h_1, h_2, \ldots, h_n)$9

shows that intermediate fusion can also be understood as a space of latent-combination learners coupled by soft information sharing rather than as a single fixed architecture (Liang et al., 27 Jul 2025).

4. Representative application domains

Intermediate-modality fusion has been used across recognition, image synthesis, image fusion, remote sensing, object detection, and biomedical prediction. CentralNet validates multi-layer fusion on Multimodal MNIST, Audiovisual MNIST, Montalbano, and MM-IMDb, and reports that it consistently improves the accuracy of existing multimodal fusion approaches on these tasks (Vielzeuf et al., 2018). In one reported audiovisual setting, CentralNet reaches $y = f(h)$0, compared with $y = f(h)$1 for weighted mean and $y = f(h)$2 for GMU (Vielzeuf et al., 2018).

Infrared-visible image fusion has become a particularly dense site of intermediate-fusion research. MMA-UNet reports best SSIM $y = f(h)$3, best PSNR $y = f(h)$4, and best $y = f(h)$5 $y = f(h)$6 on M3FD, while also reporting best SSIM $y = f(h)$7, best VIF $y = f(h)$8, and best $y = f(h)$9 $x=\mathscr{F}(x_1,\ldots,x_n)$0 on MSRS (Huang et al., 2024). FusionNet, evaluated on M3FD, is implemented in PyTorch, uses Adam with learning rate $x=\mathscr{F}(x_1,\ldots,x_n)$1, batch size $x=\mathscr{F}(x_1,\ldots,x_n)$2, images resized to $x=\mathscr{F}(x_1,\ldots,x_n)$3, and has about $x=\mathscr{F}(x_1,\ldots,x_n)$4M parameters with runtime over $x=\mathscr{F}(x_1,\ldots,x_n)$5 FPS on an NVIDIA RTX 4080 (Sun et al., 14 Sep 2025). The intervention-stable framework reports SOTA performance on TNO, MSRS, and M$x=\mathscr{F}(x_1,\ldots,x_n)$6FD and further generalizes to object detection, semantic segmentation, and medical image fusion (Wang et al., 24 Mar 2026).

Object detection provides several distinct intermediate-fusion instantiations. FMCAF performs frequency-domain filtering of RGB and IR, then cross-attention-based fusion before YOLOv11, and reports $x=\mathscr{F}(x_1,\ldots,x_n)$7 mAP@50 over concatenation on VEDAI and $x=\mathscr{F}(x_1,\ldots,x_n)$8 on LLVIP (Berjawi et al., 20 Oct 2025). MDQF intersperses query fusion between RGB and TIR DETR-like decoders and achieves $x=\mathscr{F}(x_1,\ldots,x_n)$9 mAP and $y=f(x)$0 mAP50 on FLIR, as well as $y=f(x)$1 mAP and $y=f(x)$2 mAP50 on M3FD (Tian et al., 13 Jan 2026). VLC Fusion places CBAM-based feature fusion and FiLM conditioning between modality encoders and detector heads, using VLM-derived condition vectors such as nighttime, rain, fog or mist, lens flare, and blur-like degradation to reweight fused features (Taparia et al., 19 May 2025).

Remote sensing and hyperspectral classification also employ explicitly structured intermediate fusion. The Interconnected Fusion framework achieves OA $y=f(x)$3, AA $y=f(x)$4, and Kappa $y=f(x)$5 on Houston; OA $y=f(x)$6, AA $y=f(x)$7, and Kappa $y=f(x)$8 on MUUFL; and OA $y=f(x)$9, AA $h_i=f_i(x_i)$0, and Kappa $h_i=f_i(x_i)$1 on Trento (Huo et al., 2023). The paper attributes these results to modeling intra-modality structure, cross-modality dependencies, and center-patch context simultaneously.

Intermediate fusion is not confined to sensor fusion. In diffusion-based text-to-image generation, the intermediate-fusion U-ViT delays text injection until the image stream has reached a more semantic bottleneck representation. On MS-COCO, the cross-attention variant improves from FID $h_i=f_i(x_i)$2 and CLIP $h_i=f_i(x_i)$3 under early fusion to FID $h_i=f_i(x_i)$4 and CLIP $h_i=f_i(x_i)$5 under intermediate fusion, while also reducing GFLOPs from $h_i=f_i(x_i)$6 to $h_i=f_i(x_i)$7 and increasing training speed from $h_i=f_i(x_i)$8 to $h_i=f_i(x_i)$9 iters/s (Hu et al., 2024). The paper’s headline summary is about $h=\mathscr{F}(h_1,\ldots,h_n)$0 reduced FLOPs and $h=\mathscr{F}(h_1,\ldots,h_n)$1 increased training speed compared to a strong early-fusion U-ViT baseline (Hu et al., 2024).

5. Interpretability, adaptive weighting, and robustness

A major attraction of intermediate fusion is that many implementations expose internal allocation variables rather than hiding fusion in a single concatenated tensor. CentralNet initializes its fusion weights uniformly so that training starts from an average, and the learned $h=\mathscr{F}(h_1,\ldots,h_n)$2 values then reveal whether the model behaves more like early fusion, late fusion, or a hybrid across depths (Vielzeuf et al., 2018). On MM-MNIST, low layers rely more on modality inputs while deeper layers rely more on the central hidden state; on MM-IMDb, the model leans toward earlier fusion but still uses both modalities throughout (Vielzeuf et al., 2018).

FusionNet makes interpretability even more direct. Its feature-level attention mask determines whether the fused intermediate representation leans toward infrared or visible features at each location, and its pixel-wise alpha map provides a spatially varying convex combination for the final image (Sun et al., 14 Sep 2025). The paper explicitly describes the alpha map as interpretable: brighter regions correspond to greater infrared contribution, darker regions to greater visible contribution. This places intermediate fusion and output synthesis on the same explanatory continuum.

Robustness-oriented work tends to convert intermediate fusion into a problem of selective trust. The intervention-stable framework states that large gate values favor complementary cross-modal features and small gate values favor local modality-specific features, with gate activation guided by behavior under perturbation (Wang et al., 24 Mar 2026). MDQF uses confidence-based filtering so that low-confidence queries from a degraded modality are excluded from propagation, while high-quality queries from the other modality correct the next refinement stage (Tian et al., 13 Jan 2026). VLC Fusion applies the same principle at the scene-conditioning level: the fusion block is generic, but a FiLM layer modulates the fused tensor according to VLM-derived boolean responses about the environment (Taparia et al., 19 May 2025).

These patterns suggest that intermediate-modality fusion has increasingly been formulated as adaptive reliability allocation. Rather than merely merging features, modern architectures typically learn where one modality should dominate, where a second modality should compensate, and where the model should preserve modality-specific structure instead of collapsing it into a shared state. That interpretation is consistent with the review literature, which highlights attention, calibration, and knowledge-sharing as the main directions beyond simple concatenation (Guarrasi et al., 2024).

6. Critiques, limitations, and research directions

Intermediate-modality fusion is not treated as universally optimal in the literature. Fuse4Seg explicitly argues against feature-level fusion for multi-modality medical image segmentation, claiming that existing methods are prone to semantic inconsistencies and misalignments because they merge features at intermediate layers “without evaluative control” (Guo et al., 2024). Its alternative is image-level fusion before segmentation, coupled through a bi-level learning framework, and the paper states that this avoids misalignment and semantic displacement because “the pixel is matched scale for the inputs and corresponding output” (Guo et al., 2024). This is a direct challenge to the assumption that richer hidden-level interaction is always preferable.

A second critique concerns optimization under modality imbalance. “Beyond Modality Fusion: Deep Ensembles for Multimodal Classification” reports that deep ensembles of unimodal networks consistently outperform state-of-the-art late-fusion methods, the intermediate-fusion techniques evaluated there, and hybrid methods that combine unimodal and multimodal predictions (Burenko et al., 6 Jul 2026). The paper’s explanation is that under modality imbalance a fusion network can “underexplore” the slower-converging modality, whereas independent unimodal optimization plus logit averaging avoids this interference (Burenko et al., 6 Jul 2026). In that setting, explicit fusion is not merely unnecessary; it can be a liability.

Even when intermediate fusion is effective, limitations recur. CentralNet notes that when modalities are truly independent or share little information, the learned joint representation may be harder to stabilize, and in some such cases a simple early-fusion ensemble can be competitive or even slightly better (Vielzeuf et al., 2018). The biomedical reviews emphasize additional difficulties: missing modalities, computational cost, black-box behavior, data scarcity, unclear design choices for representation dimensionality, and underexplored multimodal module design (Guarrasi et al., 2024). The 2020 medical segmentation review likewise frames the central challenge as how to fuse different modalities to get the best compromise for precise segmentation (Zhou et al., 2020).

Current research directions therefore move in two partly divergent directions. One direction seeks stronger intermediate fusion through more explicit structure: intervention-based stability, environmental conditioning, semantic pruning, query exchange, and cross-scale alignment (Wang et al., 24 Mar 2026). The other direction reframes the design space itself. Meta Fusion treats early, intermediate, and late fusion as special cases of a broader latent-combination framework, while the systematic review calls for more sophisticated and theoretically grounded fusion operators, greater robustness to missing modalities, stronger explainability, and more rigorous benchmark practice (Liang et al., 27 Jul 2025). A plausible implication is that the future of intermediate-modality fusion will be less about choosing one fixed fusion point and more about learning when, where, and whether internal cross-modal interaction is beneficial for a particular modality pair, task, and operating regime.