---
title: Progressive Intra-Layer Fusion
url: https://www.emergentmind.com/topics/progressive-intra-layer-fusion
type: topic
---

# Progressive Intra-Layer Fusion

Searching arXiv for the primary paper and closely related work on progressive fusion to ground the article.
Progressive Intra-Layer Fusion is a multimodal fusion paradigm in which cross-modal interaction is embedded inside the computation of an encoder layer and repeated across encoder depth, so that fusion becomes part of representation learning rather than a separate downstream aggregation step. In PGF-Net, the term is defined as a shift “from parallel routing to serial, intra-layer fusion,” where non-linguistic information is not fused only once at the input or output, but is instead progressively injected into the textual stream as the representation evolves across the deep layers of a Transformer encoder [2508.15852]. Related papers use “progressive fusion” for repeated fusion across blocks, stages, or feedback steps, but not always in this strict intra-layer sense, including repeated exposure-specific fusion blocks for HDR imaging [2108.08585], iterative late-to-early multimodal feedback [2209.00302], same-layer RGB-TIR fusion followed by all-layer interaction [2408.08827], and multi-stage BEV/PV fusion followed by query-level fusion for 3D detection [2410.07475].

## 1. Definitional core

In PGF-Net, Progressive Intra-Layer Fusion is the central architectural idea of the hierarchical multimodal encoder. The encoder preserves the standard text self-attention and feed-forward structure, but from a configurable starting depth \(L_0\), each layer is augmented with a Cross-Attention Gated Fusion Layer. Layers with \(l < L_0\) are standard text-only Transformer layers, while for all \(l \geq L_0\) the same fusion sequence is executed within each layer [2508.15852].

The term “intra-layer” is specific. Fusion does not happen only between encoder stages or after the whole Transformer stack; it is integrated into the computation path of an individual encoder layer. The layer first contextualizes text through self-attention, then performs cross-modal retrieval, then adaptively mixes text and multimodal context, and finally refines the fused representation with an adapter. The term “progressive” is equally specific: the same logic is applied at multiple depths, so the textual state at layer \(l\) already contains multimodal information injected at earlier layers \(< l\), and can therefore query audio-visual content again in a more semantically mature way. The paper describes this as a “continuous, hierarchical process” rather than a “single, discrete fusion event” [2508.15852].

In the reported experiments, fusion begins at the very first BERT layer, \(L_0 = 0\), so multimodal fusion is applied through the entire encoder depth. This makes the method simultaneously layer-internal and depth-progressive [2508.15852].

## 2. Architectural realization in PGF-Net

PGF-Net is organized into three parts: multimodal input representation, a hierarchical multimodal encoder, and a lightweight regression/classification head. Progressive Intra-Layer Fusion resides entirely in the hierarchical multimodal encoder [2508.15852].

The model assumes three time-aligned sequences: a tokenized text sequence \(T\), an acoustic sequence \(A\), and a visual sequence \(V\). Text is processed by the standard BERT tokenizer into `input_ids`, `attention_mask`, and `token_type_ids`, while audio and visual streams are provided as preprocessed descriptor sequences. The acoustic descriptors encode prosodic and spectral information; the visual descriptors reflect facial actions, pose, and related cues. Unlike token-wise synchronous fusion schemes, PGF-Net does not merge audio and visual features one-by-one with each token. Instead, after projection and concatenation, they form a multimodal feature bank that text can attend over [2508.15852].

Before fusion, acoustic and visual features are separately projected into the BERT hidden dimension \(D_{\text{bert}}\):

$$
X'_a = X_a W_a + b_a \; ; \; X'_v = X_v W_v + b_v
\tag{2}
$$

They are then concatenated along the sequence dimension:

$$
H_{av} = \mathrm{Concat}(X'_a, X'_v)
\tag{3}
$$

Within a fusion-enabled layer, the previous text hidden state \(H_{l-1}\) is first contextualized by self-attention:

$$
H_{\text{text}} = \mathrm{Self\mbox{-}Attention}(H_{l-1})
\tag{1}
$$

Cross-modal retrieval is then performed by using text as the query and the concatenated audio-visual bank as key and value:

$$
H_{\text{cross}} = \mathrm{Cross\mbox{-}Attention}(Q = H_{\text{text}}, K = H_{av}, V = H_{av})
\tag{4}
$$

This design makes the query modality text and the key/value modality the unified audio-visual memory. The paper explicitly interprets this as an information extraction routine in which each text token actively extracts the most relevant non-linguistic evidence from the multimodal repository [2508.15852].

After retrieval, PGF-Net applies Adaptive Gated Arbitration rather than direct addition. The gate is computed from the concatenation of the contextualized text and the cross-attended output:

$$
g = \sigma \big( W_g [H_{\text{text}} ; H_{\text{cross}}] + b_g \big)
\tag{5}
$$

The fused representation is then

$$
H_{\text{fused}} = g \odot H_{\text{text}} + (1-g) \odot H_{\text{cross}}
\tag{6}
$$

where \([\, ; \,]\) denotes concatenation, \(\sigma\) is the sigmoid, and \(\odot\) is element-wise multiplication. If \(g \to 1\), the representation remains close to text; if \(g \to 0\), it moves toward the cross-attended multimodal context [2508.15852].

A Post-Fusion Adapter then refines \(H_{\text{fused}}\) to produce the layer output \(H_l\). The paper describes this adapter as a bottleneck-structured network in the spirit of Houlsby adapters, although it does not give an explicit adapter equation [2508.15852].

## 3. Operational sequence, implementation constraints, and parameter-efficient adaptation

The per-layer fusion logic in PGF-Net has a fixed operational sequence. First, self-attention produces \(H_{\text{text}}\). Second, audio and visual sequences are projected to the BERT hidden dimension and concatenated into \(H_{av}\). Third, cross-attention with \(Q = H_{\text{text}}\) and \(K = V = H_{av}\) produces \(H_{\text{cross}}\). Fourth, the adaptive gate combines \(H_{\text{text}}\) and \(H_{\text{cross}}\) into \(H_{\text{fused}}\). Fifth, the post-fusion adapter refines the result to produce \(H_l\). This same sequence is repeated for every encoder layer from \(L_0\) onward [2508.15852].

Several reproduction constraints are explicit. The backbone is `bert-base-uncased`, so the text hidden size is 768 and therefore \(D_{\text{bert}} = 768\). The projected acoustic and visual features must each be mapped into 768-dimensional embeddings before concatenation. LoRA is applied to the BERT backbone, specifically to the query and value projections in attention, with rank 32. The Post-Fusion Adapter bottleneck dimension is 64. Batch size is 128, the optimizer is AdamW, the learning rate is \(10^{-3}\), weight decay is \(10^{-4}\), StepLR uses `step_size = 9` and \(\gamma = 0.1\), and training runs for up to 20 epochs with early stopping patience 10 [2508.15852].

The paper also states several nontrivial absences. It does not expand the cross-attention into the full scaled dot-product formula with explicit \(W_Q, W_K, W_V\), head partitioning, or residual/LayerNorm equations. It does not explicitly state the number of heads used in the added cross-attention module, and it does not report a custom dropout rate inside the fusion block or explicit normalization modifications. Accordingly, self-attention, residual connections, normalization, and feed-forward ordering should be treated as inherited from the underlying BERT/Transformer implementation, while the paper-specific specification is concentrated in Equations (1)–(6) [2508.15852].

A plausible implication is that the main novelty is not a modified Transformer algebra, but the insertion of a repeated cross-attention–gate–adapter sequence into the internal computation path of each selected encoder layer.

## 4. Relation to adjacent progressive-fusion formulations

The literature uses the label “progressive fusion” for several related but distinct mechanisms. PSFNet for HDR imaging defines progressive fusion as multi-step feature fusion over a stack of six repeated PSFBs, where three exposure-specific streams are repeatedly fused, compared against a fused consensus, and updated residually. The paper explicitly notes that this is better described as progressive fusion across repeated stages/blocks and across exposure-specific feature streams, not intra-layer fusion in the strict sense [2108.08585].

“Progressive Fusion for Multimodal Integration” defines progression differently: a fused multimodal context vector is sent backward to modality-specific encoders through iterative unrolling. The method is therefore iterative cross-layer feedback fusion rather than same-depth intra-layer fusion. Its own language is “backprojective connections,” “backward connections,” and “iterative representation refinement” [2209.00302].

AINet for RGBT tracking is closer to a two-level progressive scheme. It performs same-layer RGB/TIR fusion after every ViT block using a Difference-based Fusion Mamba, then concatenates the fused outputs of all layers and performs all-layer interaction with an Order-dynamic Fusion Mamba. In that formulation, fusion is progressive because it begins with local same-layer fusion and is then elevated to all-layer interaction [2408.08827].

Other papers are explicitly adjacent rather than exact. GPF-Net for polyp re-identification uses a 4-layer gated progressive fusion network for repeated image-text fusion, but the paper does not formalize fusion within a single backbone layer or across a clearly defined multiscale feature hierarchy; it is better described as progressive cross-modal layer-wise fusion [2512.21476]. IAFI-FCOS introduces an intra-layer context augmentation block and across-layer feature weighting, but its ICA block is a single-shot intra-layer enhancement rather than a clearly progressive same-level refinement process [2409.00694]. ProFusion3D performs progressive fusion across architectural stages—cross-view mapping, intermediate BEV/PV fusion, and query-level fusion—rather than repeated fusion throughout backbone depth [2410.07475]. DCFG for thermal infrared tracking is closest to progressive cross-channel intra-block fusion with multi-level Siamese fusion, because its layer-wise combination unit progressively extracts and merges feature groups inside the fine-grained block and then combines Conv3/Conv4/Conv5 outputs by weighted fusion [2504.14311].

This suggests that “Progressive Intra-Layer Fusion” should be reserved for architectures in which the fusion routine is both layer-internal and repeatedly executed across depth, while broader “progressive fusion” includes block-wise refinement, feedback refinement, cross-layer aggregation, and staged multi-view fusion.

## 5. Empirical profile, interpretability, and trade-offs

On MOSI, full PGF-Net reports **MAE 0.691, Corr 0.809, Acc-7 49.4, Acc-2 86.8, F1 86.9**. The strongest ablation evidence for Progressive Intra-Layer Fusion comes from removing cross-attention: the “w/o CA” variant degrades to **MAE 0.725, Corr 0.783, Acc-7 45.8, Acc-2 84.5, F1 84.6**. Removing the gate also degrades performance, though less severely, to **MAE 0.710, Corr 0.796, Acc-7 47.5, Acc-2 85.7, F1 85.8**. Removing both cross-attention and gate yields **Acc-7 45.2** and **F1 84.0**. The paper explicitly interprets this as evidence that cross-attention-based fusion is the cornerstone of effective modality integration and that adaptive arbitration materially improves fusion quality [2508.15852].

The comparison to MMA is the main empirical argument against flat or one-shot routing. On MOSI, PGF-Net reports **MAE 0.691 vs 0.693**, **Corr 0.809 vs 0.803**, **Acc-7 49.4 vs 46.9**, **Acc-2 86.8 vs 86.4**, and **F1 86.9 vs 86.4**, with **3.09M vs 5.7M** trainable parameters. On MOSEI, it again improves over MMA, particularly in **Acc-7 58.5 vs 55.2** and **F1 86.1 vs 85.7**, with about half the trainable parameters [2508.15852].

Efficiency is central to the formulation. Progressive Intra-Layer Fusion introduces repeated cross-attention between the text sequence and the concatenated audio-visual bank at every fusion-enabled layer, so relative to a text-only Transformer it increases per-layer cost by attending over \(T_a + T_v\) auxiliary tokens. However, the paper argues that the full model remains lightweight because most backbone parameters are frozen under the PEFT strategy, with trainable adaptation concentrated in LoRA matrices, gates, projections, and adapters. PGF-Net reports **3.09M trainable parameters on MOSI** and **4.72M on MOSEI** [2508.15852].

The interpretability claim is architectural rather than visualization-based. The mechanism is transparent in the sense that text first contextualizes itself, then explicitly queries multimodal memory, then a gate decides how much of that memory to trust. At the same time, the paper does not provide explicit attention visualizations, layer-wise heatmaps, or token-level case studies of the progressive fusion process. Its interpretability argument is therefore conceptual and indirect, supported partly by gains in **7-class accuracy**, which the authors attribute to better capture of nuanced emotional cues [2508.15852].

The paper also states several limitations. On the more complex MOSEI dataset, MAE and Pearson correlation are competitive but not always best-in-class. Fusion at every selected layer likely incurs more compute than a single-step late-fusion design, although FLOPs are not quantified. The need for the adaptive gate itself reflects a limitation of repeated cross-modal injection: without gating, noisy or irrelevant non-linguistic signals can corrupt the text representation [2508.15852].

## 6. Research significance and open directions

The principal significance of Progressive Intra-Layer Fusion is the relocation of multimodal fusion from a terminal aggregation step into the internal dynamics of encoder depth. In PGF-Net, this means that textual representations do not merely receive multimodal information once; they repeatedly re-query the audio-visual bank as semantic abstraction increases, with adaptive arbitration protecting the linguistic stream from noisy evidence [2508.15852].

Within the broader literature, related architectures show that “progression” can be instantiated in multiple ways: repeated block-wise refinement in PSFNet [2108.08585], backward multimodal feedback in Pro-Fusion [2209.00302], same-layer plus all-layer interaction in AINet [2408.08827], or multi-stage BEV/PV and query-level fusion in ProFusion3D [2410.07475]. What distinguishes the strict intra-layer form is that the fusion routine is embedded inside the layer computation itself and then repeated across depth.

The open problems stated for PGF-Net concern noise suppression, longer sequences, dynamic emotional shifts, and heterogeneous or depth-varying adapters [2508.15852]. A plausible implication is that future work will continue to treat progressive fusion not as a single operator but as a design space defined by where fusion is inserted, how often it is repeated, what modality serves as query, and how strongly the model constrains multimodal evidence before it can alter the evolving hidden state.

Source: https://www.emergentmind.com/topics/progressive-intra-layer-fusion