Progressive Intra-Layer Fusion

• The paper introduces Progressive Intra-Layer Fusion in PGF-Net by embedding cross-modal interactions directly within each encoder layer, which progressively refines the text representation.
• It implements a gated cross-attention mechanism that adaptively combines textual and multimodal features, significantly enhancing fusion quality and reducing parameter overhead.
• Empirical results on MOSI and MOSEI benchmarks show improved accuracy and correlation metrics, demonstrating that repeated in-layer fusion outperforms traditional one-shot aggregation methods.

Searching arXiv for the primary paper and closely related work on progressive fusion to ground the article. Progressive Intra-Layer Fusion is a multimodal fusion paradigm in which cross-modal interaction is embedded inside the computation of an encoder layer and repeated across encoder depth, so that fusion becomes part of representation learning rather than a separate downstream aggregation step. In PGF-Net, the term is defined as a shift “from parallel routing to serial, intra-layer fusion,” where non-linguistic information is not fused only once at the input or output, but is instead progressively injected into the textual stream as the representation evolves across the deep layers of a Transformer encoder (Wen et al., 20 Aug 2025). Related papers use “progressive fusion” for repeated fusion across blocks, stages, or feedback steps, but not always in this strict intra-layer sense, including repeated exposure-specific fusion blocks for HDR imaging (Ye et al., 2021), iterative late-to-early multimodal feedback (Shankar et al., 2022), same-layer RGB-TIR fusion followed by all-layer interaction (Lu et al., 2024), and multi-stage BEV/PV fusion followed by query-level fusion for 3D detection (Mohan et al., 2024).

1. Definitional core

In PGF-Net, Progressive Intra-Layer Fusion is the central architectural idea of the hierarchical multimodal encoder. The encoder preserves the standard text self-attention and feed-forward structure, but from a configurable starting depth $L_0$, each layer is augmented with a Cross-Attention Gated Fusion Layer. Layers with $l < L_0$ are standard text-only Transformer layers, while for all $l \geq L_0$ the same fusion sequence is executed within each layer (Wen et al., 20 Aug 2025).

The term “intra-layer” is specific. Fusion does not happen only between encoder stages or after the whole Transformer stack; it is integrated into the computation path of an individual encoder layer. The layer first contextualizes text through self-attention, then performs cross-modal retrieval, then adaptively mixes text and multimodal context, and finally refines the fused representation with an adapter. The term “progressive” is equally specific: the same logic is applied at multiple depths, so the textual state at layer $l$ already contains multimodal information injected at earlier layers $< l$, and can therefore query audio-visual content again in a more semantically mature way. The paper describes this as a “continuous, hierarchical process” rather than a “single, discrete fusion event” (Wen et al., 20 Aug 2025).

In the reported experiments, fusion begins at the very first BERT layer, $L_0 = 0$, so multimodal fusion is applied through the entire encoder depth. This makes the method simultaneously layer-internal and depth-progressive (Wen et al., 20 Aug 2025).

2. Architectural realization in PGF-Net

PGF-Net is organized into three parts: multimodal input representation, a hierarchical multimodal encoder, and a lightweight regression/classification head. Progressive Intra-Layer Fusion resides entirely in the hierarchical multimodal encoder (Wen et al., 20 Aug 2025).

The model assumes three time-aligned sequences: a tokenized text sequence $T$, an acoustic sequence $A$, and a visual sequence $V$. Text is processed by the standard BERT tokenizer into input_ids, attention_mask, and token_type_ids, while audio and visual streams are provided as preprocessed descriptor sequences. The acoustic descriptors encode prosodic and spectral information; the visual descriptors reflect facial actions, pose, and related cues. Unlike token-wise synchronous fusion schemes, PGF-Net does not merge audio and visual features one-by-one with each token. Instead, after projection and concatenation, they form a multimodal feature bank that text can attend over (Wen et al., 20 Aug 2025).

Before fusion, acoustic and visual features are separately projected into the BERT hidden dimension $D_{\text{bert}}$:

$l < L_0$0

They are then concatenated along the sequence dimension:

$l < L_0$1

Within a fusion-enabled layer, the previous text hidden state $l < L_0$2 is first contextualized by self-attention:

$l < L_0$3

Cross-modal retrieval is then performed by using text as the query and the concatenated audio-visual bank as key and value:

$l < L_0$4

This design makes the query modality text and the key/value modality the unified audio-visual memory. The paper explicitly interprets this as an information extraction routine in which each text token actively extracts the most relevant non-linguistic evidence from the multimodal repository (Wen et al., 20 Aug 2025).

After retrieval, PGF-Net applies Adaptive Gated Arbitration rather than direct addition. The gate is computed from the concatenation of the contextualized text and the cross-attended output:

$l < L_0$5

The fused representation is then

$l < L_0$6

where $l < L_0$7 denotes concatenation, $l < L_0$8 is the sigmoid, and $l < L_0$9 is element-wise multiplication. If $l \geq L_0$0, the representation remains close to text; if $l \geq L_0$1, it moves toward the cross-attended multimodal context (Wen et al., 20 Aug 2025).

A Post-Fusion Adapter then refines $l \geq L_0$2 to produce the layer output $l \geq L_0$3. The paper describes this adapter as a bottleneck-structured network in the spirit of Houlsby adapters, although it does not give an explicit adapter equation (Wen et al., 20 Aug 2025).

3. Operational sequence, implementation constraints, and parameter-efficient adaptation

The per-layer fusion logic in PGF-Net has a fixed operational sequence. First, self-attention produces $l \geq L_0$4. Second, audio and visual sequences are projected to the BERT hidden dimension and concatenated into $l \geq L_0$5. Third, cross-attention with $l \geq L_0$6 and $l \geq L_0$7 produces $l \geq L_0$8. Fourth, the adaptive gate combines $l \geq L_0$9 and $l$0 into $l$1. Fifth, the post-fusion adapter refines the result to produce $l$2. This same sequence is repeated for every encoder layer from $l$3 onward (Wen et al., 20 Aug 2025).

Several reproduction constraints are explicit. The backbone is bert-base-uncased, so the text hidden size is 768 and therefore $l$4. The projected acoustic and visual features must each be mapped into 768-dimensional embeddings before concatenation. LoRA is applied to the BERT backbone, specifically to the query and value projections in attention, with rank 32. The Post-Fusion Adapter bottleneck dimension is 64. Batch size is 128, the optimizer is AdamW, the learning rate is $l$5, weight decay is $l$6, StepLR uses step_size = 9 and $l$7, and training runs for up to 20 epochs with early stopping patience 10 (Wen et al., 20 Aug 2025).

The paper also states several nontrivial absences. It does not expand the cross-attention into the full scaled dot-product formula with explicit $l$8, head partitioning, or residual/LayerNorm equations. It does not explicitly state the number of heads used in the added cross-attention module, and it does not report a custom dropout rate inside the fusion block or explicit normalization modifications. Accordingly, self-attention, residual connections, normalization, and feed-forward ordering should be treated as inherited from the underlying BERT/Transformer implementation, while the paper-specific specification is concentrated in Equations (1)–(6) (Wen et al., 20 Aug 2025).

A plausible implication is that the main novelty is not a modified Transformer algebra, but the insertion of a repeated cross-attention–gate–adapter sequence into the internal computation path of each selected encoder layer.

4. Relation to adjacent progressive-fusion formulations

The literature uses the label “progressive fusion” for several related but distinct mechanisms. PSFNet for HDR imaging defines progressive fusion as multi-step feature fusion over a stack of six repeated PSFBs, where three exposure-specific streams are repeatedly fused, compared against a fused consensus, and updated residually. The paper explicitly notes that this is better described as progressive fusion across repeated stages/blocks and across exposure-specific feature streams, not intra-layer fusion in the strict sense (Ye et al., 2021).

“Progressive Fusion for Multimodal Integration” defines progression differently: a fused multimodal context vector is sent backward to modality-specific encoders through iterative unrolling. The method is therefore iterative cross-layer feedback fusion rather than same-depth intra-layer fusion. Its own language is “backprojective connections,” “backward connections,” and “iterative representation refinement” (Shankar et al., 2022).

AINet for RGBT tracking is closer to a two-level progressive scheme. It performs same-layer RGB/TIR fusion after every ViT block using a Difference-based Fusion Mamba, then concatenates the fused outputs of all layers and performs all-layer interaction with an Order-dynamic Fusion Mamba. In that formulation, fusion is progressive because it begins with local same-layer fusion and is then elevated to all-layer interaction (Lu et al., 2024).

Other papers are explicitly adjacent rather than exact. GPF-Net for polyp re-identification uses a 4-layer gated progressive fusion network for repeated image-text fusion, but the paper does not formalize fusion within a single backbone layer or across a clearly defined multiscale feature hierarchy; it is better described as progressive cross-modal layer-wise fusion (Xiang et al., 25 Dec 2025). IAFI-FCOS introduces an intra-layer context augmentation block and across-layer feature weighting, but its ICA block is a single-shot intra-layer enhancement rather than a clearly progressive same-level refinement process (Guan et al., 2024). ProFusion3D performs progressive fusion across architectural stages—cross-view mapping, intermediate BEV/PV fusion, and query-level fusion—rather than repeated fusion throughout backbone depth (Mohan et al., 2024). DCFG for thermal infrared tracking is closest to progressive cross-channel intra-block fusion with multi-level Siamese fusion, because its layer-wise combination unit progressively extracts and merges feature groups inside the fine-grained block and then combines Conv3/Conv4/Conv5 outputs by weighted fusion (Xiong et al., 19 Apr 2025).

This suggests that “Progressive Intra-Layer Fusion” should be reserved for architectures in which the fusion routine is both layer-internal and repeatedly executed across depth, while broader “progressive fusion” includes block-wise refinement, feedback refinement, cross-layer aggregation, and staged multi-view fusion.

5. Empirical profile, interpretability, and trade-offs

On MOSI, full PGF-Net reports MAE 0.691, Corr 0.809, Acc-7 49.4, Acc-2 86.8, F1 86.9. The strongest ablation evidence for Progressive Intra-Layer Fusion comes from removing cross-attention: the “w/o CA” variant degrades to MAE 0.725, Corr 0.783, Acc-7 45.8, Acc-2 84.5, F1 84.6. Removing the gate also degrades performance, though less severely, to MAE 0.710, Corr 0.796, Acc-7 47.5, Acc-2 85.7, F1 85.8. Removing both cross-attention and gate yields Acc-7 45.2 and F1 84.0. The paper explicitly interprets this as evidence that cross-attention-based fusion is the cornerstone of effective modality integration and that adaptive arbitration materially improves fusion quality (Wen et al., 20 Aug 2025).

The comparison to MMA is the main empirical argument against flat or one-shot routing. On MOSI, PGF-Net reports MAE 0.691 vs 0.693, Corr 0.809 vs 0.803, Acc-7 49.4 vs 46.9, Acc-2 86.8 vs 86.4, and F1 86.9 vs 86.4, with 3.09M vs 5.7M trainable parameters. On MOSEI, it again improves over MMA, particularly in Acc-7 58.5 vs 55.2 and F1 86.1 vs 85.7, with about half the trainable parameters (Wen et al., 20 Aug 2025).

Efficiency is central to the formulation. Progressive Intra-Layer Fusion introduces repeated cross-attention between the text sequence and the concatenated audio-visual bank at every fusion-enabled layer, so relative to a text-only Transformer it increases per-layer cost by attending over $l$9 auxiliary tokens. However, the paper argues that the full model remains lightweight because most backbone parameters are frozen under the PEFT strategy, with trainable adaptation concentrated in LoRA matrices, gates, projections, and adapters. PGF-Net reports 3.09M trainable parameters on MOSI and 4.72M on MOSEI (Wen et al., 20 Aug 2025).

The interpretability claim is architectural rather than visualization-based. The mechanism is transparent in the sense that text first contextualizes itself, then explicitly queries multimodal memory, then a gate decides how much of that memory to trust. At the same time, the paper does not provide explicit attention visualizations, layer-wise heatmaps, or token-level case studies of the progressive fusion process. Its interpretability argument is therefore conceptual and indirect, supported partly by gains in 7-class accuracy, which the authors attribute to better capture of nuanced emotional cues (Wen et al., 20 Aug 2025).

The paper also states several limitations. On the more complex MOSEI dataset, MAE and Pearson correlation are competitive but not always best-in-class. Fusion at every selected layer likely incurs more compute than a single-step late-fusion design, although FLOPs are not quantified. The need for the adaptive gate itself reflects a limitation of repeated cross-modal injection: without gating, noisy or irrelevant non-linguistic signals can corrupt the text representation (Wen et al., 20 Aug 2025).

6. Research significance and open directions

The principal significance of Progressive Intra-Layer Fusion is the relocation of multimodal fusion from a terminal aggregation step into the internal dynamics of encoder depth. In PGF-Net, this means that textual representations do not merely receive multimodal information once; they repeatedly re-query the audio-visual bank as semantic abstraction increases, with adaptive arbitration protecting the linguistic stream from noisy evidence (Wen et al., 20 Aug 2025).

Within the broader literature, related architectures show that “progression” can be instantiated in multiple ways: repeated block-wise refinement in PSFNet (Ye et al., 2021), backward multimodal feedback in Pro-Fusion (Shankar et al., 2022), same-layer plus all-layer interaction in AINet (Lu et al., 2024), or multi-stage BEV/PV and query-level fusion in ProFusion3D (Mohan et al., 2024). What distinguishes the strict intra-layer form is that the fusion routine is embedded inside the layer computation itself and then repeated across depth.

The open problems stated for PGF-Net concern noise suppression, longer sequences, dynamic emotional shifts, and heterogeneous or depth-varying adapters (Wen et al., 20 Aug 2025). A plausible implication is that future work will continue to treat progressive fusion not as a single operator but as a design space defined by where fusion is inserted, how often it is repeated, what modality serves as query, and how strongly the model constrains multimodal evidence before it can alter the evolving hidden state.