Frame-level Gated Fusion

• Frame-level Gated Fusion is an adaptive design pattern that applies learned gating at each frame to fuse features from multiple modalities.
• It leverages diverse formulations—such as multiplicative sigmoid masks and additive corrections—to enhance tasks like video restoration, captioning, and active speaker detection.
• Empirical studies show that context-conditioned gating improves performance metrics by selectively balancing temporal and cross-modal information.

Frame-level Gated Fusion (FGF) denotes a family of adaptive fusion mechanisms in which gating decisions are made at the granularity of frame-aligned representations rather than by a single clip-wide weight. Across the literature, the term covers several closely related patterns: gating between current and previous frame features in recurrent video restoration, gating between multiple frame-derived context streams in sequence decoding, and gating between synchronized modalities such as audio, RGB, optical flow, thermal imagery, or heterogeneous sensors at each frame or frame-aligned time step. What remains invariant is the central operation: a learned gate, conditioned on the compared features and sometimes on task context, modulates how much information from each source is retained, suppressed, or injected into the fused representation (Guo et al., 2024, Yang et al., 3 Aug 2025, Wang et al., 17 Dec 2025, Jin et al., 2023).

1. Conceptual scope and historical placement

FGF is not a single architecture but a design pattern. In the most explicit formulation, it is a frame-level, text-guided gate that blends audio and visual features for text–video retrieval, producing one gate per sampled frame (Yang et al., 3 Aug 2025). In recurrent video denoising, the same idea appears as frame-level dense feature gating: the current frame feature and a recurrent feature state from the previous frame are aligned, filtered by a reset gate, refined, and then merged through an update gate to form the next state (Guo et al., 2024). In active speaker detection, frame-level fusion occurs at each aligned audio–video time index and across multiple Transformer depths via bimodally conditioned gates (Wang et al., 17 Dec 2025). In dynamic saliency prediction, the gate is a spatially varying map computed for each frame, deciding how much of the appearance saliency map versus the motion saliency map should contribute at each location (Kocak et al., 2021).

Earlier sensor-fusion work established important precursors. “Optimized Gated Deep Learning Architectures for Sensor Fusion” introduced feature-level, group-level, and two-stage gating applied per temporal window of multimodal sensor data; the paper does not literally use the phrase Frame-level Gated Fusion, but its mechanisms are described as exactly of that type when interpreted at each time window (Shim et al., 2018). “GFD-SSD: Gated Fusion Double SSD for Multispectral Pedestrian Detection” similarly defines Gated Fusion Units (GFUs) for color–thermal fusion at feature-map level within a single frame; this is not temporal gating, but the paper’s details explicitly note that the mechanism is directly reusable for frame-level temporal fusion by treating “current frame vs previous frame” analogously to “color vs thermal” (Zheng et al., 2019).

A recurrent source of ambiguity is the meaning of “frame-level.” In some works it refers literally to one gate per frame-aligned token or sampled frame. In others it denotes dense gating over a full-frame feature tensor, hence per spatial location and per channel, still indexed by frame. In video captioning, the gate may fuse context vectors that were themselves obtained by attention over all frames, so the gate is not over raw frames but over frame-derived streams (Jin et al., 2023). This suggests that FGF is best understood as gating over representations whose temporal unit is the frame, rather than as a commitment to one particular tensor shape.

2. Core formulations and gating primitives

Despite architectural diversity, most FGF instantiations reduce to a small set of gating primitives.

Setting	Gated quantity	Representative formulation
Recurrent temporal fusion	Previous state vs current-frame candidate	$$F_B^{n} = W_{UG}^{n} \odot F_{TD}^{n} + (1 - W_{UG}^{n}) \odot F_{WB}^{n-1}$$
Text-guided audio–visual fusion	Audio vs visual feature at frame $i$	$v_i = g_i \cdot a_i + (1-g_i) \cdot f_i$
Decoder-time fusion of frame-derived contexts	Stream $X$ vs stream $Y$	$$C_t = \lambda_t \odot f(X_t) + (1-\lambda_t)\odot f(Y_t)$$
Hierarchical cross-modal injection	Context modality into primary modality	$$\tilde{f}_p \leftarrow \mathrm{LN}(\tilde{f}_p + g^l \odot \tilde{h}_c^l)$$
Dynamic saliency fusion	Appearance map vs motion map	$S_{\text{final}} = S_A \odot G_A + S_T \odot G_T$

In “A Practical Gated Recurrent Transformer Network Incorporating Multiple Fusions for Video Denoising,” the gate is dense in space and channels. The reset gate is computed as

$$W_{RG}^{n} = H_{RG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$

and selects which parts of the warped previous-frame state remain relevant before temporal denoising. The update gate,

$$W_{UG}^{n} = H_{UG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$

then controls interpolation between the candidate temporal feature and the previous warped state. Both gates are produced by four convolution layers with sigmoid at the end, with the first two convolutions grouped with group $i$0, and yield per-channel, per-spatial-location values in $i$1 (Guo et al., 2024).

In GAID, the gate is scalar per sampled frame: $i$2 Here the gate is conditioned jointly on the visual feature, the temporally aligned audio feature, and the global text embedding, so the frame-wise balance between modalities is query-dependent (Yang et al., 3 Aug 2025).

In the dual-graph video captioning model, the generic gate is vector-valued and conditioned on decoder state: $i$3 The same primitive is instantiated hierarchically: first between GAT- and FORG-derived appearance contexts, then between the corresponding motion contexts, and finally between appearance and motion (Jin et al., 2023).

A different but related formulation appears in GateFusion’s HiGate: $i$4 This is not a convex combination between two peers; it is a gated residual injection from a context modality into a primary modality, repeated at selected encoder depths $i$5 after temporal alignment (Wang et al., 17 Dec 2025).

FGF does not always use multiplicative sigmoid masks. GFD-SSD’s GFU variants use additive adjustments with ReLU: $i$6 followed by fusion through a $i$7 convolution. The paper explicitly contrasts this with sigmoid-and-product gating and interprets the units as learned feature increments or corrections rather than bounded attention masks (Zheng et al., 2019). This suggests that “gated fusion” in the literature is broader than the strict sigmoid-mask archetype.

3. Recurrent and temporally structured FGF

The most temporally explicit FGF instance in the provided literature is the gated recurrent Transformer network for video denoising. Its pipeline for frame $i$8 begins with a spatial denoising module operating on the current noisy frame $i$9 and its noise level $v_i = g_i \cdot a_i + (1-g_i) \cdot f_i$0: $v_i = g_i \cdot a_i + (1-g_i) \cdot f_i$1 The previous recurrent feature state $v_i = g_i \cdot a_i + (1-g_i) \cdot f_i$2 is motion-compensated via guided deformable alignment,

$v_i = g_i \cdot a_i + (1-g_i) \cdot f_i$3

then filtered by the reset gate and concatenated with the current spatial feature: $v_i = g_i \cdot a_i + (1-g_i) \cdot f_i$4 After temporal denoising,

$v_i = g_i \cdot a_i + (1-g_i) \cdot f_i$5

the update gate produces the next recurrent feature state

$v_i = g_i \cdot a_i + (1-g_i) \cdot f_i$6

Finally, reconstruction fuses the updated state with the current spatial feature and predicts a residual: $v_i = g_i \cdot a_i + (1-g_i) \cdot f_i$7 The architecture is explicitly recurrent in time and uses only a single-frame delay because only $v_i = g_i \cdot a_i + (1-g_i) \cdot f_i$8 is carried forward (Guo et al., 2024).

A notable aspect of this model is that gating is not the only temporal mechanism; it is embedded inside a denoising backbone that uses the residual simplified Swin Transformer with Euclidean distance (RSSTE). Within each window, attention is computed as

$v_i = g_i \cdot a_i + (1-g_i) \cdot f_i$9

so closer features in Euclidean space receive higher weight. The paper argues that this is empirically more robust under noise than dot-product attention, and the RSSTE block is inserted in both spatial and temporal denoising modules (Guo et al., 2024). In this design, the gates decide which temporal content enters the fusion loop and how much of the refined result overwrites memory, whereas RSSTE performs fine-grained integration on the already gated feature mixture.

A structurally different temporal use of FGF appears in video captioning with dual graphs and gated fusion. There, frame-wise appearance and motion features are first temporally enriched by BiLSTMs, then transformed into four frame-level sequences: appearance GAT-enhanced, motion GAT-enhanced, appearance FORG-enhanced, and motion FORG-enhanced. At each decoding step $X$0, the attention LSTM state $X$1 queries each sequence, yielding four context vectors: $X$2 The gated fusion is then hierarchical: $X$3 This is frame-level in an indirect but technically important sense: each gate acts on vectors obtained by attention over frame sequences, and the gating decision is conditioned on the current decoder state, hence on the word-generation context (Jin et al., 2023).

These two cases define two major temporal FGF regimes. One is recurrent state management, where gates control persistence and overwrite of temporally propagated features. The other is task-conditioned aggregation, where gates select among multiple frame-derived summaries at higher semantic level. The shared principle is selective temporal retention rather than uniform temporal averaging.

4. Cross-modal, multispectral, and multi-stream realizations

FGF is especially prominent in multimodal settings because modality reliability often varies across frames. GateFusion addresses frame-level active speaker detection with synchronized face tracks and audio. Video frames $X$4, with $X$5, are encoded by the first 12 Transformer layers of Noise-Augmented AV-HuBERT Large, while audio log-Mel spectrograms $X$6 from Whisper’s official extraction pipeline are encoded by the first 12 Whisper large-v3 layers. Hidden states from layers $X$7 are projected and temporally average-pooled to a common length, then injected hierarchically across modalities through bimodally conditioned gates. The final multimodal feature is

$X$8

and the model produces per-frame active-speaker probabilities. In this system, a frame is a visual frame plus the matched audio chunk after temporal pooling; all fusion operations are applied per aligned time index and feature dimension (Wang et al., 17 Dec 2025).

GAID implements a lighter audio–visual FGF for text–video retrieval. Videos are uniformly sampled into a maximum of 12 frames, CLIP provides visual frame embeddings, Whisper or Wav2Vec2.0 provides high-frequency audio features that are average-pooled into the same 12 segments, and a global CLIP text embedding conditions the frame-wise gate. Silent videos are handled by inserting zero vectors for audio. The result is a sequence of fused frame embeddings $X$9, later aggregated into a video embedding and further enhanced by a lightweight cross-attention with text (Yang et al., 3 Aug 2025). Relative to fully token-level cross-modal transformers, this is a deliberately low-overhead form of temporal multimodal gating.

Dynamic saliency prediction uses another cross-stream formulation. GFSalNet has two ResNet-50 streams, one for RGB appearance and one for optical flow generated by PWC-Net between consecutive frames. Each stream uses multi-level information fusion, spatial attention, and channel-wise attention to produce stream-specific saliency maps $Y$0 and $Y$1. The gated fusion block concatenates these maps, applies a $Y$2 convolution and sigmoid to produce a probability map $Y$3, sets $Y$4 and $Y$5, and computes

$Y$6

The gate is spatially varying, so motion can dominate one region while appearance dominates another within the same frame (Kocak et al., 2021).

Multispectral pedestrian detection provides an antecedent that is per-frame but not temporal. GFD-SSD runs two VGG16-based SSD branches, one for color and one for thermal input, and inserts GFUs at feature pyramid levels such as $Y$7, $Y$8, and deeper SSD maps. In GFU_v1, the adjustments $Y$9 and $$C_t = \lambda_t \odot f(X_t) + (1-\lambda_t)\odot f(Y_t)$$0 are computed from the concatenated feature $$C_t = \lambda_t \odot f(X_t) + (1-\lambda_t)\odot f(Y_t)$$1; in GFU_v2 they are computed from each modality separately. Both variants produce a fused feature map $$C_t = \lambda_t \odot f(X_t) + (1-\lambda_t)\odot f(Y_t)$$2 of the same spatial and channel size as a single branch output, and the paper notes that the same mechanism can be adapted to current-vs-previous frame fusion by replacing modalities with time steps (Zheng et al., 2019).

The sensor-fusion literature broadens the picture further. The baseline netgated architecture computes per-sensor scalar fusion weights from concatenated sensor features, FG-GFA replaces these with group-level gates, and 2S-GFA multiplies feature-level and group-level weights,

$$C_t = \lambda_t \odot f(X_t) + (1-\lambda_t)\odot f(Y_t)$$3

before final fusion. Although framed in terms of driving mode prediction and human activity recognition windows rather than video frames, the mechanism is explicitly applied independently at each temporal window, making it an important coarse-grained FGF precursor (Shim et al., 2018).

5. Empirical behavior and design trade-offs

Across tasks, FGF is repeatedly supported by ablations that compare adaptive gates against static fusion or ungated concatenation. In recurrent video denoising, disabling the gated scheme reduces PSNR by about $$C_t = \lambda_t \odot f(X_t) + (1-\lambda_t)\odot f(Y_t)$$4 dB at $$C_t = \lambda_t \odot f(X_t) + (1-\lambda_t)\odot f(Y_t)$$5, while replacing Euclidean attention with dot-product attention reduces PSNR by about $$C_t = \lambda_t \odot f(X_t) + (1-\lambda_t)\odot f(Y_t)$$6 dB at $$C_t = \lambda_t \odot f(X_t) + (1-\lambda_t)\odot f(Y_t)$$7. The paper interprets both gating and RSSTE-based attention as crucial to obtaining denoising performance comparable to state-of-the-art multi-frame-delay networks with only a single-frame delay (Guo et al., 2024).

In dual-graph video captioning, hierarchical gating improves over simple concatenation. On MSVD, AMS-DG reports CIDEr $$C_t = \lambda_t \odot f(X_t) + (1-\lambda_t)\odot f(Y_t)$$8, whereas AMS-DG-GATE reports CIDEr $$C_t = \lambda_t \odot f(X_t) + (1-\lambda_t)\odot f(Y_t)$$9; on MSR-VTT, the same comparison is $$\tilde{f}_p \leftarrow \mathrm{LN}(\tilde{f}_p + g^l \odot \tilde{h}_c^l)$$0 versus $$\tilde{f}_p \leftarrow \mathrm{LN}(\tilde{f}_p + g^l \odot \tilde{h}_c^l)$$1. METEOR also improves, while BLEU-4 changes only slightly. The paper explicitly contrasts gated fusion with a no-gate variant that concatenates context features and feeds them directly to the language LSTM (Jin et al., 2023).

In active speaker detection, hierarchical frame-level gating substantially outperforms simple late sum. On Ego4D-ASD, the baseline without HiGate, MAL, or OPP scores $$\tilde{f}_p \leftarrow \mathrm{LN}(\tilde{f}_p + g^l \odot \tilde{h}_c^l)$$2 mAP; HiGate alone reaches $$\tilde{f}_p \leftarrow \mathrm{LN}(\tilde{f}_p + g^l \odot \tilde{h}_c^l)$$3, and HiGate with MAL and OPP reaches $$\tilde{f}_p \leftarrow \mathrm{LN}(\tilde{f}_p + g^l \odot \tilde{h}_c^l)$$4. On AVA-ActiveSpeaker, the corresponding values are $$\tilde{f}_p \leftarrow \mathrm{LN}(\tilde{f}_p + g^l \odot \tilde{h}_c^l)$$5, $$\tilde{f}_p \leftarrow \mathrm{LN}(\tilde{f}_p + g^l \odot \tilde{h}_c^l)$$6, and $$\tilde{f}_p \leftarrow \mathrm{LN}(\tilde{f}_p + g^l \odot \tilde{h}_c^l)$$7. Additional ablations show that spaced hierarchical fusion across selected layers outperforms single-stage fusion, and that dense fusion at all 12 layers increases cost without improving much (Wang et al., 17 Dec 2025).

In text–video retrieval, GAID directly compares fusion granularity. On MSR-VTT with ViT-B/32, no fusion yields R@1 $$\tilde{f}_p \leftarrow \mathrm{LN}(\tilde{f}_p + g^l \odot \tilde{h}_c^l)$$8, R@5 $$\tilde{f}_p \leftarrow \mathrm{LN}(\tilde{f}_p + g^l \odot \tilde{h}_c^l)$$9, R@10 $S_{\text{final}} = S_A \odot G_A + S_T \odot G_T$0, MnR $S_{\text{final}} = S_A \odot G_A + S_T \odot G_T$1; sample-level gating yields $S_{\text{final}} = S_A \odot G_A + S_T \odot G_T$2; frame-level gating yields $S_{\text{final}} = S_A \odot G_A + S_T \odot G_T$3; token-level gating yields $S_{\text{final}} = S_A \odot G_A + S_T \odot G_T$4. The paper therefore identifies frame-level gating as the best trade-off between granularity and efficiency (Yang et al., 3 Aug 2025).

In dynamic saliency prediction, replacing the gated fusion block with a standard $S_{\text{final}} = S_A \odot G_A + S_T \odot G_T$5 convolution decreases performance on UCF-Sports from AUC-J $S_{\text{final}} = S_A \odot G_A + S_T \odot G_T$6, CC $S_{\text{final}} = S_A \odot G_A + S_T \odot G_T$7, NSS $S_{\text{final}} = S_A \odot G_A + S_T \odot G_T$8, SIM $S_{\text{final}} = S_A \odot G_A + S_T \odot G_T$9, KLDiv $$W_{RG}^{n} = H_{RG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$0 to AUC-J $$W_{RG}^{n} = H_{RG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$1, CC $$W_{RG}^{n} = H_{RG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$2, NSS $$W_{RG}^{n} = H_{RG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$3, SIM $$W_{RG}^{n} = H_{RG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$4, KLDiv $$W_{RG}^{n} = H_{RG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$5. The paper emphasizes that dynamic weighting according to content is superior to a fixed weighting scheme learned by convolution alone (Kocak et al., 2021).

Multispectral detection yields a more nuanced result: pure gated fusion is not uniformly best, and placement matters. On SSD300, Double + Stack reaches $$W_{RG}^{n} = H_{RG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$6 logMR, while Double + GFU_v1 and Double + GFU_v2 reach $$W_{RG}^{n} = H_{RG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$7 and $$W_{RG}^{n} = H_{RG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$8, respectively; but on SSD512, Double + Stack is $$W_{RG}^{n} = H_{RG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$9, Double + GFU_v1 is $$W_{UG}^{n} = H_{UG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$0, and Double + GFU_v2 is $$W_{UG}^{n} = H_{UG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$1. Mixed_Early with GFU_v2 yields $$W_{UG}^{n} = H_{UG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$2 on SSD300 and $$W_{UG}^{n} = H_{UG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$3 on SSD512, indicating that early high-resolution fusion can be preferable to gating all pyramid levels (Zheng et al., 2019).

Sensor-fusion experiments show a similar regularity under noise and failures. In driving prediction with $$W_{UG}^{n} = H_{UG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$4 noise, Non-NetGated achieves $$W_{UG}^{n} = H_{UG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$5, NetGated $$W_{UG}^{n} = H_{UG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$6, Group-level $$W_{UG}^{n} = H_{UG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$7, and Two-stage $$W_{UG}^{n} = H_{UG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$8; under sensor failures, the respective results are $$W_{UG}^{n} = H_{UG}(\mathrm{InCon}(F_{SD}^{n}, F_{WB}^{n-1})),$$9, $i$00, $i$01, and $i$02. Human activity recognition exhibits the same ordering. This supports the view that hierarchical gating can improve robustness when reliability varies across input sources (Shim et al., 2018).

Taken together, these results show that FGF is rarely beneficial merely because it is a gate. Performance gains depend on where the gate is inserted, what context conditions it, and whether the gating granularity matches the temporal and semantic variability of the task.

6. Misconceptions, limitations, and research directions

A common misconception is that FGF is synonymous with temporal recurrence. The literature does not support that restriction. In GRTN, FGF is recurrent and stateful (Guo et al., 2024); in GAID, it is frame-wise and query-conditioned but not recurrent (Yang et al., 3 Aug 2025); in GateFusion, it is hierarchical and non-causal because the encoders use bidirectional context within the chunk (Wang et al., 17 Dec 2025); in GFD-SSD, it is modality-level fusion within a single frame rather than across time (Zheng et al., 2019). The unifying property is adaptive fusion at frame granularity, not any single temporal dependency pattern.

A second misconception is that gating always means a bounded multiplicative mask. GFD-SSD explicitly replaces sigmoid and element-wise product with ReLU and element-wise sum inside the GFU, making the “gate” an additive correction (Zheng et al., 2019). Conversely, GFSalNet and GAID use sigmoid-based complementary weights, while GateFusion uses sigmoid gates for residual context injection (Kocak et al., 2021, Yang et al., 3 Aug 2025, Wang et al., 17 Dec 2025). The term therefore spans both convex-combination and residual-injection formulations.

The literature also delineates clear limitations. Gate efficacy depends on feature resolution and placement: GFUs are more effective in SSD512 than SSD300 and benefit more from early than late placement (Zheng et al., 2019). In active speaker detection, dense fusion at all layers increases cost and does not improve much relative to spaced hierarchical fusion (Wang et al., 17 Dec 2025). In text–video retrieval, token-level gating does not outperform frame-level gating despite greater complexity (Yang et al., 3 Aug 2025). In the dual-graph captioning model, the authors note future work on more graph types and more efficient use of many feature representations, implying that hierarchical pairwise gating may become cumbersome as the number of streams grows (Jin et al., 2023). The sensor-fusion study further indicates that fine-grained feature-level gating alone can overfit, motivating group-level and two-stage formulations (Shim et al., 2018).

A plausible synthesis of these results is that FGF works best when it aligns with a task’s natural unit of uncertainty. When reliability varies mainly across frames, frame-wise scalar or vector gates are effective. When reliability varies within frame regions, spatially varying gates are preferable. When semantic abstraction shifts across network depth, hierarchical gates across layers become useful. When long-term memory must be maintained under noise and motion, recurrent reset/update gating is appropriate. This suggests that the main research problem is not whether to gate, but how to match gate granularity, conditioning variables, and insertion depth to the structure of the underlying temporal decision process.