DeepStack Multimodal Fusion

• Multimodal fusion is a strategy that integrates feature representations from diverse data modalities at intermediate embedding levels.
• DeepStack approaches stack fusion modules deeply within networks, enabling rich cross-modal interactions with improved parameter efficiency.
• Empirical evaluations demonstrate that DeepStack fusion enhances accuracy, reduces memory costs, and scales effectively across varied applications.

Multimodal embedding-level fusion with DeepStack refers to a family of architectural and algorithmic strategies for fusing feature representations from multiple data modalities (such as vision, audio, and language) at the intermediate embedding level, typically by stacking fusion modules deeply throughout a backbone network. The “DeepStack” paradigm is characterized by explicit stacking of fusion operations at several network depths, often via parameter-efficient modules and learned gating, enabling rich cross-modal interactions while controlling computational and memory costs. This approach has demonstrated strong performance and scalability across domains, including large multimodal LLMs, visual-language transformers, biometric identification, and multimodal sentiment analysis (Meng et al., 2024, Georgiou et al., 15 Apr 2025, Soleymani et al., 2018).

1. Conceptual Foundations of Embedding-Level Multimodal Fusion

DeepStack-style embedding-level fusion is based on the insight that cross-modal integration benefits from combining features across multiple depths of abstraction—ranging from low-level local structure to high-level semantic representations. Unlike score- or decision-level fusion (late fusion), embedding-level (feature-level) fusion integrates signals earlier, permitting the network to jointly learn composite features and shared representations. The DeepStack framework extends this notion by introducing fusion operations at multiple points or layers within a network, creating both deep and flexible inter-modal connectivity.

In the DeepMLF architecture, a series of “fusion tokens” are interleaved with textual tokens and updated through stacked cross-attention modules placed after multiple transformer layers, yielding a deeply-layered multimodal representation (Georgiou et al., 15 Apr 2025). Similarly, in large multimodal transformers, DeepStack injects successive groups of visual tokens into progressively deeper layers of the LLM, encouraging each layer to specialize in interactions at distinct spatial or semantic scales (Meng et al., 2024). In early convolutional approaches, features from multiple abstraction depths (e.g., shallow and deep embeddings pre- and post-pooling) are concatenated and processed through a joint fusion network (Soleymani et al., 2018).

This paradigm stands in contrast to “single-point” or “single-layer” fusion and is justified empirically by improved accuracy, convergence, and parameter/memory efficiency compared to alternative fusion strategies (Meng et al., 2024, Georgiou et al., 15 Apr 2025, Soleymani et al., 2018).

2. Architectural Design Patterns: DeepStack Strategies

DeepStack-style fusion can be realized via several architectural variants, all emphasizing repeated or stacked feature interactions at the embedding level:

Visual-Language Transformers

In “DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs” (Meng et al., 2024), M visual tokens are grouped and injected additively into successive layers of a language transformer. For L transformer layers and S token-groups, at every n layers (starting at ℓ₀), a new group of tokens is merged via residual addition into the hidden state positions corresponding to visuals: $$H_{ℓ}[\,\mathit{vis\_pos}\,]\;←\;H_{ℓ}[\,\mathit{vis\_pos}\,] + X^{stack^{\,i}}$$ This process yields O(1) context length, O(S·M·d) extra memory for tokens, and O(1) additional attention cost per layer (compared to O(s·M) context when stringing all tokens at input). The approach achieves superior performance and computational efficiency relative to conventional prefix-fusion (Meng et al., 2024).

Deep Fusion with Learnable Fusion Tokens

DeepMLF employs a “DeepStack” of MM fusion blocks: at selected layers within a frozen decoder-only LLM, learnable fusion tokens undergo gated cross-attention with modality-specific audiovisual embeddings (Georgiou et al., 15 Apr 2025). At layer l, the MM block updates the fusion tokens $X_f$ using the audiovisual output $Z$ via: $$\hat A = \mathrm{softmax}\!\left(\frac{Q_{f}K_{av}^{T}}{\sqrt{d}}\right)V_{av} \ X_f'' = X_f' + \sigma(a_1^{(l)})\odot\hat A$$ By stacking 5–7 such MM blocks, the architecture supports deep, progressive fusion. Text features and AV features are propagated independently with fusion tokens serving as the only bridge.

Multi-Level CNN-Based Fusion

In DeepStack for biometric identification, modality-specific CNN streams (face, iris, fingerprint) extract features at both shallow (pool3/FC3) and deep (pool5/FC6) layers. These 1024-dimensional embeddings are concatenated and mapped into a joint feature space: $$H = [f^{(1)}_3, f^{(1)}_6, f^{(2)}_3, f^{(2)}_6, f^{(3)}_3, f^{(3)}_6]$$ A fully-connected fusion layer outputs the final fused embedding, which is used for classification. Joint optimization ensures backpropagation through all modality-specific and fusion layers (Soleymani et al., 2018).

DEQ Fusion and Stack Equilibrium

Deep Equilibrium Multimodal Fusion models the limit of an infinite-depth stack of residual fusion modules via fixed-point solvers (e.g., Broyden’s method). Embedding-wise updates of unimodal ($z_i$) and fused ($z_\mathrm{fuse}$) states are performed recursively: $$z_fuse^{[j+1]} = \mathrm{GroupNorm}\Big( \mathrm{ReLU}( \theta_\mathrm{fuse} \sum z'_i + b_\mathrm{fuse} + x_\mathrm{fuse} ) \Big)$$ The final state serves as the all-depths-fused embedding. The backward pass is handled via the implicit function theorem, avoiding the need to store activations for every depth (Ni et al., 2023).

3. Fusion Mechanisms and Mathematical Formalism

The core mechanism underlying DeepStack approaches is the repeated, depth-wise integration of cross-modality embeddings, combined via either concatenation, residual addition, attention, or gating. Several representative formulations are prominent:

• Concatenation + Fusion Layer: Multi-level features $f^{(m)}_l$ from all modalities are concatenated to form $H$, fused via a learned fully-connected mapping:

$X_f$0

(Soleymani et al., 2018)

• Stacked Attention Fusion: Fusion tokens attend over external modality embeddings via cross-attention in stacked MM blocks, with output gated by a learnable scalar $X_f$1:

$X_f$2

(Georgiou et al., 15 Apr 2025)

• Layerwise Additive Stacking: In transformer LMMs, M-sized blocks of visual tokens are added into visual positions at selected depths (Meng et al., 2024):

| Operation | Formula | |---------------------------|---------------------------------------------------------------------| | Layer update (if inject) | $X_f$3 | | Else | $X_f$4 |

• Implicit Equilibrium (DEQ) Fusion: Stack of residual fusion blocks recursed to equilibrium, formulated as:

$X_f$5

(Ni et al., 2023)

The depth, position, and nature (sum, attention, gated combination) of embedding-level fusion are hyperparameters that affect performance, with 5–7 fusion sites and small bottleneck capacity (e.g., 8–20 tokens) typically optimal (Georgiou et al., 15 Apr 2025, Meng et al., 2024).

4. Optimization, Training, and Complexity

End-to-end training is standard in DeepStack frameworks, with several strategies evident:

• Joint Optimization: Loss is computed on the fused output and backpropagated through all embedding extraction and fusion layers, enforcing co-adaptation of modality-specific and fusion modules (Soleymani et al., 2018, Georgiou et al., 15 Apr 2025).
• Auxiliary Losses & Regularization: Multimodal tasks often benefit from auxiliary losses on unimodal outputs, fusion tokens, and an overall fusion regularizer. DeepMLF incorporates multiple L1 regression losses per head and a language modeling loss on text outputs:

$X_f$6

(Georgiou et al., 15 Apr 2025).

• Root-finding and Implicit Gradients: Deep Equilibrium Models use Broyden’s method or Anderson acceleration to converge to a fixed-point embedding, with gradients computed via the IFT (Ni et al., 2023).

In transformer-based DeepStack, the approach yields substantial reductions in complexity and memory: O(1) context length compared to O(s·M) when all tokens are presented at input, leading to 3–10× speedup and 4× lower GPU memory for high-res tasks (Meng et al., 2024).

5. Empirical Results, Benchmarks, and Practical Impact

Evaluation across several application domains consistently demonstrates the performance advantages of DeepStack-style deep embedding-level fusion:

Task / Benchmark	DeepStack Variant	Key Result/Improvement	Reference
High-Res VQA	DeepStack-L/V-7B LMM	+2.7 to +3.8 avg, +11 on DocVQA (short context)	(Meng et al., 2024)
Sentiment Analysis	DeepMLF (DeepStack)	Optimal at 5–7 MM blocks, few tokens	(Georgiou et al., 15 Apr 2025)
Biometric ID	CNN Multi-abstract	99.34% vs 99.14% (late fusion)	(Soleymani et al., 2018)
Multimodal DEQ	Stack-equilibrium	+1.2–2 pts macro/micro F1 on MM-IMDB, MOSI	(Ni et al., 2023)

Parameter and compute efficiency is a recurring advantage: CNN DeepStack reduces trainable weights by ~60–70% versus naive concatenation, transformer DeepStack reduces attention FLOPs by >20×, and bottlenecked fusion tokens constrain the compute cost to O(n_f^2 + n_f L_av) (Meng et al., 2024, Soleymani et al., 2018, Georgiou et al., 15 Apr 2025).

Empirical ablations confirm that effective fusion is achieved with moderate stacking depth (4–7), small bottlenecks (8–20 tokens), and that deeper or shallower configurations underperform (Meng et al., 2024, Georgiou et al., 15 Apr 2025).

6. Variations, Extensions, and Comparison with Alternative Fusion Strategies

Several extensions and alternative architectures are documented:

• Bidirectional and Asymmetric Multi-layer Fusion: Using channel shuffle and pixel-shift operations, bidirectional fusion can be performed within shared-parameter convolutional networks, relying on privatized batch-normalization. This achieves parameter-efficient fusion at every layer with negligible compute overhead (Wang et al., 2021).
• CentralNet / Central Stack: CentralNet generalizes deep stacking to a central “fusion network” that aggregates hidden states from each modality network at every depth, mediating via a weighted sum and fusion layer, with multi-task learning to retain unimodal branch potency (Vielzeuf et al., 2018). This bridges single-point and full-depth fusion.
• Fixed-Point/Equilibrium Stacking (DEQ): Abstracts a deep stack to its fixed point, enabling memory-efficient infinite-depth embedding integration (Ni et al., 2023).

Compared to late-fusion (score or decision level) and flat concatenation, DeepStack approaches have shown consistent improvements in accuracy, convergence/capacity, and parameter/memory scaling (Meng et al., 2024, Georgiou et al., 15 Apr 2025, Soleymani et al., 2018).

7. Theoretical and Practical Implications

The demonstrated efficacy of DeepStack-style deep stacking is attributed to:

• Enabling layer-specialization, where lower layers integrate fine-grained or local multimodal information, and higher layers synthesize global or abstract representations (Meng et al., 2024);
• Avoiding the bottleneck or overwhelming of early layers by incrementally integrating partial modality information;
• Reducing compute/memory costs by limiting context expansion and parameterizing the fusion bottleneck;
• Allowing flexible insertion at varying depths, aligning fusion dynamics with task requirements.

The modularity and efficiency of the approach support transfer to other multimodal learning settings by adapting the plug-in "fusion stack" to the encoders/decoders of interest, and by configuring depth, gating, and bottleneck as hyperparameters. These properties are directly exploited in language–vision transformers, biometric systems, multimodal sentiment models, and DEQ-fusion pipelines (Meng et al., 2024, Georgiou et al., 15 Apr 2025, Soleymani et al., 2018, Ni et al., 2023).

A plausible implication is that future scaling of multimodal LLMs and deep networks will rely on increasingly sophisticated, dynamic, and hierarchical versions of DeepStack-style embedding-level fusion for tractable, high-fidelity cross-modal reasoning.

---

References:

(Soleymani et al., 2018, Vielzeuf et al., 2018, Georgiou et al., 15 Apr 2025, Ni et al., 2023, Wang et al., 2021, Meng et al., 2024)