DeepStack Fusion: Efficient Neural Feature Fusion

• DeepStack Fusion is a set of techniques that fuses high-resolution features and tokens across layers to enhance multi-scale context and gradient propagation.
• It integrates methods from multimodal transformers, focus stacking pipelines, and ensemble architectures to minimize compute overhead and maintain spatial details.
• Empirical benchmarks demonstrate significant improvements in metrics like PSNR/SSIM, accuracy, and performance across diverse applications such as imaging and multimodal reasoning.

DeepStack Fusion encompasses a set of architectural principles and algorithmic techniques for deep neural network fusion, with the explicit goal of maximizing feature interaction, effective context utilization, and computational efficiency across high-resolution and multi-focus domains. DeepStack Fusion operates at both the representational and token levels, leveraging staged or layer-wise combination of features and tokens, with notable instantiations in large multimodal models (LMMs), burst image fusion for focus stacking, and multi-network ensemble architectures. Central to the concept is the controlled injection or summation of deeply partitioned features or tokens across network layers, achieving both wide multi-scale context and steeper information flow. This entry, referencing key works by Araújo et al. ["Towards Real-World Focus Stacking with Deep Learning" (Araujo et al., 2023)], Wang et al. ["DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs" (Meng et al., 6 Jun 2024)], and the foundational fusion principles of Zhou et al. ["Deeply-Fused Nets" (Wang et al., 2016)], provides a comprehensive overview of DeepStack Fusion's technical mechanisms, empirical impacts, and theoretical implications.

1. Architectural Principles and Fusion Mechanisms

DeepStack Fusion is characterized by the staged injection of diverse feature partitions or token groups into sequential network layers. In visual LMMs, DeepStack arranges high-resolution visual tokens into $N$ groups, each infused into specific transformer layers via residual connections while keeping the sequence length fixed, circumventing input bottlenecks and minimizing compute overhead. The formal update at each injection layer $\ell_i$ is:

$$\widetilde H_\ell = \begin{cases} H_{\ell-1} + [0, \ldots, 0, V^{(i)}, 0, \ldots, 0] & \ell = \ell_i \in \mathcal{L} \ H_{\ell-1} & \text{otherwise} \end{cases}$$

where each $V^{(i)} \in \mathbb{R}^{T \times c}$ is a spatially coherent high-res chunk, aligned to the global token grid.

In focus stacking, FocusDeep leverages deeply-aligned convolutional features via deformable convolutions and concatenative fusion across raw image bursts, culminating in a shared feature map that is serially refined and upsampled through wide-activation residual blocks and long-range residual connections.

In deeply-fused nets, fusion occurs post block, with intermediate representations from each base network combined via element-wise summation, propagating fused activation and gradient signals through all subsequent layers of all base networks.

2. Detailed Workflow: Image and Multimodal Fusion

Focus Stacking Pipeline

The fusion process for real-world focus stacking (FocusDeep) involves:

1. Demosaicing raw frames to RGB.
2. High-resolution affine alignment using the ECC algorithm.
3. Pyramidal pseudo ground-truth generation via HeliconFocus.
4. Extraction of overlapping crops for efficient training.
5. Triple-stage deep fusion network:
   • Shared CNN encoding and deformable feature alignment.
   • Concatenative fusion via $1\times1$ convolution.
   • Reconstruction through LRCN and WARB blocks with skip connections. The formal mapping is:

$$I_{\mathrm{out}} = \mathcal{F}_\theta(\tilde R_1, \ldots, \tilde R_N)$$

with $L_1$ reconstruction loss against pyramidal pseudo ground-truth.

DeepStack for Multimodal Transformers

Visual tokens are partitioned and residually added layer-wise—pseudocode specifies per-layer addition with minimal architectural change and no modifications to attention masks or self-attention. Only the global token slots are updated with new high-res chunks, and compute/memory cost is proportional to $(S+T)^2$, not $(S+N\,T)^2$ as in naive shallow fusion.

Deeply-Fused Ensemble Nets

Fusions are placed after every block of $K$ base networks. Each block's output is projected (if necessary), summed, and fed as input to each subsequent block across all networks, optimizing both multi-scale context and activation/gradient path length.

3. Information Flow and Empirical Effects

DeepStack Fusion architectures demonstrably enhance both information and gradient flow:

• Gradients are aggregated from fusion points, allowing shallow paths to supplement deep, weak-signal gradients, and reducing vanishing/exploding behavior.
• Activation paths are similarly shortened—features from early deep layers may traverse shallow blocks directly, yielding improved signal preservation.
• In FocusDeep, deformable alignment and concatenative fusion maintain spatial context, maximizing detail recovery and artifact suppression.
• Layer-wise token stacking in DeepStack LMMs avoids context collapse and leverages early layer encoding capacity for spatial structure.

4. Computational Efficiency and Scalability

A distinctive property is compute/memory preservation. Because DeepStack maintains fixed sequence length for all layers and injects only $T$ tokens per stage, it scales to an $N$-fold increase in visual vocabulary without quadratic growth in cost:

$\text{Time}_{\rm DeepStack} \approx L (S + T)^2 d$

compared to

$\text{Time}_{\rm shallow} \approx L (S + N\,T)^2 d$

when $N \gg 1$ (Meng et al., 6 Jun 2024). Fusion in deeply-fused nets and FocusDeep similarly require only additive overhead plus occasional $1\times1$ projections.

5. Quantitative and Qualitative Benchmarks

Large Multimodal Models (LMMs)

DeepStack-L-7B exhibits $+4.2$, $+11.0$, and $+4.0$ point improvements (TextVQA, DocVQA, InfoVQA) over baseline LLaVA-1.5-7B, averaged $+2.7$ across nine tasks. With just one-fifth of context length, performance approaches full-length baselines (Meng et al., 6 Jun 2024).

Focus Stacking

PSNR/SSIM for FocusDeep:

• RGB input: $38.47$ dB, $0.965$
• Raw input: $32.89$ dB, $0.898$ Substantially surpassing baseline stacking and wavelet methods, and matching commercial HeliconFocus in both detail and noise suppression (Araujo et al., 2023).

Deeply-Fused Nets

DFN-19 (deep+shallow) improves accuracy on CIFAR-10 ($93.98\%$) and CIFAR-100 ($72.64\%$), outperforming ResNet-20 ($71.17\%$) and Highway-19 ($67.97\%$) (Wang et al., 2016).

Model	Dataset	Metric	Value
DeepStack-L-7B	DocVQA	Score	39.1
FocusDeep (RGB)	Focus Crop	PSNR / SSIM	38.47 / 0.965
DFN-19	CIFAR-100	Accuracy %	72.64

6. Design Choices and Ablation Insights

Empirical ablation studies highlight critical factors:

• Early-layer injection of high-res tokens is robust; late-layer stacking degrades.
• 2D sampling for spatially coherent chunks outperforms gridwise flattening.
• Evenly spaced layer injection ($\Delta=8$ for $L=32$) is optimal.
• Dummy (repeated low-res) token injection yields no gain, attributing improvements solely to detailed spatial context (Meng et al., 6 Jun 2024).
• In FocusDeep, realistic noise augmentation and ECC alignment are essential for handling real-world burst variability (Araujo et al., 2023).

7. Advantages and Implications

DeepStack Fusion offers:

1. Multi‐scale, multi‐path representations through fusion of diverse blocks or token groups, enabling exchangeability and richer receptive-field ensembles (Wang et al., 2016).
2. Improved training dynamics from gradient and activation flow shortcuts.
3. Reduced effective network depth for signal propagation, facilitating training on deep architectures.
4. Negligible parameter and compute overhead due to summation-based fusion and fixed-length token injection.
5. Generalizability across real-world imaging and multimodal reasoning, with empirically demonstrated robustness to input noise and burst variability. A plausible implication is that DeepStack Fusion presents a canonical mechanism for both large-scale visual-linguistic integration and detailed frame fusion in computational photography.

• Araújo et al., "Towards Real-World Focus Stacking with Deep Learning" (Araujo et al., 2023).
• Wang et al., "DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs" (Meng et al., 6 Jun 2024).
• Zhou et al., "Deeply-Fused Nets" (Wang et al., 2016).