DeepStack Cross-Layer Feature Injection

• The paper demonstrates that layer-wise injection of visual tokens enables iterative refinement in multimodal models, mitigating information loss inherent in standard pipelines.
• It details a method where high-resolution visual tokens are partitioned and aligned with transformer layers, reducing bottlenecks and preserving spatial details.
• Empirical results show notable improvements (e.g., +2.7 to +2.9 on 7B/13B models and enhanced VQA scores), underscoring the practical impact of cross-layer feature injection.

Searching arXiv for the papers on arXiv and closely related work to ground the article in current literature. DeepStack cross-layer feature injection denotes an architectural strategy in which representations are introduced repeatedly across depth rather than being concatenated once at the input or discarded after a single fusion stage. In its canonical large multimodal model form, DeepStack partitions visual tokens into multiple groups and feeds each group to its aligned transformer layer from bottom to top, so that visual information is refined across successive layers instead of being consumed only by the first layer (Meng et al., 2024). Closely related work instantiates the same principle through attentive aggregation in dense prediction, aggregate-then-distribute communication in feature pyramids, bidirectional attention between mid- and high-level features, dynamic many-to-many vision-language bridges, and recurrent memory-modulated feedback inside the vision encoder (Yang et al., 2021, Li et al., 2020, Huang et al., 2022, Chen et al., 15 Jan 2026, Liu et al., 28 Feb 2026).

1. Conceptual basis and problem setting

The common motivation behind cross-layer feature injection is that standard hierarchical networks often compress or overwrite information that remains useful at later stages. In dense prediction, ordinary concatenation and summation are described as too weak because they treat all feature channels and all spatial locations uniformly, so they cannot express which layer should dominate at which pixel or which semantic region (Yang et al., 2021). In FPN-style saliency models, indirect information propagation among distant layers can lead to incomplete object structures and unclear object boundaries (Li et al., 2020). In fine-grained categorization, intermediate layers preserve finer spatial detail but lack semantic or global context, whereas the top layer contains strong semantic or global context but is spatially coarse and can miss subtle parts (Huang et al., 2022).

The same structural critique appears in multimodal systems. Standard large multimodal models commonly feed visual tokens as a sequence into only the first layer of the LLM, which increases computation and memory costs while providing only a shallow opportunity for visual refinement (Meng et al., 2024). Vision-LLMs that use only the final vision layer and project it once into the LLM input are described as having a severe visual feature bottleneck, because a static one-to-one bridge discards hierarchical visual knowledge (Chen et al., 15 Jan 2026). In stateful cross-layer vision modulation, standard ViT-style hierarchical encoding is said to progressively abstract away early visual features, while direct shallow-feature injection into the LLM creates semantic mismatch with the visual feature space that the LLM’s cross-attention layers were pretrained on (Liu et al., 28 Feb 2026).

A recurring consequence is that cross-layer injection is not merely a transport mechanism. It changes representation formation. Rather than deciding only how much to read from each layer, these methods intervene in how later features are computed from earlier ones. This suggests that “cross-layer feature injection” is best understood as a family of conditional routing mechanisms that redistribute responsibility across depth, scale, or modality, rather than as a single fixed operator.

2. DeepStack as a layer-wise token injection architecture

DeepStack formalizes cross-layer injection in large multimodal models by replacing the standard input-prefix design with a layer-wise stacking scheme. Given an input image $\mathbf{I}$, a vision encoder $\mathcal{F}^v$ extracts features $\mathbf{f}^v = \mathcal{F}^v(\mathbf{I})$, and a connection module $\mathcal{M}$ maps them into visual tokens $\mathbf{X} = \mathcal{M}(\mathbf{f}^v)$. A high-resolution image $\mathbf{I}^{hires}$ is processed to produce stacked token groups

$$\mathbf{X}^{stack} = \{\mathbf{X}^{stack^1}, \mathbf{X}^{stack^2}, \dots, \mathbf{X}^{stack^s}\} = \mathrm{Sampling2D}\left(\mathcal{M}(\mathcal{F}^v(\mathbf{I}^{hires}))\right),$$

where each group has the same length as the global visual tokens and is spatially aligned by $2$D sampling rather than arbitrary flattening (Meng et al., 2024).

The language-side injection rule is additive and layer-aligned. If $$\mathcal{B}^V = \{\mathcal{P}^{V^1},\dots,\mathcal{P}^{V^n}\}$$ denotes the DeepStack blocks and $\mathcal{P}^{\mathbb{L}}$ the later plain block, the hidden-state evolution is written as

$\mathcal{F}^v$0

$\mathcal{F}^v$1

$\mathcal{F}^v$2

The model therefore alternates between transformer computation and residual visual reinjection. In the paper’s interpretation, early decoder layers behave more like an encoder, repeatedly refining visual representations, while later layers remain available for autoregressive sequence modeling (Meng et al., 2024).

Empirically, DeepStack reports that, using the same context length, the 7B and 13B variants surpass their counterparts by $\mathcal{F}^v$3 and $\mathcal{F}^v$4 on average across $\mathcal{F}^v$5 benchmarks, respectively. Using only one-fifth of the context length, DeepStack rivals closely to counterparts that use the full context length. The gains are especially large on high-resolution tasks, with improvements of $\mathcal{F}^v$6 on TextVQA, $\mathcal{F}^v$7 on DocVQA, and $\mathcal{F}^v$8 on InfoVQA compared to LLaVA-1.5-7B. A vision-transformer variant, DeepStack-V, reports a similar average improvement of $\mathcal{F}^v$9 compared with LLaVA-1.5-7B (Meng et al., 2024).

The ablations delimit the mechanism. Injecting into earlier layers is beneficial, whereas injecting too deep hurts performance; increasing the number of stacking layers consistently improves performance, with best results around four layers; and replacing high-resolution groups with dummy repeated tokens does not improve performance. The paper therefore attributes the effect to high-resolution layer-wise infusion rather than to residual addition by itself (Meng et al., 2024).

3. Operator families for cross-layer injection in vision models

Outside large multimodal models, cross-layer feature injection appears in several operator families that differ in where the reinjected signal is computed and how it is applied.

Approach	Injection route	Characteristic operator
AFA / SSR	Shallow and deep features; late multi-scale predictions	Spatial attention, channel attention, scale-space rendering
CFPN	All pyramid levels to a global map, then back to all levels	Cross-layer aggregation and cross-layer distribution
CLAN	Top layer to mid layer, then mid layer back to top layer	Context attention and spatial attention
CFSAM	Multiple SSD feature maps jointly, then restored to each scale	Flatten-concatenate, partitioned Transformer, restoration

In Attentive Feature Aggregation, the binary-fusion unit takes a shallower feature map $\mathbf{f}^v = \mathcal{F}^v(\mathbf{I})$0 and a deeper one $\mathbf{f}^v = \mathcal{F}^v(\mathbf{I})$1. Spatial attention is computed from the shallow feature, channel attention from the deeper feature, and the aggregate is formed by a nonlinear weighted rule,

$\mathbf{f}^v = \mathcal{F}^v(\mathbf{I})$2

For multiple features, AFA uses hierarchical attentive fusion,

$\mathbf{f}^v = \mathcal{F}^v(\mathbf{I})$3

The associated Scale-Space Rendering extension fuses already-computed multi-scale predictions by

$\mathbf{f}^v = \mathcal{F}^v(\mathbf{I})$4

With $\mathbf{f}^v = \mathcal{F}^v(\mathbf{I})$5, the paper reports better gradient behavior than the softplus-based HMA formulation and better validation performance across epochs (Yang et al., 2021).

CFPN uses a different logic: it first aggregates multi-scale side outputs into a shared global feature, then redistributes that aggregated feature back into the pyramid. The global descriptor is reweighted by collaborative layer weights and assembled as

$\mathbf{f}^v = \mathcal{F}^v(\mathbf{I})$6

after which cross-layer feature distribution regenerates scale-specific maps by multi-scale pooling. The effect is that each redistributed layer receives both semantic and detail information from all other layers simultaneously rather than only through adjacent top-down steps (Li et al., 2020).

CLAN implements bidirectional injection between a mid-level feature map and the top-level feature map. Its Cross-layer Context Attention module injects top-level context into the mid-level map by computing an interaction matrix from fused mid- and top-level features, then applying a residual non-local refinement. Its Cross-layer Spatial Attention module compresses the refined mid-level map into a spatial mask and applies that mask to the top-level feature map, so that local attention estimated from the middle layer gates high-level features. The resulting mechanism is a mutual refinement loop rather than a one-way skip connection (Huang et al., 2022).

CFSAM extends the same idea to multi-scale object detection by flattening multiple SSD prediction feature maps into one concatenated sequence, partitioning that sequence, applying Transformer self-attention within partitions, and restoring the result to the original scales through residual-style fusion. The module is explicitly designed to model dependencies across the entire multi-scale set rather than only within a single layer or a pair of layers (Xie et al., 16 Oct 2025).

4. Dynamic and stateful multimodal generalizations

DeepStack’s layer-wise visual stacking is not the only multimodal interpretation of cross-layer injection. Later work reframes the same problem as dynamic selection or recurrent modulation rather than fixed additive infusion.

Cross-Layer Injection introduces a dynamic many-to-many bridge between multiple vision layers and multiple LLM decoder layers. It comprises Adaptive Multi-Projection and Adaptive Gating Fusion. AMP aligns a sampled visual layer $\mathbf{f}^v = \mathcal{F}^v(\mathbf{I})$7 by

$\mathbf{f}^v = \mathcal{F}^v(\mathbf{I})$8

where the low-rank term adapts the frozen projector to the statistics of layer $\mathbf{f}^v = \mathcal{F}^v(\mathbf{I})$9. AGF then computes a context-dependent gate from attention summaries of the visual tokens and current hidden state,

$\mathcal{M}$0

and applies a masked selective fusion rule,

$\mathcal{M}$1

The paper positions DeepStack as a one-to-many style injection that is brute-force and context-blind, whereas CLI is intended to let the LLM query the full visual hierarchy on demand (Chen et al., 15 Jan 2026).

Stateful Cross-layer Vision Modulation internalizes the cross-layer pathway inside the vision encoder itself. It maintains a recurrent memory state $\mathcal{M}$2 updated by

$\mathcal{M}$3

with an LSTM-style gated recurrence

$\mathcal{M}$4

where the update is conditioned on layer summaries and a global textual conditioning signal $\mathcal{M}$5. After the update, the Token-Adaptive Gate modulates token representations by

$\mathcal{M}$6

An auxiliary semantic alignment loss,

$\mathcal{M}$7

encourages the final memory state to store answer-relevant semantic content (Liu et al., 28 Feb 2026).

These dynamic variants sharpen an important distinction. DeepStack distributes visual evidence across depth by predetermined layer-aligned injection, CLI makes that access context-sensitive, and SCVM makes it recurrent and internal to visual representation formation. A plausible implication is that “cross-layer feature injection” has evolved from a placement strategy into a broader design space of routing, memory, and gating mechanisms.

5. Empirical signatures across tasks

Across the literature, the reported gains are task-specific but structurally consistent: methods improve when they preserve fine detail, retain intermediate states, or allow deeper stages to revisit earlier information.

For dense prediction, AFA improves the Deep Layer Aggregation model by nearly $\mathcal{M}$8 mIoU on Cityscapes in the abstract, and the detailed DLA-34 ablation shows a progression from $\mathcal{M}$9 mIoU for the original linear fusion baseline to $\mathbf{X} = \mathcal{M}(\mathbf{f}^v)$0 with channel attention, $\mathbf{X} = \mathcal{M}(\mathbf{f}^v)$1 with spatial plus channel attention, $\mathbf{X} = \mathcal{M}(\mathbf{f}^v)$2 after auxiliary heads, $\mathbf{X} = \mathcal{M}(\mathbf{f}^v)$3 with multiple feature fusion, and $\mathbf{X} = \mathcal{M}(\mathbf{f}^v)$4 when SSR replaces average pooling or HMA-style multi-scale inference. The full AFA-DLA-X-102 model reaches $\mathbf{X} = \mathcal{M}(\mathbf{f}^v)$5 mIoU on Cityscapes validation, compared with $\mathbf{X} = \mathcal{M}(\mathbf{f}^v)$6 for the original DLA baseline under the paper’s full training and inference recipe. On boundary detection, AFA-DLA reports $\mathbf{X} = \mathcal{M}(\mathbf{f}^v)$7 ODS / $\mathbf{X} = \mathcal{M}(\mathbf{f}^v)$8 OIS on NYUDv2 and $\mathbf{X} = \mathcal{M}(\mathbf{f}^v)$9 ODS on BSDS500, which the paper attributes to better preservation of shallow spatial detail and stronger semantic selection from deeper layers (Yang et al., 2021).

For saliency and fine-grained recognition, the same pattern appears. On the ResNet-50 backbone, CFPN reports a sequence on DUT-OMRON and PASCAL-S from $\mathbf{I}^{hires}$0 and $\mathbf{I}^{hires}$1 for the backbone baseline, to $\mathbf{I}^{hires}$2 and $\mathbf{I}^{hires}$3 with FPN, to $\mathbf{I}^{hires}$4 and $\mathbf{I}^{hires}$5 with CFA, and finally to $\mathbf{I}^{hires}$6 and $\mathbf{I}^{hires}$7 with CFA plus CFD. CLAN reports state-of-the-art classification accuracy on CUB-200-2011, Stanford Cars, and FGVC-Aircraft, including $\mathbf{I}^{hires}$8 for the ResNet-50 backbone and $\mathbf{I}^{hires}$9 for ResNet-101 (Li et al., 2020, Huang et al., 2022).

For multimodal reasoning, DeepStack reports average gains of $$\mathbf{X}^{stack} = \{\mathbf{X}^{stack^1}, \mathbf{X}^{stack^2}, \dots, \mathbf{X}^{stack^s}\} = \mathrm{Sampling2D}\left(\mathcal{M}(\mathcal{F}^v(\mathbf{I}^{hires}))\right),$$0 and $$\mathbf{X}^{stack} = \{\mathbf{X}^{stack^1}, \mathbf{X}^{stack^2}, \dots, \mathbf{X}^{stack^s}\} = \mathrm{Sampling2D}\left(\mathcal{M}(\mathcal{F}^v(\mathbf{I}^{hires}))\right),$$1 across $$\mathbf{X}^{stack} = \{\mathbf{X}^{stack^1}, \mathbf{X}^{stack^2}, \dots, \mathbf{X}^{stack^s}\} = \mathrm{Sampling2D}\left(\mathcal{M}(\mathcal{F}^v(\mathbf{I}^{hires}))\right),$$2 benchmarks for 7B and 13B variants, with particularly large gains on OCR-heavy tasks; CLI raises LLaVA-OneVision-7B from $$\mathbf{X}^{stack} = \{\mathbf{X}^{stack^1}, \mathbf{X}^{stack^2}, \dots, \mathbf{X}^{stack^s}\} = \mathrm{Sampling2D}\left(\mathcal{M}(\mathcal{F}^v(\mathbf{I}^{hires}))\right),$$3 to $$\mathbf{X}^{stack} = \{\mathbf{X}^{stack^1}, \mathbf{X}^{stack^2}, \dots, \mathbf{X}^{stack^s}\} = \mathrm{Sampling2D}\left(\mathcal{M}(\mathcal{F}^v(\mathbf{I}^{hires}))\right),$$4 and LLaVA-1.5 partial sums from $$\mathbf{X}^{stack} = \{\mathbf{X}^{stack^1}, \mathbf{X}^{stack^2}, \dots, \mathbf{X}^{stack^s}\} = \mathrm{Sampling2D}\left(\mathcal{M}(\mathcal{F}^v(\mathbf{I}^{hires}))\right),$$5 to $$\mathbf{X}^{stack} = \{\mathbf{X}^{stack^1}, \mathbf{X}^{stack^2}, \dots, \mathbf{X}^{stack^s}\} = \mathrm{Sampling2D}\left(\mathcal{M}(\mathcal{F}^v(\mathbf{I}^{hires}))\right),$$6; and SCVM reports $$\mathbf{X}^{stack} = \{\mathbf{X}^{stack^1}, \mathbf{X}^{stack^2}, \dots, \mathbf{X}^{stack^s}\} = \mathrm{Sampling2D}\left(\mathcal{M}(\mathcal{F}^v(\mathbf{I}^{hires}))\right),$$7 on DocVQA versus $$\mathbf{X}^{stack} = \{\mathbf{X}^{stack^1}, \mathbf{X}^{stack^2}, \dots, \mathbf{X}^{stack^s}\} = \mathrm{Sampling2D}\left(\mathcal{M}(\mathcal{F}^v(\mathbf{I}^{hires}))\right),$$8 for baseline LLaVA-1.5-7B, $$\mathbf{X}^{stack} = \{\mathbf{X}^{stack^1}, \mathbf{X}^{stack^2}, \dots, \mathbf{X}^{stack^s}\} = \mathrm{Sampling2D}\left(\mathcal{M}(\mathcal{F}^v(\mathbf{I}^{hires}))\right),$$9 on MME versus $2$0, and $2$1 on SQA while preserving the LLM and visual token budget. SCVM also reports POPE $2$2, which the paper frames as compatible with hallucination mitigation (Meng et al., 2024, Chen et al., 15 Jan 2026, Liu et al., 28 Feb 2026).

For multi-scale object detection, CFSAM raises SSD300 from $2$3 mAP to $2$4 mAP on PASCAL VOC and from $2$5 [email protected] to $2$6 on COCO, while the paper reports faster convergence and a practical runtime of $2$7 FPS with $2$8 GFLOPs and $2$9 MB parameters. The ablations show that applying the module to all six SSD prediction layers performs best, reinforcing the claim that the whole feature hierarchy should be modeled jointly (Xie et al., 16 Oct 2025).

6. Broader extensions, misconceptions, and scope

The phrase “cross-layer feature injection” is not confined to visual token stacking. In biomedical prediction, CrossADR uses cross-layer feature integration and cross-level associative learning in a hierarchical pipeline from chemical descriptors through knowledge-graph propagation to a learnable ADR embedding space for $$\mathcal{B}^V = \{\mathcal{P}^{V^1},\dots,\mathcal{P}^{V^n}\}$$0 organ systems. Its gated residual-flow update

$$\mathcal{B}^V = \{\mathcal{P}^{V^1},\dots,\mathcal{P}^{V^n}\}$$1

is explicitly described as the main injection mechanism, and the model preserves all graph-layer states for later fusion rather than collapsing to the last layer. CrossADR reports ROC-AUC $$\mathcal{B}^V = \{\mathcal{P}^{V^1},\dots,\mathcal{P}^{V^n}\}$$2 and PR-AUC $$\mathcal{B}^V = \{\mathcal{P}^{V^1},\dots,\mathcal{P}^{V^n}\}$$3 on CrossADR-Dataset D and ROC-AUC $$\mathcal{B}^V = \{\mathcal{P}^{V^1},\dots,\mathcal{P}^{V^n}\}$$4 on Dataset R (Cheung, 16 Mar 2026).

In hardware reliability assessment, ENFOR-SA uses “cross-layer simulation” in a different sense: most DNN inference runs in PyTorch, while only the relevant systolic-array mesh is simulated in RTL during matrix multiplication and transient fault injection. The altered outputs are then returned to the DNN pipeline. The framework reports RTL-accurate fault injection with only $$\mathcal{B}^V = \{\mathcal{P}^{V^1},\dots,\mathcal{P}^{V^n}\}$$5 average slowdown compared to software-based injection, an average $$\mathcal{B}^V = \{\mathcal{P}^{V^1},\dots,\mathcal{P}^{V^n}\}$$6 speedup over full-SoC RTL simulation, and a $$\mathcal{B}^V = \{\mathcal{P}^{V^1},\dots,\mathcal{P}^{V^n}\}$$7 improvement over HDFIT (Tonetto et al., 31 Jan 2026). This broadens the phrase from representational fusion to the selective coupling of abstraction layers.

Several misconceptions are therefore incorrect. Cross-layer injection is not identical to simple concatenation; the surveyed methods are introduced precisely because concatenation, summation, or single-point projection are considered too restrictive. It is not always a late-fusion technique; some methods intervene during representation formation itself, as in SCVM’s recurrent memory and TAG, or DeepStack’s repeated layer-wise residual infusion. It is also not uniformly beneficial regardless of placement or density: DeepStack reports that injections placed too deep hurt performance, and CLI reports that single-point injection performs worst while high-density injection performs best, with medium density sometimes worse than sparser alternatives (Meng et al., 2024, Chen et al., 15 Jan 2026).

Taken together, the literature presents DeepStack cross-layer feature injection as a general architectural response to hierarchical information loss. In one line of work, it means bottom-to-top stacking of aligned visual token groups across transformer depth; in another, it denotes adaptive cross-layer mixing, redistribution, or feedback; and in broader applications it refers to coupling representations across biological, algorithmic, or hardware layers. The unifying property is repeated access to intermediate structure, so that later computation can remain conditioned on information that standard deep pipelines would otherwise compress, dilute, or discard.