DEHVF: Hierarchical Visual Feature Fusion

• DEHVF is a vision-language fine-tuning method that dynamically fuses hierarchical visual features to align visual and textual semantics without increasing sequence length.
• The technique employs a lightweight Hierarchical Visual Fuser that selectively projects and embeds multi-level visual information directly into the FFN layers of LLMs.
• The approach is parameter-efficient, training only additional modules such as the fuser and projectors, and demonstrates improved performance on benchmarks like ScienceQA and COCO Captions.

DEHVF, short for Dynamic Embedding of Hierarchical Visual Features for Efficient Vision-Language Fine-Tuning, is a vision-language fine-tuning method for Large Vision-LLMs (LVLMs) that is designed to avoid input sequence expansion while exploiting the hierarchical semantic representations of both visual encoders and LLMs. Its central mechanism is a lightweight Hierarchical Visual Fuser (HVF) that dynamically selects and fuses hierarchical visual features for corresponding LLM layers, after which the fused features are projected, aligned, and directly embedded into the Feed-Forward Network (FFN) of the corresponding LLM layer. The method is presented as parameter-efficient: only the fuser, projectors, and positional embeddings are trained, while the visual encoder and LLM are frozen (Wei et al., 25 Aug 2025).

1. Motivation and problem setting

Conventional LVLMs commonly follow a paradigm that projects visual features and then concatenates them with text tokens to form a unified sequence input for LLMs. The stated drawback is a significant increase in the length of the input sequence, which produces substantial computational overhead because attention computation complexity grows quadratically with sequence length, $\mathcal{O}(n^2)$ (Wei et al., 25 Aug 2025).

The method is motivated by two limitations attributed to prior art. First, cross-attention at every layer adds many parameters. Second, visual features, often only from the final encoder layer, are fed uniformly to all LLM layers, which disregards granularity compatibility and the possibility of more nuanced cross-modal fusion. DEHVF is proposed from the observation that shallow layers capture local, fine-grained details, while deeper layers encode high-level semantics, and that the hierarchical features of image encoders such as CLIP-ViT and LLMs are naturally compatible when aligned by semantic granularity.

A common misconception in this area is that reducing sequence length necessarily requires sacrificing fine-grained visual information. DEHVF is explicitly framed against that assumption: it seeks efficient, flexible cross-modal fusion by injecting suitably matched hierarchical visual features into corresponding LLM layers without increasing sequence length. A plausible implication is that the method treats efficiency and hierarchical semantic matching as coupled design objectives rather than as separate optimization targets.

2. Overall architecture

The framework comprises five main modules: a Visual Encoder, an Input Embedding Network, a Hierarchical Visual Fuser, Dynamic Embedding, and an LLM (Wei et al., 25 Aug 2025).

The visual encoder, exemplified by CLIP-ViT, produces hierarchical features from multiple layers. Given an image $\mathbf{I} \in \mathbb{R}^{C\times H\times W}$, the visual encoder outputs

$$\mathbf{F} = Transformer^{(L_v)}(\mathbf{Z}) \in \mathbb{R}^{L_v \times (N+1) \times D},$$

where $N$ is the number of patches, $D$ is the hidden feature dimension, and a $<\mathrm{cls}>$ token is included for global representation.

The input embedding network is a 2-layer MLP that generates compressed embeddings specific to both the LLM layer and FFN position. For each LLM layer $l$ and FFN position $p$, it produces

$I_{l, p} = h(l_i, p_j),$

with $l_i$ the layer embedding and $\mathbf{I} \in \mathbb{R}^{C\times H\times W}$0 the position embedding.

The HVF then dynamically selects and fuses visual features of the appropriate granularity for each LLM layer. The fused features are projected and aligned, and dynamic embedding injects them directly into the FFN weight matrices of the corresponding LLM layers. The LLM therefore receives dynamically enriched FFN weights for cross-modal fusion during forward computation.

This modular decomposition suggests a separation of concerns: hierarchical feature extraction occurs in the visual encoder, semantic matching is handled by the fuser, and cross-modal incorporation occurs inside the FFN rather than through token-sequence augmentation.

3. Hierarchical visual feature selection and fusion

The core innovation is the HVF, which dynamically fuses features from an appropriate group of visual layers for each group of LLM layers by leveraging alignment in semantic granularity (Wei et al., 25 Aug 2025).

The hierarchy grouping mechanism divides the $\mathbf{I} \in \mathbb{R}^{C\times H\times W}$1 visual encoder layers into $\mathbf{I} \in \mathbb{R}^{C\times H\times W}$2 groups, each holding consecutive layers plus the penultimate layer, and similarly divides the $\mathbf{I} \in \mathbb{R}^{C\times H\times W}$3 LLM layers into $\mathbf{I} \in \mathbb{R}^{C\times H\times W}$4 groups, giving a mapping between language and vision groups. For visual group $\mathbf{I} \in \mathbb{R}^{C\times H\times W}$5,

$\mathbf{I} \in \mathbb{R}^{C\times H\times W}$6

Here, $\mathbf{I} \in \mathbb{R}^{C\times H\times W}$7 denotes the $\mathbf{I} \in \mathbb{R}^{C\times H\times W}$8 feature from visual encoder layer $\mathbf{I} \in \mathbb{R}^{C\times H\times W}$9.

For each LLM layer $$\mathbf{F} = Transformer^{(L_v)}(\mathbf{Z}) \in \mathbb{R}^{L_v \times (N+1) \times D},$$0 in group $$\mathbf{F} = Transformer^{(L_v)}(\mathbf{Z}) \in \mathbb{R}^{L_v \times (N+1) \times D},$$1, the fuser uses $$\mathbf{F} = Transformer^{(L_v)}(\mathbf{Z}) \in \mathbb{R}^{L_v \times (N+1) \times D},$$2 as a query and the visual group’s features as keys and values in a stack of cross-attention Transformer blocks. The output is a normalized weight vector,

$$\mathbf{F} = Transformer^{(L_v)}(\mathbf{Z}) \in \mathbb{R}^{L_v \times (N+1) \times D},$$3

which is then used to generate fused features:

$$\mathbf{F} = Transformer^{(L_v)}(\mathbf{Z}) \in \mathbb{R}^{L_v \times (N+1) \times D},$$4

In the description provided for the method, this dynamic, layer-guided fuser allows each LLM layer to attend to the most relevant visual abstraction. Visualizations are reported to show the dynamic usage of low-, mid-, and high-level visual features as per the language layer’s abstraction needs, which is presented as validation of the method’s motivation.

A common simplification in VL systems is to treat the last visual layer as universally sufficient. DEHVF explicitly rejects that simplification. Its design assumes that cross-modal alignment should vary across depth, with different LLM layers requiring different visual abstraction levels.

4. Dynamic embedding into the FFN

A defining feature of DEHVF is that visual features are injected directly into the FFN’s memory structure instead of being appended as additional input tokens (Wei et al., 25 Aug 2025).

The FFN in a Transformer layer is written as

$$\mathbf{F} = Transformer^{(L_v)}(\mathbf{Z}) \in \mathbb{R}^{L_v \times (N+1) \times D},$$5

where $$\mathbf{F} = Transformer^{(L_v)}(\mathbf{Z}) \in \mathbb{R}^{L_v \times (N+1) \times D},$$6 and $$\mathbf{F} = Transformer^{(L_v)}(\mathbf{Z}) \in \mathbb{R}^{L_v \times (N+1) \times D},$$7 are interpreted as key and value pairs encoding “knowledge.” DEHVF modifies this mechanism by projecting the fused visual features into the LLM hidden space through a projector $$\mathbf{F} = Transformer^{(L_v)}(\mathbf{Z}) \in \mathbb{R}^{L_v \times (N+1) \times D},$$8 and summing them with positional embeddings:

$$\mathbf{F} = Transformer^{(L_v)}(\mathbf{Z}) \in \mathbb{R}^{L_v \times (N+1) \times D},$$9

$N$0

The amended FFN at layer $N$1 is then expressed as

$N$2

Accordingly, the amended FFN’s key/value matrices become

$N$3

and

$N$4

Only the fuser, projectors, and positional embeddings are trained. The visual encoder and LLM are frozen. This is the basis for the method’s characterization as parameter-efficient. A plausible implication is that DEHVF uses the FFN not merely as a generic nonlinear block but as a controllable locus for cross-modal memory augmentation.

5. Parameter efficiency and computational profile

The method is presented as avoiding input sequence expansion and therefore avoiding the quadratic attention cost increase associated with longer multimodal token sequences (Wei et al., 25 Aug 2025).

The added module count is described as small. The paper reports 4.2M new parameters for LLaMA-7B and 5.8M for LLaMA-13B. The HVF is described as lightweight, with 4 cross-attention transformers and hidden dim 64. In a FLOPs comparison shown in Fig. 2, for a typical case with text length 32, DEHVF uses only 6% of LLaVA-v1.5’s FLOPs when increasing the number of visual tokens.

The efficiency claims are tied directly to the architectural choice of embedding visual knowledge into FFN weights instead of appending visual tokens to the input sequence. The reported training and inference speed is close to the fastest baselines and more than 1.5x faster than LoRA-based PEFT, which the paper attributes to avoiding long input sequences.

These results do not imply that DEHVF eliminates the cost of cross-modal fusion; rather, they indicate a reallocation of computation from sequence-level attention toward lightweight hierarchical fusion and FFN-level augmentation. This suggests a design space in which multimodal efficiency depends not only on parameter count but also on where in the Transformer computation graph the fusion occurs.

6. Empirical evaluation

DEHVF is evaluated on multiple vision-language benchmarks, including visual question answering on ScienceQA and image captioning on COCO Captions (Wei et al., 25 Aug 2025).

On ScienceQA, the reported results are:

• LLaMA-7B: 93.54% avg accuracy, stated as more than 0.4% higher than the best prior PEFT method (MemVP) and 3.7% above LaVIN.
• LLaMA-13B: 94.19% avg accuracy, stated as more than 0.4% improvement.
• Parameter count: ~4.2M new parameters for 7B and ~5.8M for 13B.

On COCO Captions, the reported results are:

• BLEU-4: 37.1, stated as beating AdapterV2 and LaVIN by 0.7–0.9.
• CIDEr: 128.9, stated as outperforming AdapterV2 and LaVIN by 2–6 points.
• Parameter/compute efficiency: only ~5.8M trainable params, and less than 1/100th the GPU hours of BLIP-2, while being described as competitive in performance.

The paper’s ablations report that removing hierarchical visual fusion, adaptive weighting, or multi-layer fusion all reduce performance by up to 1–2%. It also reports that injecting prompts into only keys or only values lowers accuracy. These ablations are used to support the claim that the method’s gains depend on the joint use of hierarchical grouping, adaptive fusion, and dual key/value injection.

Taken together, the benchmark results position DEHVF as a PEFT method that seeks a specific balance: higher accuracy than existing PEFT baselines while maintaining efficient training and inference. This suggests that the claimed improvement is not tied to a single dataset or metric, but to a consistent architectural pattern across VQA and captioning tasks.

7. Interpretation and relation to efficient vision-language fine-tuning

DEHVF is defined by three linked commitments: dynamic mapping between LLM layers and visual hierarchy, fusion of multi-layer visual information, and direct embedding into FFN weights instead of token concatenation (Wei et al., 25 Aug 2025).

Its central claim is not simply that hierarchical visual features are useful, but that they should be matched to language-layer semantic granularity. In this formulation, shallow visual signals and deep semantic signals are not competing sources of information; they are layer-appropriate sources that should be dynamically selected. The method also treats FFN weights as a location for multimodal knowledge injection, rather than reserving all cross-modal interaction for self-attention or explicit cross-attention blocks.

The paper frames this as an alternative to two common strategies: prompt concatenation, which expands the sequence, and layerwise cross-attention, which adds many parameters. A plausible implication is that DEHVF belongs to a broader class of methods that reconceptualize the LLM interior as the target of modality alignment rather than as a fixed text processor receiving external multimodal prompts.

Another potential misconception is that parameter-efficient fine-tuning necessarily means using a single low-rank or adapter mechanism uniformly across the network. DEHVF instead distributes a small number of trainable parameters across the fuser, projectors, and positional embeddings, with dynamic behavior varying by LLM layer and FFN position. In that sense, its efficiency is structured rather than uniform.

The method is therefore best understood as a hierarchical, FFN-centric PEFT framework for LVLMs: it preserves frozen backbone models, avoids sequence expansion, and performs cross-modal alignment at the same semantic granularity through dynamic fusion and embedding.