---
title: Cross-Layer Vision Smoothing (CLVS)
url: https://www.emergentmind.com/topics/cross-layer-vision-smoothing-clvs
type: topic
---

# Cross-Layer Vision Smoothing (CLVS)

Cross-Layer Vision Smoothing (CLVS) denotes a class of cross-depth mechanisms for stabilizing and enriching visual representations, and more specifically names a method for Large Vision-Language Models (LVLMs) that maintains sustained focus on key objects by smoothing visual attention across layers with a recursively updated vision memory [2509.12897]. In its formal LVLM instantiation, CLVS is motivated by the observation that models can accurately locate salient objects, yet that focus is often brief; the method therefore intervenes in layerwise attention dynamics so that historically salient visual tokens continue to influence subsequent layers, with the stated goal of improving visual grounding, reducing hallucinations, and especially strengthening relation and attribute understanding [2509.12897].

## 1. Problem Formulation and Conceptual Scope

The CLVS paper identifies a specific failure mode in LVLMs: attention to key objects is often fleeting, after which the model shifts toward irrelevant or “sink” tokens in later layers [2509.12897]. This phenomenon is termed **advantageous attention decay**, and it is presented as a source of weakened visual grounding, particularly for tasks that require reliable attribute binding and relational reasoning rather than mere coarse object detection [2509.12897].

Within the broader literature summarized here, the phrase “cross-layer vision smoothing” also functions as a more general organizing concept for methods that regulate representation evolution across depth. In HalluRNN, abrupt representational drift across Transformer layers is treated as a source of hallucination, and the proposed recurrent mechanism is described as achieving adaptive, depth-wise “smoothing” akin to CLVS, but through a learnable recurrent process rather than static averaging or fixed skip connections [2506.17587]. In graph-based vision backbones such as PVG, the same phrase is used to describe mechanisms that prevent excessive feature flattening across layers and preserve diversity under deep propagation [2308.00574]. This suggests that CLVS has both a narrow sense—a particular LVLM attention-smoothing method—and a broader methodological sense involving cross-layer control of visual information flow.

A central distinction in this literature is between **post hoc fusion** and **intervention in representation formation**. Static concatenation or aggregation after visual encoding is repeatedly contrasted with methods that act during the layer-by-layer evolution of representations, whether via attention memory, recurrent state propagation, sparse aggregation hubs, or dynamic cross-modal injection [2603.00655].

## 2. Core LVLM Mechanism: Vision Memory and Attention Smoothing

In the formal CLVS method, the key device is a **vision memory** that smooths attention distributions across layers [2509.12897]. The method begins with **unified visual positions** in the first layer to remove early-layer positional bias over image tokens:

$$
\mathbf{p} = [\{i\}^{N_s}_{i=0}, \{N_s +1\}^{N_v}_{i=0}, \{i+1\}^{(N_s+N_i)}_{i=N_i+1} ]
$$

Here, only the visual tokens in the first layer are position-debiased; the stated purpose is to avoid the tendency of LVLMs to concentrate on bottom-right visual tokens due to sequential ordering [2509.12897].

For each attention head $h$ and layer $l$, attention weights are written as

$$
\bm{\alpha}_h^{(l)} = \mathrm{softmax}\left(\frac{\bm{q}\bm{K}^{\mathsf{T}}}{\sqrt{d}}\right),
$$

and are partitioned into visual and non-visual components,

$$
\bm{\alpha}_h^{(l)} = \{\bm{\lambda}_h^{(l)}, \bm{\mu}_h^{(l)}\}.
$$

The visual memory is initialized in the first layer by max-pooling visual attention across heads:

$$
\bm{m}^{(1)}[i] = \max_{1 \le h \le H} \bm{\lambda}_h^{(1)}[i].
$$

At subsequent layers, CLVS smooths each head’s visual attention with the previous memory state:

$$
\bm{\hat{\lambda}}_h^{(l)} = \beta \bm{\lambda}_h^{(l)} + (1-\beta)\bm{m}^{(l-1)}, \quad 1 \leq h \leq H,
$$

with $\beta \in [0,1]$ and default $0.8$ [2509.12897]. The attention is then re-normalized so that the softmax sum-to-one property is preserved. The memory itself is updated by a leaky averaging rule:

$$
\bm{m}^{(l)}[i] = \gamma \bm{m}^{(l-1)}[i] + (1-\gamma)\max_{1 \leq h \leq H}\bm{\lambda}_h^{(l)}[i],
$$

with $\gamma \in [0,1]$, reported as default between $0.7$ and $0.8$ in the ablations [2509.12897].

The intended effect is not uniform persistence of all visual evidence, but selective persistence of **historically salient** visual tokens. In the authors’ formulation, the model’s current visual attention jointly considers the present layer’s evidence and the accumulated memory from previous layers, thereby maintaining smooth attention on key objects across depth rather than allowing abrupt decay [2509.12897].

## 3. Uncertainty-Aware Scheduling and Pipeline Integration

CLVS is not designed as an all-layer intervention. The paper states that visual understanding primarily occurs in the early and middle layers of the model, whereas adjusting visual attention in the final Transformer layers is counterproductive because those layers mainly serve language generation rather than visual grounding [2509.12897]. On that basis, the method introduces **uncertainty-based early termination**.

Uncertainty is defined as normalized entropy over the predicted distribution for the top-$|V|$ tokens, usually top-10:

$$
u = \frac{\sum_{k=1}^{|V|} -p_t \log p_t}{\log |V|}.
$$

Beginning from the network midpoint, written as $(L+1)/2$, CLVS monitors this uncertainty and terminates smoothing once uncertainty falls below threshold $\delta$, whose default is $0.5$ [2509.12897]. The stated interpretation is that low uncertainty signals completed visual grounding.

Operationally, the reported inference-time procedure is: concatenate system, image, and user tokens; apply unified visual position to image tokens in the first layer; compute standard attention and initialize vision memory; smooth the visual attention and update the memory at subsequent layers; and, from the midpoint onward, stop smoothing once the uncertainty criterion is met [2509.12897]. The method is described as both **training-free** at test time and integrable into fine-tuning, with no architectural changes beyond modification of attention routines [2509.12897].

This design places CLVS among cross-layer methods that regulate internal dynamics rather than merely fusing outputs. A closely related idea appears in SCVM, which introduces a recursively updated cross-layer memory state inside the vision encoder and a layer-wise feedback modulation mechanism to regulate the representation evolution trajectory; however, SCVM acts inside the vision encoder and includes an auxiliary semantic alignment objective, whereas CLVS intervenes directly in LVLM attention distributions [2603.00655].

## 4. Empirical Behavior on Hallucination and Visual Understanding Benchmarks

The CLVS paper evaluates the method on four benchmarks—**AMBER**, **R-Bench**, **POPE**, and **MME**—across **LLaVA-1.5-7B/13B**, **LLaVA-Next**, and **Qwen2.5-VL** [2509.12897]. The reported pattern is task-dependent: gains are strongest where sustained focus on present objects benefits compositional visual reasoning.

On **AMBER**, the paper reports state-of-the-art performance across nearly all categories, particularly **Relation** and **Attribute**. For **LLaVA-1.5-7B**, the reported scores are **77.6** on Existence, **72.7** on Attribute, **77.1** on Relation, and **75.0** overall, compared with **71.4**, **72.0**, **74.1**, and **72.1** for VANILLA, and **75.7**, **71.9**, **75.8**, and **73.7** for the previous SOTA (**PAI**) [2509.12897].

On **R-Bench**, CLVS is reported to achieve the highest F1 and accuracy on all tested models, often with **+1 to +2% F1 improvement** over the best prior baselines [2509.12897]. On **POPE**, the method does **not boost overall accuracy much** because it is ineffective when objects are absent, but it **improves recall by 1–3%** for present objects, which the paper interprets as evidence that the mechanism helps when there is a salient visual target to preserve across layers [2509.12897]. On **MME**, CLVS is reported to achieve the highest overall score, with notable gains on **Color**, **Count**, and **OCR**, while showing no improvement and sometimes a negative effect on **Position** [2509.12897].

| Benchmark | Reported effect | Stated implication |
|---|---|---|
| AMBER | SOTA across nearly all categories | Strongest gains in relation and attribute understanding |
| R-Bench | Highest F1 and accuracy on all tested models | Better relation hallucination handling |
| POPE | Little change in overall accuracy; recall improves by 1–3% | Benefit depends on presence of relevant objects |
| MME | Highest overall score; gains on Color, Count, OCR; not on Position | Helps detail-centric visual tasks more than spatial layout |

The paper also provides qualitative and statistical analyses of attention dynamics. Baseline LVLMs are described as peaking on key-object attention briefly—often around layer 14—and then losing that focus in deeper layers, while CLVS sustains and strengthens key-object attention for many subsequent layers, including a reported **+31.65% attention from layers 15–17** [2509.12897]. The authors interpret this as direct evidence that cross-layer smoothing increases both the magnitude and duration of object-centered attention.

## 5. Relations to Hallucination Mitigation and Cross-Depth Memory Methods

CLVS sits within a broader line of work that treats hallucination and grounding errors as failures of depth-wise consistency. HalluRNN proposes an architecture-level solution based on **recurrent cross-layer reasoning**, in which hidden activations across Transformer layers are treated as a temporal sequence and refined by a shared **Dual-Gated Depth Propagation Unit (DG-DPU)** [2506.17587]. Its recurrence is written over the inter-layer transition $m_t^i = h_t^{i+1} - h_t^i$ and recurrent state $v_t^i$, with the hidden state updated as $h_t^{i+1} = h_t^i + v_t^{i+1}$ after DG-DPU refinement [2506.17587]. The paper explicitly states that DG-DPU achieves adaptive, depth-wise “smoothing” akin to CLVS.

HalluRNN’s empirical results reinforce the broader premise that cross-layer control can reduce hallucinations. It reportedly improves **MME** over Vanilla by **+20.66**, achieves the best overall **POPE** accuracy and F1 across splits, outperforms **DeCO** by up to **+5.47%** in POPE accuracy, and reduces **CHAIR** hallucination rates to **33.4/11.2** compared with **50.8/14.2** for Vanilla and **42.4/11.6** for DeCO [2506.17587]. In ablation, **DG-DPU** yields **90.20** POPE accuracy on MSCOCO Random and **82.49** adversarial F1, compared with **82.57** and **76.27** for a GRU and **89.70** and **81.70** for Vanilla [2506.17587]. These results support the claim that not all cross-layer recurrence is equivalent; the specific gating design matters.

A related but architecturally distinct direction is **Stateful Cross-layer Vision Modulation (SCVM)**, which introduces a recursively updated cross-layer memory state inside the vision encoder, a **Text-Modulated State Update (TMSU)**, and a **Token-Adaptive Gate (TAG)** for memory-driven token refinement [2603.00655]. SCVM is explicitly framed as regulating representation evolution rather than performing static fusion after encoding, and it is reported to achieve consistent improvements on **DocVQA**, **MME**, **SQA**, **MMB**, and **POPE** without expanding visual tokens, introducing additional vision encoders, or modifying or fine-tuning the language model [2603.00655]. This suggests a convergence between CLVS and stateful modulation frameworks around the idea that depth-wise memory can be a control variable for visual grounding.

## 6. Antecedents in Over-smoothing, Feature Diversity, and Selective Aggregation

The conceptual background for CLVS includes earlier work on **over-smoothing** in graphs and Transformers. In **PVG**, over-smoothing is defined as the tendency of node features in deep GNN layers to become excessively similar, causing loss of feature discrimination [2308.00574]. PVG addresses this with **Progressively Separated Graph Construction (PSGC)**, **MaxE** node aggregation, and **GraphLU**, and reports that **PVG-S** achieves **83.0% Top-1 accuracy on ImageNet-1K**, surpassing **ViG-S by +0.9** while reducing parameters by **18.5%**, while **PVG-B** reaches **84.2%**, improving **ViG-B by +0.5**; on **COCO**, PVG-S gains **+1.3 box AP** and **+0.4 mask AP** over ViG-S [2308.00574]. GraphLU is presented as directly targeting CLVS by preventing excessive feature flattening across layers, and the diversity analysis reports that, with GraphLU, diversity remains at healthy, non-zero levels even at **21 layers** [2308.00574].

A theoretical analogue appears in **“Revisiting Over-smoothing in BERT from the Perspective of Graph”**, which models self-attention as a normalized adjacency matrix and states that layer normalization plays a key role in over-smoothing [2202.08625]. The paper defines the identical-token subspace
$$
\mathcal{M} := \{ \boldsymbol{Y} \in \mathbb{R}^{n \times d} \mid \boldsymbol{Y} = \boldsymbol{eC}, \; \boldsymbol{C} \in \mathbb{R}^{1 \times d} \},
$$
and gives a contraction recurrence
$$
d_{\mathcal{M}}(\boldsymbol{H}_{l+1}) \leq v \, d_{\mathcal{M}}(\boldsymbol{H}_l), \quad
v = \frac{(1 + s^2)(1 + \sqrt{\lambda}hs)}{\sigma_1 \sigma_2}.
$$
The reported interpretation is that large $\sigma_1\sigma_2$ accelerates convergence toward the low-rank smoothed subspace [2202.08625]. To counter this, the paper proposes **Concat Fusion**, **Max Fusion**, and **Gate Fusion**, with Gate Fusion improving average **GLUE** performance from **83.8%** for vanilla BERT to **85.1%**, and **SQuAD v1.1** from **EM 79.7, F1 87.1** to **EM 80.7, F1 88.0** [2202.08625]. Although the domain is NLP rather than vision, the paper explicitly describes these hierarchical fusion strategies as directly analogous to CLVS.

Other related systems emphasize that cross-layer smoothing need not be uniform. **Famba-V** shows that token fusion applied at all layers harms accuracy, whereas **upper-layer token fusion** preserves much more accuracy while still saving training time and memory on **CIFAR-100** [2409.09808]. **SparX** similarly argues for sparse, selected cross-layer aggregation through interleaved **ganglion layers** and **normal layers**, reporting that **SparX-Mamba-T** improves **VMamba-T** from **82.5%** to **83.5%** Top-1 and that **SparX-Swin-T** gains **+1.3%** over **Swin-T** [2409.09649]. These results are consistent with the CLVS design choice that selective and scheduled smoothing is preferable to indiscriminate depth-wide averaging.

## 7. Limitations, Misconceptions, and Broader Interpretations

The CLVS results also delimit the method’s scope. The paper explicitly notes **little to no improvement when no key object is present**, as in pure negative samples for existence hallucination, because there is no salient visual target on which to maintain focus [2509.12897]. It also states that the method **does not directly improve spatial (Position) or language-only tasks**, which is consistent with the benchmark breakdown on MME [2509.12897].

A common misconception would be to equate smoothing with “more smoothing everywhere.” The CLVS design rejects that view: always smoothing later layers is reported to be less efficient and unnecessary because those layers mainly perform language generation rather than visual grounding [2509.12897]. Related work reinforces the same point from different angles. In Famba-V, **lower-layer fusion** gives the lowest memory usage but the biggest accuracy drop, while **upper-layer fusion** yields the best accuracy-efficiency trade-off [2409.09808]. In SparX, dense connections are reported to increase memory and reduce speed without better accuracy; in VMamba, throughput halves and GPU memory rises by about **1GB** under dense connections, while FcaFormer-style spatial cross-attention grows memory by **more than 80%** [2409.09649].

Another misconception would be to treat CLVS as synonymous with static feature mixing. Several papers in this corpus distinguish adaptive smoothing from fixed heuristics. HalluRNN emphasizes token-specific, example-adaptive recurrent refinement [2506.17587]. CLI describes a many-to-many bridge between vision layers and LLM layers, with **Adaptive Multi-Projection (AMP)** and **Adaptive Gating Fusion (AGF)** allowing the LLM to selectively inject the most relevant visual information based on real-time decoding context; on **LLaVA-OneVision**, CLI reports **74.5** on **LLaVA-in-the-Wild** versus **68.0** for the baseline and **70.5** on **MME** versus **69.5**, together with a reported **+4.7** gain on **OCR-Bench** [2601.10710]. This suggests that the broader CLVS paradigm is moving from one-size-fits-all fusion toward stateful, uncertainty-aware, and context-conditioned control of cross-layer information flow.

In that broader sense, CLVS can be read as a unifying theme across several strands of research: prevention of over-smoothing, mitigation of representational drift, maintenance of feature diversity, selective reuse of hierarchical vision features, and explicit control over how visual evidence persists or is reintroduced across depth. The most formal realization of the term remains the LVLM method built around vision memory and uncertainty-aware attention smoothing [2509.12897], but the surrounding literature indicates that cross-layer smoothing has become a general framework for reasoning about visual representation dynamics in LVLMs, vision Transformers, Mamba-based backbones, and multimodal fusion systems.

Source: https://www.emergentmind.com/topics/cross-layer-vision-smoothing-clvs