Cross-Layer Vision Smoothing (CLVS)

• Cross-Layer Vision Smoothing (CLVS) is a method that sustains key-object visual attention across transformer layers in large vision-language models.
• The approach uses a recursively updated vision memory and uncertainty-aware early termination to ensure stable visual grounding and reduce hallucinations.
• Empirical evaluations on benchmarks like AMBER and R-Bench show significant gains, including up to +31.65% sustained key-object attention in deeper layers.

Cross-Layer Vision Smoothing (CLVS) denotes a class of cross-depth mechanisms for stabilizing and enriching visual representations, and more specifically names a method for Large Vision-LLMs (LVLMs) that maintains sustained focus on key objects by smoothing visual attention across layers with a recursively updated vision memory (Zhao et al., 16 Sep 2025). In its formal LVLM instantiation, CLVS is motivated by the observation that models can accurately locate salient objects, yet that focus is often brief; the method therefore intervenes in layerwise attention dynamics so that historically salient visual tokens continue to influence subsequent layers, with the stated goal of improving visual grounding, reducing hallucinations, and especially strengthening relation and attribute understanding (Zhao et al., 16 Sep 2025).

1. Problem Formulation and Conceptual Scope

The CLVS paper identifies a specific failure mode in LVLMs: attention to key objects is often fleeting, after which the model shifts toward irrelevant or “sink” tokens in later layers (Zhao et al., 16 Sep 2025). This phenomenon is termed advantageous attention decay, and it is presented as a source of weakened visual grounding, particularly for tasks that require reliable attribute binding and relational reasoning rather than mere coarse object detection (Zhao et al., 16 Sep 2025).

Within the broader literature summarized here, the phrase “cross-layer vision smoothing” also functions as a more general organizing concept for methods that regulate representation evolution across depth. In HalluRNN, abrupt representational drift across Transformer layers is treated as a source of hallucination, and the proposed recurrent mechanism is described as achieving adaptive, depth-wise “smoothing” akin to CLVS, but through a learnable recurrent process rather than static averaging or fixed skip connections (Yu et al., 21 Jun 2025). In graph-based vision backbones such as PVG, the same phrase is used to describe mechanisms that prevent excessive feature flattening across layers and preserve diversity under deep propagation (Wu et al., 2023). This suggests that CLVS has both a narrow sense—a particular LVLM attention-smoothing method—and a broader methodological sense involving cross-layer control of visual information flow.

A central distinction in this literature is between post hoc fusion and intervention in representation formation. Static concatenation or aggregation after visual encoding is repeatedly contrasted with methods that act during the layer-by-layer evolution of representations, whether via attention memory, recurrent state propagation, sparse aggregation hubs, or dynamic cross-modal injection (Liu et al., 28 Feb 2026).

2. Core LVLM Mechanism: Vision Memory and Attention Smoothing

In the formal CLVS method, the key device is a vision memory that smooths attention distributions across layers (Zhao et al., 16 Sep 2025). The method begins with unified visual positions in the first layer to remove early-layer positional bias over image tokens:

$$\mathbf{p} = [\{i\}^{N_s}_{i=0}, \{N_s +1\}^{N_v}_{i=0}, \{i+1\}^{(N_s+N_i)}_{i=N_i+1} ]$$

Here, only the visual tokens in the first layer are position-debiased; the stated purpose is to avoid the tendency of LVLMs to concentrate on bottom-right visual tokens due to sequential ordering (Zhao et al., 16 Sep 2025).

For each attention head $h$ and layer $l$, attention weights are written as

$$\bm{\alpha}_h^{(l)} = \mathrm{softmax}\left(\frac{\bm{q}\bm{K}^{\mathsf{T}}}{\sqrt{d}}\right),$$

and are partitioned into visual and non-visual components,

$$\bm{\alpha}_h^{(l)} = \{\bm{\lambda}_h^{(l)}, \bm{\mu}_h^{(l)}\}.$$

The visual memory is initialized in the first layer by max-pooling visual attention across heads:

$$\bm{m}^{(1)}[i] = \max_{1 \le h \le H} \bm{\lambda}_h^{(1)}[i].$$

At subsequent layers, CLVS smooths each head’s visual attention with the previous memory state:

$$\bm{\hat{\lambda}}_h^{(l)} = \beta \bm{\lambda}_h^{(l)} + (1-\beta)\bm{m}^{(l-1)}, \quad 1 \leq h \leq H,$$

with $\beta \in [0,1]$ and default $0.8$ (Zhao et al., 16 Sep 2025). The attention is then re-normalized so that the softmax sum-to-one property is preserved. The memory itself is updated by a leaky averaging rule:

$$\bm{m}^{(l)}[i] = \gamma \bm{m}^{(l-1)}[i] + (1-\gamma)\max_{1 \leq h \leq H}\bm{\lambda}_h^{(l)}[i],$$

with $h$0, reported as default between $h$1 and $h$2 in the ablations (Zhao et al., 16 Sep 2025).

The intended effect is not uniform persistence of all visual evidence, but selective persistence of historically salient visual tokens. In the authors’ formulation, the model’s current visual attention jointly considers the present layer’s evidence and the accumulated memory from previous layers, thereby maintaining smooth attention on key objects across depth rather than allowing abrupt decay (Zhao et al., 16 Sep 2025).

3. Uncertainty-Aware Scheduling and Pipeline Integration

CLVS is not designed as an all-layer intervention. The paper states that visual understanding primarily occurs in the early and middle layers of the model, whereas adjusting visual attention in the final Transformer layers is counterproductive because those layers mainly serve language generation rather than visual grounding (Zhao et al., 16 Sep 2025). On that basis, the method introduces uncertainty-based early termination.

Uncertainty is defined as normalized entropy over the predicted distribution for the top-$h$3 tokens, usually top-10:

$h$4

Beginning from the network midpoint, written as $h$5, CLVS monitors this uncertainty and terminates smoothing once uncertainty falls below threshold $h$6, whose default is $h$7 (Zhao et al., 16 Sep 2025). The stated interpretation is that low uncertainty signals completed visual grounding.

Operationally, the reported inference-time procedure is: concatenate system, image, and user tokens; apply unified visual position to image tokens in the first layer; compute standard attention and initialize vision memory; smooth the visual attention and update the memory at subsequent layers; and, from the midpoint onward, stop smoothing once the uncertainty criterion is met (Zhao et al., 16 Sep 2025). The method is described as both training-free at test time and integrable into fine-tuning, with no architectural changes beyond modification of attention routines (Zhao et al., 16 Sep 2025).

This design places CLVS among cross-layer methods that regulate internal dynamics rather than merely fusing outputs. A closely related idea appears in SCVM, which introduces a recursively updated cross-layer memory state inside the vision encoder and a layer-wise feedback modulation mechanism to regulate the representation evolution trajectory; however, SCVM acts inside the vision encoder and includes an auxiliary semantic alignment objective, whereas CLVS intervenes directly in LVLM attention distributions (Liu et al., 28 Feb 2026).

4. Empirical Behavior on Hallucination and Visual Understanding Benchmarks

The CLVS paper evaluates the method on four benchmarks—AMBER, R-Bench, POPE, and MME—across LLaVA-1.5-7B/13B, LLaVA-Next, and Qwen2.5-VL (Zhao et al., 16 Sep 2025). The reported pattern is task-dependent: gains are strongest where sustained focus on present objects benefits compositional visual reasoning.

On AMBER, the paper reports state-of-the-art performance across nearly all categories, particularly Relation and Attribute. For LLaVA-1.5-7B, the reported scores are 77.6 on Existence, 72.7 on Attribute, 77.1 on Relation, and 75.0 overall, compared with 71.4, 72.0, 74.1, and 72.1 for VANILLA, and 75.7, 71.9, 75.8, and 73.7 for the previous SOTA (PAI) (Zhao et al., 16 Sep 2025).

On R-Bench, CLVS is reported to achieve the highest F1 and accuracy on all tested models, often with +1 to +2% F1 improvement over the best prior baselines (Zhao et al., 16 Sep 2025). On POPE, the method does not boost overall accuracy much because it is ineffective when objects are absent, but it improves recall by 1–3% for present objects, which the paper interprets as evidence that the mechanism helps when there is a salient visual target to preserve across layers (Zhao et al., 16 Sep 2025). On MME, CLVS is reported to achieve the highest overall score, with notable gains on Color, Count, and OCR, while showing no improvement and sometimes a negative effect on Position (Zhao et al., 16 Sep 2025).

Benchmark	Reported effect	Stated implication
AMBER	SOTA across nearly all categories	Strongest gains in relation and attribute understanding
R-Bench	Highest F1 and accuracy on all tested models	Better relation hallucination handling
POPE	Little change in overall accuracy; recall improves by 1–3%	Benefit depends on presence of relevant objects
MME	Highest overall score; gains on Color, Count, OCR; not on Position	Helps detail-centric visual tasks more than spatial layout

The paper also provides qualitative and statistical analyses of attention dynamics. Baseline LVLMs are described as peaking on key-object attention briefly—often around layer 14—and then losing that focus in deeper layers, while CLVS sustains and strengthens key-object attention for many subsequent layers, including a reported +31.65% attention from layers 15–17 (Zhao et al., 16 Sep 2025). The authors interpret this as direct evidence that cross-layer smoothing increases both the magnitude and duration of object-centered attention.

5. Relations to Hallucination Mitigation and Cross-Depth Memory Methods

CLVS sits within a broader line of work that treats hallucination and grounding errors as failures of depth-wise consistency. HalluRNN proposes an architecture-level solution based on recurrent cross-layer reasoning, in which hidden activations across Transformer layers are treated as a temporal sequence and refined by a shared Dual-Gated Depth Propagation Unit (DG-DPU) (Yu et al., 21 Jun 2025). Its recurrence is written over the inter-layer transition $h$8 and recurrent state $h$9, with the hidden state updated as $l$0 after DG-DPU refinement (Yu et al., 21 Jun 2025). The paper explicitly states that DG-DPU achieves adaptive, depth-wise “smoothing” akin to CLVS.

HalluRNN’s empirical results reinforce the broader premise that cross-layer control can reduce hallucinations. It reportedly improves MME over Vanilla by +20.66, achieves the best overall POPE accuracy and F1 across splits, outperforms DeCO by up to +5.47% in POPE accuracy, and reduces CHAIR hallucination rates to 33.4/11.2 compared with 50.8/14.2 for Vanilla and 42.4/11.6 for DeCO (Yu et al., 21 Jun 2025). In ablation, DG-DPU yields 90.20 POPE accuracy on MSCOCO Random and 82.49 adversarial F1, compared with 82.57 and 76.27 for a GRU and 89.70 and 81.70 for Vanilla (Yu et al., 21 Jun 2025). These results support the claim that not all cross-layer recurrence is equivalent; the specific gating design matters.

A related but architecturally distinct direction is Stateful Cross-layer Vision Modulation (SCVM), which introduces a recursively updated cross-layer memory state inside the vision encoder, a Text-Modulated State Update (TMSU), and a Token-Adaptive Gate (TAG) for memory-driven token refinement (Liu et al., 28 Feb 2026). SCVM is explicitly framed as regulating representation evolution rather than performing static fusion after encoding, and it is reported to achieve consistent improvements on DocVQA, MME, SQA, MMB, and POPE without expanding visual tokens, introducing additional vision encoders, or modifying or fine-tuning the LLM (Liu et al., 28 Feb 2026). This suggests a convergence between CLVS and stateful modulation frameworks around the idea that depth-wise memory can be a control variable for visual grounding.

6. Antecedents in Over-smoothing, Feature Diversity, and Selective Aggregation

The conceptual background for CLVS includes earlier work on over-smoothing in graphs and Transformers. In PVG, over-smoothing is defined as the tendency of node features in deep GNN layers to become excessively similar, causing loss of feature discrimination (Wu et al., 2023). PVG addresses this with Progressively Separated Graph Construction (PSGC), MaxE node aggregation, and GraphLU, and reports that PVG-S achieves 83.0% Top-1 accuracy on ImageNet-1K, surpassing ViG-S by +0.9 while reducing parameters by 18.5%, while PVG-B reaches 84.2%, improving ViG-B by +0.5; on COCO, PVG-S gains +1.3 box AP and +0.4 mask AP over ViG-S (Wu et al., 2023). GraphLU is presented as directly targeting CLVS by preventing excessive feature flattening across layers, and the diversity analysis reports that, with GraphLU, diversity remains at healthy, non-zero levels even at 21 layers (Wu et al., 2023).

A theoretical analogue appears in “Revisiting Over-smoothing in BERT from the Perspective of Graph”, which models self-attention as a normalized adjacency matrix and states that layer normalization plays a key role in over-smoothing (Shi et al., 2022). The paper defines the identical-token subspace

$l$1

and gives a contraction recurrence

$l$2

The reported interpretation is that large $l$3 accelerates convergence toward the low-rank smoothed subspace (Shi et al., 2022). To counter this, the paper proposes Concat Fusion, Max Fusion, and Gate Fusion, with Gate Fusion improving average GLUE performance from 83.8% for vanilla BERT to 85.1%, and SQuAD v1.1 from EM 79.7, F1 87.1 to EM 80.7, F1 88.0 (Shi et al., 2022). Although the domain is NLP rather than vision, the paper explicitly describes these hierarchical fusion strategies as directly analogous to CLVS.

Other related systems emphasize that cross-layer smoothing need not be uniform. Famba-V shows that token fusion applied at all layers harms accuracy, whereas upper-layer token fusion preserves much more accuracy while still saving training time and memory on CIFAR-100 (Shen et al., 2024). SparX similarly argues for sparse, selected cross-layer aggregation through interleaved ganglion layers and normal layers, reporting that SparX-Mamba-T improves VMamba-T from 82.5% to 83.5% Top-1 and that SparX-Swin-T gains +1.3% over Swin-T (Lou et al., 2024). These results are consistent with the CLVS design choice that selective and scheduled smoothing is preferable to indiscriminate depth-wide averaging.

7. Limitations, Misconceptions, and Broader Interpretations

The CLVS results also delimit the method’s scope. The paper explicitly notes little to no improvement when no key object is present, as in pure negative samples for existence hallucination, because there is no salient visual target on which to maintain focus (Zhao et al., 16 Sep 2025). It also states that the method does not directly improve spatial (Position) or language-only tasks, which is consistent with the benchmark breakdown on MME (Zhao et al., 16 Sep 2025).

A common misconception would be to equate smoothing with “more smoothing everywhere.” The CLVS design rejects that view: always smoothing later layers is reported to be less efficient and unnecessary because those layers mainly perform language generation rather than visual grounding (Zhao et al., 16 Sep 2025). Related work reinforces the same point from different angles. In Famba-V, lower-layer fusion gives the lowest memory usage but the biggest accuracy drop, while upper-layer fusion yields the best accuracy-efficiency trade-off (Shen et al., 2024). In SparX, dense connections are reported to increase memory and reduce speed without better accuracy; in VMamba, throughput halves and GPU memory rises by about 1GB under dense connections, while FcaFormer-style spatial cross-attention grows memory by more than 80% (Lou et al., 2024).

Another misconception would be to treat CLVS as synonymous with static feature mixing. Several papers in this corpus distinguish adaptive smoothing from fixed heuristics. HalluRNN emphasizes token-specific, example-adaptive recurrent refinement (Yu et al., 21 Jun 2025). CLI describes a many-to-many bridge between vision layers and LLM layers, with Adaptive Multi-Projection (AMP) and Adaptive Gating Fusion (AGF) allowing the LLM to selectively inject the most relevant visual information based on real-time decoding context; on LLaVA-OneVision, CLI reports 74.5 on LLaVA-in-the-Wild versus 68.0 for the baseline and 70.5 on MME versus 69.5, together with a reported +4.7 gain on OCR-Bench (Chen et al., 15 Jan 2026). This suggests that the broader CLVS paradigm is moving from one-size-fits-all fusion toward stateful, uncertainty-aware, and context-conditioned control of cross-layer information flow.

In that broader sense, CLVS can be read as a unifying theme across several strands of research: prevention of over-smoothing, mitigation of representational drift, maintenance of feature diversity, selective reuse of hierarchical vision features, and explicit control over how visual evidence persists or is reintroduced across depth. The most formal realization of the term remains the LVLM method built around vision memory and uncertainty-aware attention smoothing (Zhao et al., 16 Sep 2025), but the surrounding literature indicates that cross-layer smoothing has become a general framework for reasoning about visual representation dynamics in LVLMs, vision Transformers, Mamba-based backbones, and multimodal fusion systems.