Cross-Layer Value Routing (CLVR) in Deep Learning

• CLVR is a design principle that routes internal value signals across layers, overcoming the compression limits of standard residual streams.
• DeltaNet-style and Depth-Attention implementations use distinct routing mechanisms—integrating write values into hidden streams and attention modules—to maintain efficiency and KV cache compatibility.
• Empirical studies show modest but consistent improvements in validation loss, perplexity, and zero-shot accuracy in deep sequence models when using CLVR.

Searching arXiv for papers on Cross-Layer Value Routing and closely related cross-layer mechanisms. Cross-Layer Value Routing (CLVR) denotes a class of cross-depth mechanisms in which a model routes value-like internal signals from earlier layers to later ones, rather than relying exclusively on residual addition or layer-local write targets. In recent arXiv usage, the term has two closely related technical instantiations: a refined routing mechanism for DeltaNet-style linear-attention models, introduced explicitly as Cross-Layer Value Routing in "Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing" (Cerruti et al., 8 Jul 2026), and an attention-native value-mixing mechanism in "Depth-Attention: Cross-Layer Value Mixing for LLMs" (Zeng et al., 3 Jun 2026). Across both formulations, the central objective is selective reuse across depth with tighter alignment to the model’s native state geometry and, in the Depth-Attention case, compatibility with the standard key-value cache.

1. Conceptual definition and scope

CLVR addresses a structural limitation shared by several deep sequence architectures: later layers often receive earlier computation only through a compressed hidden or residual stream. In a vanilla decoder, each layer receives a single hidden state via the residual stream, so all earlier-layer computation is compressed into one vector and later layers cannot directly choose a particular earlier layer’s representation (Zeng et al., 3 Jun 2026). In DeltaNet-style linear attention, a different but related issue appears: lower-layer memory updates may contain potentially useful information, but naïvely forwarding the wrong internal signal into the wrong representational space does not reliably help (Cerruti et al., 8 Jul 2026).

The term therefore refers less to a single canonical algorithm than to a design principle: route values, or value-like content proposals, across depth in a way that is aligned with the receiving computation. In the DeltaNet formulation, the routed signal is the write value projected into the shared hidden stream. In the Depth-Attention formulation, the current layer’s query attends over earlier-layer keys at the same token position and recursively mixes earlier depth-mixed values into the value tensor used by self-attention.

Work	Host architecture	Core routed object
"Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing" (Cerruti et al., 8 Jul 2026)	DeltaNet, Gated DeltaNet	write value $v_{l,t}$ projected into residual stream
"Depth-Attention: Cross-Layer Value Mixing for LLMs" (Zeng et al., 3 Jun 2026)	Qwen3-style decoders, looped Transformers	depth-mixed values $\widetilde{V}_\ell$ inside attention

This dual usage is technically coherent because both mechanisms move cross-layer reuse away from purely implicit residual accumulation and toward explicit value routing.

2. Architectural motivation

The immediate motivation for CLVR is selective information flow across depth. Depth-Attention frames the problem by contrasting sequence-wise self-attention, which freely selects across tokens, with standard depth processing, where Transformers merely add each layer’s output to the residual stream (Zeng et al., 3 Jun 2026). This makes the residual pathway good for optimization but not selective. Hidden-state-based cross-layer methods such as DenseFormer, Hyper-Connections / mHC, Attention Residuals, and related depth-reuse methods improve information flow, but they mostly operate on hidden states outside the attention module. The stated drawbacks are extra inference state beyond the KV cache and memory and communication overhead that grows with context, especially when modern LLMs already compress the key-value cache using GQA or MLA.

The linear-attention motivation is more specific. DeltaNet-style layers maintain a recurrent associative memory $W$ with feature-mapped keys $\kappa^{(i)}$, predicted values $\bar v^{(i)} = W^{(i-1)}\kappa^{(i)}$, and delta-rule residuals $r^{(i)} = v^{(i)} - \bar v^{(i)}$. The most direct routing idea, called Cross-Layer Error Residuals (CLER), forwards a lower layer’s write residual into the next layer’s value target, but the reported explanation for its failure is a space mismatch: the routed residual is computed in the lower layer’s memory/value geometry, while the receiving layer has an independently learned value space (Cerruti et al., 8 Jul 2026).

A common implication of these two lines of work is that cross-layer reuse is not only a question of passing more information upward. The representational location of that information matters. Routing into a shared aligned stream, or performing routing inside the attention mechanism itself, is treated as the decisive architectural move.

3. Formal mechanisms

In the DeltaNet-style formulation, CLVR replaces the original CLER design. The earlier CLER rule modifies the current layer’s value target as

$$\tilde{v}_{l,t} = v_{l,t} + \Gamma_l \rho(r_{p(l),t}),$$

after which the current layer computes

$r_{l,t} = \tilde{v}_{l,t} - W_{l,t-1}\kappa_{l,t}.$

This formulation did not improve over matched baselines. The refined CLVR mechanism instead makes two changes: it routes into the aligned hidden stream instead of the per-layer value target, and it forwards the write value $v_{l,t}$ rather than the write error $r_{l,t}$ (Cerruti et al., 8 Jul 2026). The routing is

$\widetilde{V}_\ell$0

where $\widetilde{V}_\ell$1 is the routed internal signal, $\widetilde{V}_\ell$2 is the ordinary hidden/residual stream, and $\widetilde{V}_\ell$3 is a learned per-layer projection. The paper studies both CLER-H, with $\widetilde{V}_\ell$4, and CLVR, with $\widetilde{V}_\ell$5. Because $\widetilde{V}_\ell$6, both candidates live in the same value space and differ only in whether the routed content is the error or the full write value.

Depth-Attention implements cross-layer value routing directly inside the attention module. For decoder layer $\widetilde{V}_\ell$7, with

$\widetilde{V}_\ell$8

the current layer’s query at token position $\widetilde{V}_\ell$9 attends over the keys of the current and earlier layers at the same position: $W$0 The depth-mixed value is then formed recursively as

$W$1

The resulting tensor $W$2 replaces the original value tensor in standard causal self-attention: $W$3 Here, queries and keys are unchanged, and only values are modified. The source set is not all $W$4 in practice, but a sparse strided subset of earlier layers (Zeng et al., 3 Jun 2026).

The contrast between the two formulations is instructive. DeltaNet CLVR projects routed values into the residual stream; Depth-Attention routes values inside attention and stores the mixed values in the normal V-cache slot. Both are explicitly designed to place routed content in a space that later computation can use directly.

4. Cache structure, state, and computational implications

A major distinction of Depth-Attention is that it preserves the standard KV cache structure. The method reuses the standard attention queries, keys, and value-cache slots, stores the depth-mixed values in place of the original values, and keeps the KV cache size the same as a vanilla decoder (Zeng et al., 3 Jun 2026). The paper therefore reports zero persistent inference memory overhead beyond the vanilla KV cache. For long-context prefill at 128K tokens, the reported extra persistent memory is 0 GiB. In the 3B Qwen3-style model, the theoretical extra compute is normalized to about 0.004% of standard one-token forward/decode compute, with 282 KiB extra activation I/O per token. Measured inference with batch size 64, prefill 2048, and decode 2048 gives 526.6 s for vanilla and 532.8 s for Depth-Attention, an overhead of +1.18%; measured training wall-clock is 4.63 s/step for vanilla and 5.15 s/step for Depth-Attention, an overhead of +11.2%.

The linear-attention CLVR paper emphasizes a different efficiency axis. Its broader study compares softmax attention with DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2 in a common recurrent-memory notation, and explicitly notes that the reported speed results measure training throughput and iteration time rather than empirical inference speed (Cerruti et al., 8 Jul 2026). For CLVR itself, the relevant systems point is not KV-cache preservation but the lightness of the routing augmentation: a zero-initialized per-layer projection injects the routed write value into the aligned hidden stream, and every routing run is compared against a matched non-routing baseline with the same host architecture, optimizer, schedule, and seed.

These two efficiency profiles suggest different deployment regimes. Depth-Attention is optimized for cache-compatible decoder inference; DeltaNet CLVR is framed as a lightweight recurrent-memory augmentation whose measured effect is expressed primarily through validation loss rather than cache-state accounting.

5. Empirical behavior

The empirical record for CLVR is favorable but heterogeneous across architectures.

For DeltaNet-style linear attention, the reported effect is small but consistent validation-loss improvement when routing the write value into the hidden stream (Cerruti et al., 8 Jul 2026). At 350M parameters / 1B tokens, CLVR improves over matched baselines by $W$5 for Gated DeltaNet and $W$6 for DeltaNet; at 350M / 15B tokens, the gains are $W$7 and $W$8, respectively; and at 1.3B / 40B tokens for Gated DeltaNet, the gain is $W$9. The paper explicitly states that the improvement is real but modest, and that it shrinks as the model gets larger or trains longer. By contrast, the naïve DeltaNet-inspired routing variant CLER did not improve over matched baselines in the reported 350M / 1B-token comparisons and did not help under AdamW either.

Depth-Attention reports larger absolute gains on Qwen3-style decoders trained from scratch on 32B tokens from The Pile (Zeng et al., 3 Jun 2026). At 1.5B, it achieves PPL 7.77, compared with 8.17 for vanilla, and average zero-shot accuracy 53.60, compared with 51.26 for vanilla. At 3B, it reports PPL 7.25 versus 7.55 for vanilla, average zero-shot accuracy 55.69 versus 53.76, and five-shot average accuracy 56.17 versus 54.16. The paper states that Depth-Attention attains the lowest perplexity and the highest average downstream accuracy among the tested baselines at both scales, with improvements over the vanilla Transformer by up to 2.3 accuracy points. The gains also hold from 360M to 3B and extend to looped Transformers; on a 500M looped Transformer with three loops, validation loss improves from 2.208 to 2.194.

The ablation evidence in Depth-Attention further constrains the interpretation. A half-depth stride performs best in the reported 500M ablation; too sparse is worse and too dense is also not better. Uniform mixing over depth sources helps, but adaptive softmax mixing helps more. In the “what state should be mixed?” comparison, value-only is best, outperforming key-value and key-only mixing. This supports the specific design choice to preserve current-layer keys and mix only values.

6. Relations, misconceptions, and terminological boundaries

CLVR is best understood relative to two neighboring literatures: cross-layer reuse in neural sequence models and cross-layer routing in communications networks.

Within neural architectures, CLVR differs from DenseFormer, Hyper-Connections / mHC, and Attention Residuals because those methods mostly operate on hidden-state streams or residual streams outside the attention module, whereas Depth-Attention routes values inside attention and remains KV-cache compatible (Zeng et al., 3 Jun 2026). In the linear-attention setting, the key misconception is that any lower-layer internal signal is equally suitable for routing. The CLVR results argue against that view: forwarding the delta-rule write error into the next layer’s value target does not reliably improve performance, while routing the full write value into the aligned hidden stream does (Cerruti et al., 8 Jul 2026).

The term also sits near a much older networking tradition of cross-layer routing, but the object being routed there is different. "On Energy Efficient Hierarchical Cross-Layer Design: Joint Power Control and Routing for Ad Hoc Networks" couples physical-layer power control with network-layer route selection through information sharing, while "VL-ROUTE: A Cross-Layer Routing Protocol for Visible Light Ad Hoc Network" couples route reliability with MAC-layer link negotiation through the Route Reliability Score and a utility-based three-way handshake (0704.3588, Jagannath et al., 2019). These works are cross-layer in the OSI/networking sense: they exchange power, interference, blockage, or access information across protocol layers. They do not route value tensors or write values across model depth.

A separate terminological boundary is the acronym itself. CLVR is also used for Clever Look-ahead Volatility Reduction in automated market makers, where it denotes a greedy transaction-ordering algorithm for minimizing AMM price volatility around an opening price (McLaughlin et al., 2024). That usage is unrelated to cross-layer value routing in machine learning.

Taken together, the recent literature suggests a narrow but technically clear meaning for Cross-Layer Value Routing in sequence modeling: route the right signal, and route it into the right space. In DeltaNet-style models, that means projecting the write value into the shared residual stream. In Depth-Attention, it means performing depth-wise selection over keys at a fixed token position and storing recursively mixed values in the standard attention pathway.