Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing

Abstract: Self-attention lets each token retrieve information from the full context, but its quadratic cost in sequence length limits training and inference at long context. This paper presents a comparative study of softmax attention and four recent recurrent linear-attention architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2. We express these mechanisms in a common recurrent-memory notation, making explicit how they differ in expressivity, memory decay, erase and write control, training throughput, and implementation complexity. Our experiments center on 350M-parameter models trained for 15B tokens, and include optimizer and learning-rate comparisons, hybrid-versus-pure stack comparisons, sequence-length runtime measurements, larger DeltaNet runs at 1.3B and 3B parameters, and a small set of downstream evaluations. The reported speed results measure training throughput and iteration time; we do not provide an empirical inference-speed benchmark. Within the reported 350M-parameter, 15B-token sweep, Kimi Delta Attention with Muon reaches the lowest final validation loss, a pure Gated DeltaNet stack trained with AdamW has the highest normalized training throughput, hybrid stacks generally improve loss at a throughput cost, and Muon consistently lowers final validation loss relative to AdamW in the matched architecture settings we evaluate. We introduce and evaluate lightweight cross-layer routing mechanisms for DeltaNet-style memories. The most natural DeltaNet-inspired formulation, forwarding a lower layer's delta-rule write error into the next layer's value target, does not improve over matched baselines. Routing into the aligned hidden stream and forwarding the write value instead yields a modest improvement in the matched runs we report: Cross-Layer Value Routing (CLVR) lowers final validation loss for both DeltaNet and Gated DeltaNet.

• The paper demonstrates that linear attention with recurrent-memory frameworks alleviates quadratic time complexity inherent in softmax attention.
• It rigorously compares DeltaNet family variants, revealing optimizer-specific performance gains and throughput trade-offs.
• The study introduces cross-layer value routing (CLVR), which modestly improves information propagation and validation loss in deep linear stacks.

Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing

Overview and Motivation

This paper provides a systematic comparison of softmax attention with four recent recurrent linear-attention architectures—DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2—expressed in a unified recurrent-memory framework. The study dissects the algebraic mechanism, expressive power, memory control, training/inference speed, and architectural trade-offs within these families. Central to the investigation is the need to address the prohibitive quadratic time and memory complexity of vanilla softmax attention, particularly as model deployments push toward longer-context language modeling.

The authors also introduce and ablate new cross-layer routing mechanisms for linear attention, aiming to mitigate the layerwise information dilution seen in deep stacks, while preserving the efficiency of linear recurrent architectures.

Recurrent Linear Attention Architectures

Softmax Attention and Its Limitations

Softmax attention provides full-sequence mixing via a normalized kernel, affording high expressivity at the cost of $\mathcal{O}(T^2)$ time and memory for sequence length $T$. Linear attention, by contrast, eliminates explicit token-to-token computations by exploiting feature-map kernel decompositions, reducing both inference and training cost to $\mathcal{O}(T)$ and enabling constant-time per-token computation.

DeltaNet Family

DeltaNet introduces a delta-rule memory update: instead of adding the raw value $v^{(i)}$, it adds the write residual $r^{(i)} = v^{(i)} - W^{(i-1)} \kappa^{(i)}$, where $\kappa^{(i)}$ is the feature-mapped key. This error-correcting approach mitigates interference between overlapping key-value pairs.

Gated DeltaNet incorporates a token-dependent scalar $\alpha^{(i)}$ to decay the previous memory, enabling global forgetting and thus controlling interference over long-range contexts. The memory update consists of selectively forgetting past memory and then writing the current delta.

Kimi Delta Attention increases granularity, replacing scalar forgetting with channel-wise gates $\bm{\alpha}^{(i)}$, allowing selective decay in key-feature space.

Gated DeltaNet-2 further decouples erase and write directions with independent channel-wise gates for erasure $\bm b^{(i)}$ and write $\bm w^{(i)}$, maximizing flexibility in memory editing.

The unifying formalism makes precise the axis along which each variant navigates the design space: selectivity versus broad-forgetting, control granularity (scalar vs. vectorial), and update-specificity (single vs. decoupled erase/write).

Cross-Layer Routing in Recurrent Stacks

Problem Formulation

With increasing architectural depth, valuable signals extracted at lower layers in transformer-style models often diminish, impacting representational fidelity. Previous strategies (e.g., Attention Residuals, Mixture-of-Depths Attention) introduce explicit depth-wise communication, but integrating such mechanisms in linear-attention stacks risks undermining their efficiency.

Proposed Routing Variants

Cross-Layer Error Residuals (CLER) attempts to forward a lower-layer's delta-rule error ($T$0) into the value target of the next routing-capable layer. However, results demonstrate no significant improvement, attributed to space misalignment as the residual propagates through independently learned value spaces.

Figure 1: Cross-Layer Error Residuals (CLER); residuals are forwarded via a side-channel across routing-capable layers, but do not yield improvements when injected into the next layer’s value target.

Cross-Layer Value Routing (CLVR) projects the internal write value ($T$1) into the shared hidden (residual) stream via a learned, zero-initialized projection, ensuring alignment with subsequent layers’ representations.

Figure 2: Cross-Layer Value Routing (CLVR); internal write values are projected into the residual stream, ensuring architectural alignment and accessible signals for all upstream layers.

CLVR leads to small but systematic improvements in final validation loss, providing empirical support for routing internal write values—rather than delta errors—across depth, without sacrificing the linear-time property of host architectures.

Empirical Evaluation

Validation Loss, Throughput, and Scaling

A series of controlled experiments on 350M-parameter models trained up to 15B tokens on FineWeb-Edu (LLaMA2 tokenizer) forms the core empirical sweep. Both pure and hybrid (every third layer) stacks of linear/softmax attention are evaluated, together with optimizer (AdamW, Muon) and learning-rate sensitivity.

Figure 3: Validation-loss scaling for 350M-parameter runs; hybrid Kimi Delta Attention with Muon achieves the lowest recorded loss, with DeltaNet and Gated DeltaNet remaining competitive.

The principal findings are:

• Best final loss: Hybrid Kimi Delta Attention with Muon achieves the best final validation loss (2.273), with DeltaNet (2.299) and Gated DeltaNet-2 (2.345) competitive.
• Throughput: The fastest training is observed in pure Gated DeltaNet with AdamW (normalized speed), but this configuration is outperformed by hybrid stacks in validation loss.
• Muon optimizer benefit: The Muon optimizer consistently reduces final loss across all evaluated architectures (improvements ranging $T$2 to $T$3), superseding AdamW in every matched pair.

Learning-rate sensitivity is significant, with Muon and AdamW variants preferring different regimes; architectural conclusions must be accompanied by compatible optimizer and LR selection.

Figure 4: Learning-rate ablation; optimal learning rate exhibits optimizer-architecture specificity, illustrating necessity for tuning.

Figure 5: Iso-contour loss plots across learning rates and architectures confirm flat optima for Muon and sharply peaked dependence for softmax.

Sequence-Length Scaling

Iteration time as a function of sequence length (4k–32k tokens) demonstrates the sequence-length scaling advantage of linear-attention variants:

• Softmax attention’s iteration time increases +192% from 4k to 32k tokens.
• Hybrid Gated DeltaNet increases +65%, proportional to its fraction of softmax layers.
• Pure Gated DeltaNet increases only +8%, confirming constant-time scaling.

  Figure 6: Iteration-time scaling versus sequence length; linear attention architectures demonstrate markedly slower growth, supporting their suitability for long-context modeling.

Cross-Layer Routing Results

At 350M/1B scale, CLER in its value-injection form offers negligible or negative improvements over baselines; routing into the shared hidden stream (CLER-H, CLVR) materializes modest but consistent loss reductions (e.g., -0.0103 for CLVR-Gated DeltaNet at 1B tokens, diminishing with larger scale and token count), indicating diminishing returns as model capacity or training budget increase. Ablations confirm that the write value, as opposed to residual or modulated variants, is the critical content for effective cross-layer routing.

Implications and Future Directions

The study establishes that the spectrum of linear-attention architectures—by modulating update selectivity, decay, and granular control—creates a multi-objective frontier: no design achieves optimality in both validation loss and throughput. Hybrid stacks recover a portion of the loss gap at the expense of partial speed, while pure linear stacks maximize efficiency. Optimizer and LR choices remain non-separable from architecture in influencing performance.

CLVR represents a theoretically and practically efficient cross-layer routing strategy, improving information propagation in recurrent-memory stacks without undermining linear-time scaling. However, the observed improvements are modest; their persistence at even larger scales or in richer hosts (e.g., Kimi Delta Attention, Gated DeltaNet-2) remains an open question.

Downstream evaluations on HellaSwag, PIQA, and WinoGrande reveal no evidence of quality degradation with CLVR, but do not show conclusive downstream gains either. Stronger benchmarking on tasks with deeper associative memory demands and longer context windows is needed.

Future work should (1) examine CLVR and related routing variants in larger/Kimi/GDN2 hosts, (2) delineate conditions for routing-induced gains, (3) expand iteration-time and inference benchmarks, and (4) incorporate fine-grained logging to disentangle routed signal influence from capacity and learning dynamics.

Conclusion

This paper provides a rigorous, unified analysis of current linear attention architectures, quantifies their scaling/throughput/accuracy trade-offs, and introduces a lightweight, theoretically justified cross-layer routing approach. It demonstrates that with careful choice of write/forget controls and targeted routing, linear-attention stacks can approach or match softmax-attention benchmarks in both throughput and validation loss at practical model scales while preserving their efficiency advantage—a critical consideration for scaling to longer contexts and larger models in modern language modeling.