WAV: Multi-Resolution Block Residual Routing for Deep Decoder-Only Transformers

Abstract: Residual connections are central to training deep Transformers, but standard PreNorm residual streams aggregate sublayer updates with fixed unit weights. Recent Attention Residuals replace this fixed accumulation with content-dependent depth-wise routing, and Block Attention Residuals make the mechanism efficient by routing over block-level residual summaries. However, a single block summary stores only the low-frequency total residual displacement inside a block, discarding directional structure such as attention-vs-MLP imbalance and early-vs-late block dynamics. We propose WAV v1, a lightweight multi-resolution residual routing method for decoder-only Transformers. Instead of representing each block only by its accumulated residual sum, WAV v1 augments every block with two directional detail bases: a phase basis that contrasts attention and MLP updates, and a split basis that contrasts early and late sublayer updates. These bases are routed together with standard block summaries through the same depth-wise softmax mixer, while negative detail-source initialization and detached RMS matching stabilize training. On character-level TinyStories and Text8 language modeling, WAV v1 shows a clear depth-dependent benefit. Although it is not consistently beneficial at 12 layers, it becomes competitive at 24 layers and outperforms all baselines at 48 layers. At 48 layers, WAV v1 reduces validation loss relative to Block AttnRes from 0.4960 to 0.4738 on TinyStories and from 0.9363 to 0.9305 on Text8, with negligible additional parameters. These results suggest that directional residual details, not only block-level sums, are important for scaling residual routing in deeper Transformers.

• The paper introduces WAV v1, which enhances decoder-only Transformers by incorporating phase and split detail bases for improved block residual routing.
• Experimental results show that WAV v1 outperforms Block AttnRes at deeper levels (48 layers) with lower validation loss on TinyStories and Text8.
• The method maintains computational efficiency with minimal parameter increase, highlighting potential for future multi-resolution routing strategies in deep models.

Multi-Resolution Block Residual Routing for Deep Decoder-Only Transformers

Introduction

This paper introduces WAV v1, a multi-resolution extension to Block Attention Residuals (Block AttnRes), targeting improved residual routing in deep decoder-only Transformers. The central innovation is augmenting the single block residual summary with two zero-sum, intra-block directional bases: the phase (attention-vs-MLP) and the split (early-vs-late) bases. This method addresses the hypothesis that a block's mere total displacement omits structure relevant for downstream layer computations, especially as model depth increases. WAV v1’s design leaves attention and MLP modules unaltered and employs conservative initialization and normalization for stability.

Figure 1: Schematic of WAV v1 showing per-block accumulation of state and two directional detail bases, with expanded source routing pool and detail source stabilization.

Motivation and Related Work

The work builds on the observation that standard PreNorm residuals—fixed-weight summations—may obscure or dilute useful dynamics in deep architectures. While alternatives such as ReZero and LayerScale modulate update magnitudes, they do not enable content-dependent routing across multiple prior layers or blocks. Attention Residuals (Team et al., 16 Mar 2026) address this by introducing softmax routing over prior states, but incur prohibitive costs at scale. Block AttnRes offers a tractable compromise, compressing preceding layers into per-block summaries.

However, such block summaries constitute only low-frequency (DC) information, neglecting coarse intra-block trajectories—directional bias (attention vs. MLP) and update timing (early vs. late). WAV v1 proposes that such information, when exposed and routable, is particularly valuable as the residual path deepens (i.e., with increased model depth).

Method

WAV v1 accumulates for each block three bases:

1. The block summary $C_b$ (sum of all sublayer residuals in the block),
2. The phase basis $_b$ (attention - MLP, zero-sum by construction),
3. The split basis $_b$ (first-half - second-half, also zero-sum).

Each detail basis is accumulated online and included as an additional routable source in the depth-wise mixer. To mitigate destabilization at initialization, detail sources receive a significant negative bias and per-block RMS matching, ensuring their influence is negligible unless their contributions are genuinely useful during training. For the final model readout, only embedding and block summaries are used, ensuring inference stability.

Computationally, WAV v1 increases the number of routed sources by a factor of three compared to Block AttnRes but remains in the block-level cost regime (still $O(N)$ for $N$ blocks). The parameter footprint increases minimally.

Experimental Results

WAV v1 was evaluated on character-level language modeling tasks (TinyStories (Eldan et al., 2023) and Text8 [mahoney2011text8]), using 12, 24, and 48-layer GPT decoder-only models. Main competitors included Standard residuals, Block AttnRes, ReZero, and LayerScale.

Results reveal a pronounced depth-dependence:

• At 12 layers, WAV v1 underperforms Block AttnRes, likely due to shallow residual history making directional details uninformative.
• At 24 layers, WAV v1 becomes competitive, sometimes marginally outperforming previous methods.
• At 48 layers, WAV v1 achieves the best validation loss on both datasets, improving over Block AttnRes by 0.0222 (TinyStories) and 0.0057 (Text8).

  Figure 2: WAV v1’s depth-dependent gain over Block AttnRes; its benefit emerges only at sufficient depth.

  

  Figure 3: Final validation loss across depths/datasets. WAV v1 excels in the 48-layer regime; Block AttnRes is superior at shallower depths.

  

  Figure 4: 48-layer validation loss curves demonstrate persistent and growing advantage of WAV v1 on both datasets.

WAV v1’s improvements are not limited to the late phases of training but are consistently evident once the model is sufficiently deep, suggesting that multi-resolution detail routing primarily benefits long residual trajectories.

Analysis

The zero-sum structure of WAV v1’s detail bases ensures that information complementary to the overall block displacement is available for downstream layers. With deeper residual paths, the diversity of possible intra-block directions grows, making such detail crucial—whereas for shallow models, these features may be noisy or redundant.

Comparison with residual scaling methods such as LayerScale and ReZero elucidates that simple modulation of update magnitude is insufficient; the critical component is the expansion of representational capacity for routing over the residual path. Further, by compressing deltas within a block, WAV v1 provides an efficient multi-resolution block-level variant of delta-residual routing (cf. Delta Attention Residuals (2605.18855)).

Limitations and Future Work

Current limitations include the absence of ablation studies for alternate bases, lack of large-scale/token-level evaluations, and missing computational cost analyses sans diagnostics overhead. Future directions should explore learnable or soft orthogonal detail bases, polarity-aligned routing mechanisms, and investigate scalability in larger LLMs.

Conclusion

WAV v1 provides a minimal yet effective extension to block-level residual routing by exposing structured intra-block directionality for deep decoder-only Transformers. The empirical finding that multi-resolution residual routing exhibits strong depth-dependent benefits has theoretical implications for the design of future Transformer architectures: as model depths increase, incorporating structured multi-resolution residual information becomes essential for efficient optimization and generalization.

These results motivate the continued integration of multi-resolution routing strategies and suggest further architectural enhancements for exploiting long-range dependencies in very deep Transformer stacks.