Tapered Language Models

Abstract: Modern LLMs, including transformer, recurrent, and memory-based variants, share a common chassis: a stack of identical layers in which parameters are allocated uniformly across depth. This is a default inherited from the original transformer and largely unchanged since, yet a growing body of evidence suggests that layers contribute non-uniformly to the final output, with later layers refining the residual stream rather than transforming it. We ask whether parameter capacity should reflect this asymmetry. Our controlled experiment shows that, under a fixed budget, allocating more capacity to earlier layers and less to later layers improves perplexity over a uniform-width baseline, while the reverse allocation hurts. Building on this result, we introduce Tapered LLMs (TLMs), an architectural principle in which a parameter-bearing component is monotonically tapered across depth under a fixed total budget. MLPs are the natural site for this instantiation: they dominate parameter count across all modern LM families and expose width as a single, clean axis of variation. Across three model scales and four architectures (Transformer, Gated Attention, Hope-attention, and Titans), tapering MLP width via a smooth cosine schedule consistently improves perplexity and downstream benchmark performance over uniform baselines, at no additional parameter or compute cost. These findings establish depth-aware capacity allocation as a simple, architecture-agnostic axis of LLM design, a free lever hidden in plain sight.

• The paper introduces a depth-aware tapering principle that front-loads MLP capacity to improve validation perplexity by up to 1.84 points at fixed parameter count.
• It systematically compares linear, cosine, and sigmoid tapering schedules, with the cosine schedule consistently outperforming others.
• The approach demonstrates robust improvements across various architectures and scales, challenging the conventional uniform capacity allocation in deep models.

Depth-Aware Allocation in LLMs: The Tapered LLM Principle

Introduction

The standard paradigm in LLM architecture design emphasizes a stack of architecturally identical layers, distributing parameter capacity uniformly across all depths. Despite the empirical and conceptual advances in model scaling, attention variants, and memory architectures, the assumption of uniform per-layer allocation remains largely unchallenged—a default inherited from the original Transformer formulation. However, recent findings indicate non-uniform importance and usage of layers, particularly within large-scale autoregressive decoders, calling into question the rationale for uniform allocation. "Tapered LLMs" (2606.23670) systematically investigates this assumption and formalizes an architectural principle: depth-aware tapering of parameter capacity under a fixed total budget.

Empirical Motivation: Capacity Placement and Asymmetry

An initial controlled experiment partitions a 440M-parameter Transformer into three layer groups (early, middle, late), redistributing the MLP (feed-forward subnetwork) width among groups while holding total parameters and compute constant. Front-loading MLP capacity—concentrating width in earlier layers—significantly improves validation perplexity ($-0.32$), whereas the reverse (back-loading) degrades it ($+1.01$) relative to the uniform baseline. Placement of capacity—not merely its quantity—has clear effects on optimization and generalization.

Figure 1: Perplexity changes for front-, middle-, and back-loaded MLP allocations at equal total parameter count.

Such asymmetry motivates exploration beyond coarse blockwise width changes, pointing toward smooth parameter schedules.

The Tapered LLM (TLM) Principle

TLMs instantiate a simple, general principle: monotonic tapering of a parameter-bearing architectural axis (such as MLP width) from high to low across depth, preserving both total parameter count and total compute. Tapering is not restricted to transformers and can, in theory, be applied to other parameter-dominated axes (attention head count, state sizes, etc.).

In this work, MLP intermediate width is the sole focus. Three tapering schedules are considered:

• Linear: Constant-rate width decay.
• Cosine: Smooth, slow decay at the boundaries; sharper transition in the center.
• Sigmoid: Abrupt switch in a narrow band, close to binary allocation.

  Figure 2: Evolution of MLP intermediate dimension across depth for uniform and three tapered schedules, each constrained to the same parameter budget.

Cosine scheduling consistently outperforms others, optimizing both perplexity and downstream metrics. The best configuration at 440M scale is a start/end width ratio of 1.5/0.5 (relative to the baseline), yielding a 1.84 perplexity point improvement at iso-parameter count and compute.

Figure 3: Per-layer MLP width for uniform and tapered schedules, with corresponding validation perplexities—cosine taper achieves the lowest perplexity.

Generality: Tapering Across Architectures and Scales

Tapering is robust: across four major architectures (standard Transformer, Gated Attention, Hope-attention, Titans) and multiple scales (440M–1.3B parameters), cosine-tapered MLPs yield consistent improvements. The improvement manifests not only in in-distribution and out-of-distribution perplexity but also in mean accuracy across a range of commonsense reasoning benchmarks. This robustness suggests that the gains derive from depth-aware capacity allocation rather than properties idiosyncratic to attention or specific training setups.

Notably, performance improvements do not vanish as model size increases: tapered variants yield better perplexity and benchmark accuracy at both 760M and 1.3B parameter scales.

Mechanistic Analysis: Diminishing Novelty in Depth

To uncover the underlying mechanism, the study analyzes the alignment of MLP and block-wise updates with the residual stream across depth on pretrained models. Cosine similarity between each layer's update and the incoming residual progressively increases with depth—an indicator that late-stage MLPs mostly reinforce existing features with minimal new computation.

Figure 4: Cosine similarity between layer/block updates and the residual stream grows monotonically with layer index; novelty diminishes toward deeper layers.

This pattern is robust across the GPT-2 family and scales. The capacity required for computation decays with depth: early MLPs benefit from higher representation power (orthogonal updates); later MLPs are increasingly redundant in capacity as their updates align more closely with the input stream. Tapering MLP width thus matches empirical utility of parameters to layer necessity, refining the inductive bias.

Implications and Future Directions

Practical Implications

• TLMs constitute a free improvement: Non-uniform width allocation delivers better perplexity and downstream accuracy at matched compute and parameters, making it a directly applicable switch for production and research pipelines.
• Hardware and efficiency: With per-layer widths adjusted (rounded to multiples of 16), the approach maintains hardware efficiency.
• Transferability: The best tapering schedule, determined on a single model scale/architecture, translates effectively to other settings and model families without further tuning—highlighting a design invariant.

Theoretical Implications

• Challenging uniformity: The prevailing uniformity is suboptimal for deep stacked networks, irrespective of the token-mixing module.
• Depth as a non-homogeneous axis: The factual distribution of incremental utility across depths justifies layer-aware parametrization, potentially extending to any depth-wise architectural variable controlling the parameter footprint.
• Paradigm for other modalities: The once-implicit assumption of uniform capacity allocation pervades vision transformers, diffusion models, and multimodal models—suggesting a broad cross-domain design flaw.

Open Questions and Future Work

• Beyond MLP width: The tapering principle could apply to attention head counts, memory slot allocations, or MoE expert routing. Whether similar improvements hold remains untested.
• Dynamic/tunable tapering: Investigating data- or gradient-adaptive tapering, possibly coupled with dynamic inference, may further refine allocation.
• Optimal scheduling as a function of architecture, data, and scale: The best schedule may not be universally fixed—scale-dependent or architecture-specific ablations could uncover further unattained gains.
• Theoretical formalization: The diminishing novelty per depth invites information-theoretic or optimization-theoretic interpretations of effective model depth and parametrization.

Conclusion

The Tapered LLM framework demonstrates that capacity allocation in deep networks should match the empirically observed non-uniform utility of depth. A smooth monotonic taper of MLP width yields measurable, consistent improvements across architecture classes, model sizes, and evaluation settings, without tradeoff in hardware or compute. The analysis of residual alignment provides a mechanistic account motivating the tapering direction. These findings reframe depth as a privileged axis for architectural optimization, inviting both practical deployments and further theoretical generalization.