Tapered Language Models: Optimized Architectures

• Tapered Language Models are neural architectures that taper MLP width across layers under a fixed budget, enhancing early feature discovery and overall efficiency.
• The methodology evaluates linear, cosine, and sigmoid tapering schedules, with the cosine approach achieving the most significant perplexity reduction.
• Empirical results across multiple architectures show that reallocating parameters from later to early layers improves both in-distribution and out-of-distribution performance.

Tapered LLMs (TLMs) are a class of neural LLM architectures that depart from the standard practice of allocating uniform parameter capacity across layers. Instead, TLMs optimally redistribute capacity—typically the per-layer MLP (feed-forward) width—such that early layers receive greater width and later layers are narrower, all under an invariant total parameter/FLOPs budget. This monotonic “tapering” approach is motivated by both empirical and mechanistic evidence that early layers are responsible for the discovery of novel features, whereas late layers merely refine or reinforce the activations. The TLM principle is architecture-agnostic, applies to any depth-parameter axis, and has demonstrated consistent perplexity and downstream task improvements with no increase in overall computational cost (Bayat et al., 22 Jun 2026).

1. Theoretical Foundation and Problem Statement

Conventional LLM backbones—across Transformer, recurrent, SSM, and memory-based architectures—consist of $L$ identical layers, with uniform parameter allocation across depth. This paradigm, inherited from the original Transformer, assumes each layer contributes equivalently. However, evidence from early exit, layer-skipping, structural redundancy, and interpretability studies suggests that layers contribute non-uniformly: early layers inject the bulk of feature novelty; later layers predominantly refine the residual stream. The central question addressed by TLMs is whether a uniform capacity allocation is suboptimal given a fixed parameter and FLOPs budget, and whether a depth-sensitive reallocation can yield lower validation perplexity.

Formally, for a per-layer component $C$ (e.g., MLP), let $d_C(l)$ denote the layer-$l$ width:

• Uniform baseline: $d_C(l) = d_C^{\mathrm{baseline}}\ \forall\,l$
• Tapered allocation: $$d_C(l+1)\leq d_C(l)\ \forall\,l;\ \frac{1}{L}\sum_{l=0}^{L-1} d_C(l) = d_C^{\mathrm{baseline}}$$

The objective under this constraint is to minimize the model's validation perplexity (Bayat et al., 22 Jun 2026).

2. Tapered Model Principle and Tapering Schedules

The TLM principle is to select any parameter axis controlled by depth (e.g., MLP width, attention head count, key/value dimensions, memory slots, or experts) and assign it a monotonically decreasing profile across depth, subject to the fixed-budget constraint above. The per-layer MLP intermediate dimension, $d_{\mathrm{ff}}(l)$, is the canonical axis, as MLPs dominate parameter allocation in all major model families.

Three distinct, budget-preserving decays are evaluated for $d_{\mathrm{ff}}(l)$:

• Linear: $$d_{\mathrm{ff}}(l) = d_{\mathrm{start}} - (d_{\mathrm{start}} - d_{\mathrm{end}}) \cdot \frac{l}{L-1}$$
• Cosine: $$d_{\mathrm{ff}}(l) = d_{\mathrm{end}} + \frac{d_{\mathrm{start}} - d_{\mathrm{end}}}{2}\left(1 + \cos\left(\frac{\pi l}{L-1}\right)\right)$$
• Sigmoid: $C$0

Each $C$1 is rounded to a multiple of 16 for hardware efficiency. $C$2 is set to $C$3, $C$4 to $C$5, and interior widths are tuned as needed to ensure the mean constraint.

3. Architectural Instantiations and Parameter Accounting

MLP blocks are the natural point of application, as they account for the majority of parameters across architectures such as Transformer, Gated Attention (output-gated self-attention), Hope-attention (nested self-modifying memory), and Titans (TTL-style attention with neural long-term memory). The parameter count per layer is:

• Uniform: $C$6 (or $C$7 for gated MLPs), total $C$8.
• Tapered: $C$9, with $d_C(l)$0 fixed.

No adjustments are made to token-mixing modules, optimizer state, or learning schedules.

4. Experimental Protocol and Evaluation Benchmarks

Experiments span three pretraining model scales: 440M (30B tokens), 760M (50B tokens), and 1.3B parameters (100B tokens). Four architecture instantiations are evaluated:

1. Transformer (softmax self-attention)
2. Gated Attention
3. Hope-attention
4. Titans

The pretraining and evaluation setup is held strictly identical between uniform and tapered configurations: Llama 3 tokenizer (32K vocabulary), sequence length 4K, batch size 0.5M tokens, AdamW optimizer (cosine annealed LR, peak $d_C(l)$1, weight decay 0.1), and identical FLOPs per model.

Performance metrics include:

• In-distribution validation perplexity (held-out split)
• Out-of-distribution perplexity (WikiText-2, LAMBADA)
• Commonsense reasoning accuracy on eight tasks (LAMBADA-accuracy, PIQA, HellaSwag, WinoGrande, ARC-easy, ARC-challenge, SocialIQA, BoolQ)

5. Empirical Results

The quantitative evaluation reveals that monotonic tapering via a cosine schedule is the most effective. In exhaustive schedule and width ratio sweeps for a 440M-parameter Transformer, a cosine schedule with width tapering from 1.5 to 0.5 (relative to baseline) achieves a perplexity reduction from 16.28 (uniform) to 14.44. Linear and sigmoid schedules are consistently less effective; too mild or too aggressive tapering degrades perplexity.

Taper Range (×baseline)	Cosine PPL	Δ vs. Uniform	Linear PPL	Δ	Sigmoid PPL	Δ
1.50→0.50	14.44	–1.84	15.96	–0.32	16.12	–0.16

At 760M and 1.3B scale (four architectures, ratio 1.5→0.5, cosine schedule), every architecture exhibits reduced perplexity and improved average commonsense accuracy versus uniform:

Model + Tapered	Wiki PPL (Δ)	LAMBADA PPL (Δ)	Commonsense (Δ)
Transformer 760M	21.42 (–0.44)	21.25 (–1.04)	52.84 (+0.59)
Titans 1.3B	15.76 (–0.29)	14.04 (–0.15)	57.08 (+0.35)

All improvements are achieved with strictly fixed parameter and compute budget, confirming that a uniform policy is suboptimal (Bayat et al., 22 Jun 2026).

6. Ablations and Mechanistic Analysis

Three-block reallocations (Early, Mid, Late) affirm that only “front-loading” (assigning surplus width to early layers) is beneficial; both late-emphasis and mid-emphasis degrade performance.

Schedule ablation confirms that a cosine taper—with its soft-plateau and gradual mid-decay—most effectively aligns capacity allocation with empirical utility across depth.

At the mechanistic level, novelty metrics calculated on pretrained GPT-2 reveal that both per-block and per-MLP update novelty (cosine with preceding residual) increase toward the network’s end, indicating later layers predominantly reinforce earlier content. Tapering MLP width exploits this, reallocating capacity from reinforcing (late) to discovery (early) layers for maximal utility under a fixed-budget constraint.

7. Implementation, Constraints, and Broader Implications

Tapered LLMs retain identical parameter counts and FLOPs to their uniform-width counterparts by enforcing:

$d_C(l)$2

No additional memory or computational resources are required, and no changes are made to non-MLP modules or to the training pipeline. Tapering is thus “free” in terms of both hardware and cost, and is readily applicable to a broad range of architectures.

A plausible implication is that further depth-aware parameter reallocations (e.g., head count, memory slot count) may capture additional non-uniformities across other axes of the model stack. The TLM principle exposes a generic, architecture-agnostic design lever for optimizing large-scale LMs that remains hidden within the conventional uniform-width practice (Bayat et al., 22 Jun 2026).