Tapered Language Model (TLM) Principle

• Tapered Language Model (TLM) is an approach that redistributes MLP width across layers, tapering capacity from early to deeper layers under a fixed parameter budget.
• It uses monotonic schedules (e.g., cosine, linear, sigmoid) to assign greater capacity to early layers, yielding significant improvements in perplexity and downstream tasks.
• TLM offers a scalable, architecture-agnostic strategy for optimizing layerwise capacity allocation, enabling performance gains without increasing overall computational cost.

The Tapered LLM (TLM) principle is an architectural strategy for neural LLMs that reallocates per-layer capacity non-uniformly along depth under a fixed total parameter and computational budget. In contrast to the long-established practice of distributing parameters identically across layers—as in the canonical transformer and its descendants—TLMs monotonically reduce (taper) a selected layer dimension, most commonly the MLP intermediate width, as a function of depth. This approach is motivated by the empirical observation that different layers contribute asymmetrically to model output, with early layers carrying greater capacity requirements for generating novel information and later layers primarily refining or reinforcing the residual signal. The TLM methodology establishes depth-aware capacity allocation as a new, architecture-agnostic axis for LLM design, yielding quantifiable performance improvements in perplexity and downstream tasks at no additional cost (Bayat et al., 22 Jun 2026).

1. Formal Statement of the TLM Principle

Let a LLM with $L$ layers, each acting on a $d$-dimensional residual stream $h_l \in \mathbb{R}^{N \times d}$, interleaves a token-mixing module $\mathcal{M}_l$ and an MLP $\mathcal{F}_l$: $$z_l = h_l + \mathcal{M}_l(h_l), \qquad h_{l+1} = z_l + \mathcal{F}_l(z_l)$$ In prevailing architecture designs, the MLP width $d_{ff}(l)$ is fixed for all $l$, resulting in constant per-layer parameter allocation. The TLM principle instead enforces two constraints on a chosen per-layer dimension $d_C(l)$:

• Monotonicity: $d_C(l+1) \leq d_C(l)$ for all $d$0,
• Budget preservation: $d$1.

For the common configuration where $d$2 denotes MLP width, and $d$3 is the width at layer $d$4, the TLM design defines a monotonically decreasing schedule from $d$5 to $d$6, maintaining the same average as the uniform baseline width. Empirically, three smooth and hardware-aligned schedules were investigated:

• Linear: $d$7
• Cosine: $d$8
• Sigmoid (steepness=10): $d$9

All intermediate widths are quantized to multiples of 16 for efficient GPU utilization (Bayat et al., 22 Jun 2026).

2. Rationale: Capacity Utilization Across Depth

Total MLP parameter count and forward FLOPs are linear in $h_l \in \mathbb{R}^{N \times d}$0; thus, matching the mean width to the baseline preserves overall budget. Controlled experiments, where a 440M-parameter transformer’s layers were partitioned into thirds and wider MLPs were placed in the early, middle, or late blocks (keeping total parameters constant), yielded the following validation perplexities:

• Uniform (all blocks $h_l \in \mathbb{R}^{N \times d}$1 width): 16.28
• Wider-early: 15.96
• Wider-late: 17.29
• Wider-middle: 16.61

This demonstrates that funneling extra width to early layers reduces perplexity, while the opposite allocation is detrimental, indicating higher marginal utility for parameters in early layers. The mechanistic basis is that early MLP blocks in deep autoregressive LMs tend to contribute more orthogonal, information-generative updates to the residual stream, whereas later blocks show higher alignment with their inputs, implying output redundancy (Bayat et al., 22 Jun 2026).

3. Schedules, Architectures, and Empirical Performance

Empirical evaluation of TLMs covered three model sizes (440M, 760M, 1.3B parameters) and four backbones (Transformer, Gated Attention, Hope-attention, Titans). Training settings and task coverage were kept constant across uniform and tapered variants except for the per-layer MLP width:

• Tokenizer: Llama 3, vocab size 32K, context length 4K
• Optimizer: AdamW, peak LR $h_l \in \mathbb{R}^{N \times d}$2, cosine scheduling, weight decay 0.1, batch size 0.5M tokens
• Datasets: In-distribution splits, WikiText-103, LAMBADA, and eight commonsense reasoning benchmarks

Among schedules, cosine tapering consistently yielded the largest perplexity gains. For the 440M Transformer, the best result was with cosine decay from $h_l \in \mathbb{R}^{N \times d}$3 times the baseline width:

• Uniform baseline: 16.28
• Cosine $h_l \in \mathbb{R}^{N \times d}$4: 14.44 ($h_l \in \mathbb{R}^{N \times d}$5 = –1.84)
• Linear $h_l \in \mathbb{R}^{N \times d}$6: 15.96 ($h_l \in \mathbb{R}^{N \times d}$7 = –0.32)
• Sigmoid $h_l \in \mathbb{R}^{N \times d}$8: 16.12 ($h_l \in \mathbb{R}^{N \times d}$9 = –0.16)

At 760M and 1.3B scales (cosine $\mathcal{M}_l$0), tapered models improved WikiText perplexity in 7/8 settings, LAMBADA perplexity in all 8, and average accuracy across eight commonsense benchmarks by 0.3–1.0 points. Long-context retrieval (Needle-in-a-Haystack) also showed no regression and sometimes improved performance (Bayat et al., 22 Jun 2026).

4. Mechanistic Analysis and Interpretation

To clarify why the TLM principle works, the cosine similarity $\mathcal{M}_l$1 and $\mathcal{M}_l$2 were computed layerwise in pretrained GPT-2 models. Both quantities rise with depth, indicating that late-layer updates are increasingly non-novel—i.e., largely reinforcing extant features in the residual stream. This redundancy suggests that high-dimension intermediate representations are underutilized in the deeper layers and can be profitably curtailed. Tapering reallocates these surplus dimensions to early blocks where additional width translates to greater functional expressivity and non-trivial contributions to computation (Bayat et al., 22 Jun 2026).

5. Implementation Guidelines

For practitioners deploying TLMs, the following workflow is recommended:

• Select a monotonically decreasing schedule for MLP width $\mathcal{M}_l$3 ensuring $\mathcal{M}_l$4 and $\mathcal{M}_l$5.
• Use cosine decay ($\mathcal{M}_l$6) as a robust default and round widths to multiples of 16 for throughput-optimized matrix computation.
• Only MLP width undergoes tapering; other dimensions (residual, attention heads, $\mathcal{M}_l$7/$\mathcal{M}_l$8) remain fixed.
• Model and training hyperparameters should otherwise not be altered, ensuring a direct comparison between uniform and tapered models.
• For new depths or architectural variants, sweeping $\mathcal{M}_l$9 ratios in $\mathcal{F}_l$0 is advised to locate the optimal point on the U-shaped perplexity curve (Bayat et al., 22 Jun 2026).

6. Significance and Implications

The TLM principle exposes a previously neglected degree of freedom for neural LLM design: depth-aware capacity allocation, achievable without altering total model size or computational cost. It is universally applicable across transformer-family and newer attention architectures, and can be retrofitted into extant stacks. Empirically, TLMs deliver model perplexity and downstream accuracy improvements solely through schedule-based redistribution of MLP width. A plausible implication is that similar monotonic tapering strategies could be extended to other per-layer dimensions in deep neural architectures. This principle offers a free lever for model engineers seeking to optimize allocation efficiency under static parameter and computational budgets (Bayat et al., 22 Jun 2026).

7. Connections and Distinctions from Alternate "TLM" Usage

The abbreviation "TLM" also appears in the literature referring to an efficient joint training paradigm for NLP from scratch, relying on task-relevance retrieval and eschewing large-scale pretraining (Yao et al., 2021). However, in this context, "Tapered LLM" specifically denotes architectural tapering along layerwise MLP width under a fixed total budget, and does not refer to training data selection or pipeline modifications. The two usages are conceptually and methodologically distinct; confusion should be avoided by attending to context and the explicit definition of “tapering” as per the architectural principle established in (Bayat et al., 22 Jun 2026).