LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws

Abstract: Existing scaling laws for LLMs, predominantly monotonic power laws, fail to explain emerging non-monotonic phenomena such as catastrophic overtraining and quantization-induced degradation, where performance deteriorates despite increased compute. We propose the Shannon Scaling Law, a unified theoretical framework that models LLM training as information transmission over a noisy channel, grounded in the Shannon-Hartley theorem. By mapping model parameters to channel bandwidth and training tokens to signal power, our formulation explicitly captures the interaction between learning signal and intrinsic noise. This perspective reveals a fundamental Shannon capacity for LLMs: scaling model size or data without preserving a sufficient signal-to-noise ratio (SNR) inevitably amplifies noise, inducing a transition from monotonic improvement to U-shaped performance degradation. We validate our theory through experiments on Pythia and OLMo2 under perturbations, including Gaussian noise, quantization and supervised fine-tuning on math, QA and code tasks. The Shannon Scaling Law consistently outperforms classical scaling laws and recent perturbation-aware laws, achieving strong $R^2$ scores and accurately capturing loss basins missed by prior approaches. It also extrapolates: fitted on $\leq$6.9B Pythia models with $\leq$180B tokens, it predicts the unseen 12B model up to 307B tokens at pooled $R^2{=}0.847$, while monotonic baselines collapse.

• The paper introduces the Shannon Scaling Law that frames LLM capacity as a function of SNR, revealing U-shaped loss basins when scaling parameters exceed noise thresholds.
• It empirically validates the law using Gaussian noise, supervised fine-tuning, and quantization, achieving high fidelity (R² > 0.95) in modeling loss degradation.
• The findings challenge traditional scaling laws by demonstrating that beyond a critical SNR limit, increased model size or token count amplifies noise, leading to performance collapse.

LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws

Recasting LLM Training Through the Lens of Shannon Theory

The paper "LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws" (2605.23901) introduces a theoretical paradigm by analogizing the training and scaling of LLMs to the fundamental principles of information transmission as formalized by the Shannon-Hartley theorem. The authors argue that traditional monotonic power-law scaling laws are inadequate for modeling emergent non-monotonic behaviors such as catastrophic overtraining and quantization-induced degradation—phenomena where LLM performance deteriorates even as compute or model size increases.

To address these anomalies, the Shannon Scaling Law is proposed. This law conceptualizes model size as channel bandwidth, token count as signal power, and rigorously incorporates sources of noise arising from data, model structure, and perturbations. The law explicitly ties model capacity to Signal-to-Noise Ratio (SNR), stipulating a fundamental threshold: scaling parameters or data beyond the critical SNR limit amplifies noise and triggers U-shaped performance curves.

Figure 1: Loss landscapes between pretraining and downstream SFT. Pretraining is monotonic, whereas SFT reveals a loss basin signifying U-shaped scaling failure.

Structural Mapping: Shannon Communication Model and LLMs

The paper grounds its analogy in the canonical Shannon-Weaver communication model, drawing explicit structural correspondences between transmitter-channel-receiver and the LLM pipeline: pretraining modulates information into weights, inference transmits information from context to output, and system noise bounds achievable performance.

Figure 2: Structural correspondence between Shannon communication model and LLMs.

The Shannon-Hartley theorem’s expression for channel capacity is adapted to LLMs:

$$C_{\text{LLM}} = aN^\alpha \log_2 \left( 1 + \frac{b D^\beta}{c (DN)^\gamma + d D^\delta + e} \right)$$

Here, $N$ is model size, $D$ is token count, and the denominator aggregates noise contributions from data, the model, and irreducible architectural sources. Loss is directly linked as the reciprocal of capacity, enforcing nonlinear relations that mirror empirical scaling trends.

Empirical Validation: Loss Landscapes and Noise-Dominated Regimes

The Shannon Scaling Law is empirically evaluated against classical monotonic scaling laws (e.g., OpenAI, Chinchilla) and perturbation-aware baselines across a spectrum of perturbations: additive Gaussian noise, quantization, and supervised fine-tuning (SFT) on diverse tasks (math, QA, coding).

Gaussian Noise Perturbation

As noise levels increase, loss landscapes transition from smooth monotonicity to pronounced U-shaped basins. Notably, conventional scaling laws collapse in noise-dominated scenarios, while the Shannon law maintains high fidelity ($R^2=0.9555$ at 10dB noise for Pythia models).

Figure 3: Evolution of Pythia loss contours under varying Gaussian noise levels. Loss basin emerges at high noise.

Figure 4: Fitting performance: Shannon Law vs. QiD Baseline. Shannon Law robustly tracks loss degradation in low SNR, outperforming QiD.

Figure 5: 3D Loss Landscapes: Shannon law demonstrates superior geometric flexibility, aligning with empirical loss even at extreme noise.

Supervised Fine-Tuning and Loss Basins

Under SFT, aggressive optimization noise (increasing learning rates) deforms the loss landscape, again producing U-shaped basins where scaling either tokens or parameters fails to improve loss.

Figure 6: Pythia loss evolution under varying SFT learning rates. Noise-induced U-shaped loss basins.

Figure 7: GSM8K loss contours under SFT: monotonic regime transitions to noise-dominated basin as learning rate escalates.

Figure 8: SiQA loss contours under SFT: breakdown of conventional scaling in noise-dominated basins.

Figure 9: StarCoder loss contours under SFT: persistent breakdown at high learning rates.

Quantization-Induced Degradation

Varying quantization bit-widths demonstrate that aggressive quantization noise similarly collapses the monotonic loss regime, again producing U-shaped basins that only the Shannon law can model without degradation.

Figure 10: Evolution of Pythia loss contours under varying quantization bit-widths. Loss basin formation at low bit-widths.

Numerical Results and Extrapolation

Across models (Pythia, OLMo2) and perturbation types, the Shannon Scaling Law consistently achieves strong $R^2$ scores ($>0.95$). Crucially, it extrapolates: fitted on $\leq$6.9B Pythia models and $\leq$180B tokens, it predicts an unseen 12B model up to 307B tokens at $R^2=0.847$—a regime where monotonic baselines collapse.

Simplified (6-parameter) variants retain near-full explanatory power for single-axis extrapolation, with the full 9-parameter law providing superior accuracy for joint scaling in $N$ and $N$0.

Exponent Analysis: Scaling Dynamics

Analysis of fitted exponents yields several insights:

• At high SNR, bandwidth ($N$1) outweighs model noise ($N$2), favoring model scaling.
• Under perturbation, noise exponent overcomes the bandwidth exponent ($N$3), signifying scaling backfire.
• Along token axis, noise exponent ($N$4) always dominates signal exponent ($N$5), confirming intrinsic U-shaped degradation is universal and not only perturbation-induced.

Practical and Theoretical Implications

The Shannon Scaling Law’s universality and robustness have significant practical implications. It directly challenges brute-force expansion strategies (parameter count, token accumulation) and underscores the necessity of maintaining information density and SNR. Practically, failure to optimize for SNR results in catastrophic performance collapse due to noise amplification. Theoretically, the law unifies monotonic and non-monotonic scaling within a single principled framework, bridging communication theory and deep learning.

Future Directions

Potential extensions include:

• Integrating explicit perturbation parameters ($N$6) for practical scenarios where external noise is controlled.
• Analyzing fitted exponents across novel architectures and tasks to predict scaling thresholds.
• Leveraging model-noise interaction terms to optimize pretraining/fine-tuning schedules, quantization strategies, and architectural choices.

Conclusion

"LLMs as Noisy Channels" establishes the Shannon Scaling Law as a comprehensive framework for understanding and modeling LLM capacity and scaling behaviors. The law robustly captures both monotonic and U-shaped loss regimes under diverse perturbations, providing reliable extrapolation and actionable structural insights. Empirical and extrapolation results decisively favor the Shannon law over classical approaches, indicating a strategic shift toward information-theoretic optimization in LLM scaling and deployment.