A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention

Abstract: Length-dependent logit rescaling is widely used to stabilize long-context self-attention, but existing analyses and methods suggest conflicting inverse-temperature laws for the context length $n$, ranging from $(\log n)^{1/2}$ to $\log n$ and $(\log n)^2$. We provide a general theory showing that the desirable scale is determined by the gap-counting function $N_n$ of each attention row. Counting how many competitors lie within each gap from the maximum, we define an upper-tail accumulation scale and prove that it gives the critical inverse-temperature scale for softmax concentration: below this scale, the top competitors remain unseparated, whereas above it, the attention entropy collapses. This framework unifies prior scaling laws as different $N_n$ and yields a direct diagnostic for attention-score families, from idealized theoretical models to more practical transformers.

• The paper introduces a deterministic gap-counting framework to unify disparate inverse-temperature scaling laws in self-attention.
• It details empirical validation using models like Qwen-7B-Chat and nanoGPT to identify optimal scaling exponents and softmax regimes.
• The study provides actionable diagnostics by relating attention score gap statistics to model stability and performance on long-context tasks.

A Unified Framework for Critical Scaling of Inverse Temperature in Self-Attention

Introduction and Motivation

Inverse-temperature rescaling of attention logits—essential for ensuring numerical stability and preventing capacity loss in long-context transformers—has been guided by disparate theoretical prescriptions, ranging from $\beta_n \propto (\log n)^{1/2}$, as motivated by Gaussian logit analyses, to $\beta_n \propto \log n$ and even $\beta_n \propto (\log n)^2$, as found in certain engineered context-extrapolating systems. The selection of the precise functional form for $\beta_n$ has thus far lacked a unifying, data-driven principle capable of reconciling the conflicting scaling laws and accommodating arbitrary attention score statistics.

This work introduces a deterministic, gap-counting theory for attention logits, viewing the scaling problem through the lens of the cumulative gap-counting function $N_n(t)$, which measures the number of scores within $t$ of the row-wise maximum. The exponential rate at which $N_n(t)$ grows with $t$—the upper-tail accumulation scale $\Lambda_n$—determines the critical inverse-temperature separating softmax regimes: above this threshold, attention entropy collapses, while below it, dominant competitors are insufficiently separated.

Empirical Analysis: Inverse-Temperature Scaling Is Not Universal

Contrary to several widespread implementations and prior theoretical arguments, the log-context scaling $\xi_\beta = 1$ (i.e., $\beta_n \propto \log n$0) is not empirically optimal across all settings. Experiments on Qwen-7B-Chat at extrapolated context lengths (with dynamic-NTK RoPE) and on GPT-124M trained with a learnable $\beta_n \propto \log n$1 vector reveal significant variability in the optimal exponent $\beta_n \propto \log n$2. Inference-time perplexity on PG-19 and Proof-Pile-2 identifies optima around $\beta_n \propto \log n$3, notably higher than the classic logarithmic law. Conversely, fits to learned temperature vectors in nanoGPT models yield $\beta_n \propto \log n$4, below the standard log scaling.

Figure 1: Perplexity on PG-19 as a function of the log-scaling exponent $\beta_n \propto \log n$5, demonstrating that $\beta_n \propto \log n$6 achieves lower perplexity at long context extrapolation.

These findings support the key claim: critical scaling of inverse temperature is governed by score-row statistics—not by context length alone. The demand for a data-driven calibration principle for $\beta_n \propto \log n$7 leads directly to the gap-counting perspective.

Theoretical Framework: Gap-Counting and the Upper-Tail Accumulation Scale

A central object of the theory is the cumulative gap-counting function $\beta_n \propto \log n$8, which encodes, for every $\beta_n \propto \log n$9, the number of positions $\beta_n \propto (\log n)^2$0 where the gap between the row-max score and $\beta_n \propto (\log n)^2$1 does not exceed $\beta_n \propto (\log n)^2$2. This function's large-$\beta_n \propto (\log n)^2$3 expansion is dominated by the exponential envelope $\beta_n \propto (\log n)^2$4, where the smallest such rate $\beta_n \propto (\log n)^2$5 is defined as the upper-tail accumulation scale.

Figure 2: Illustration of $\beta_n \propto (\log n)^2$6 and the exponential envelope $\beta_n \propto (\log n)^2$7 for a two-level block-constant gram matrix. The tangent point (contact point) determines $\beta_n \propto (\log n)^2$8 and the critical scaling.

A sequence $\beta_n \propto (\log n)^2$9 is then:

• Subcritical if $\beta_n$0: softmax does not sufficiently separate high competitors.
• Supercritical if $\beta_n$1: entropy collapses, softmax becomes overly concentrated.

$\beta_n$2 thus serves as the critical scale separating these regimes.

The contact formula relates $\beta_n$3 to the gap-counting curve's geometry:

$\beta_n$4

where $\beta_n$5 is the (largest) contact gap at which $\beta_n$6 achieves the exponential envelope, and $\beta_n$7 quantifies the "mass" of competitors at this gap.

Logarithmic Scaling Exponents and Coordinate Decomposition

Scaling laws for $\beta_n$8 can be characterized as:

$\beta_n$9

The theory shows that any non-degenerate scaling yielding both non-collapsed softmax and non-trivial entropy must match these exponents: $N_n(t)$0.

A crucial result establishes a coordinate decomposition:

$N_n(t)$1

revealing that the observed scaling exponent depends on the growth rate of both competitor counts near the maximum and the typical gaps.

Figure 3: Qwen and nanoGPT cells mapped onto the $N_n(t)$2 plane relative to the $N_n(t)$3 boundary; operational regime (sublogarithmic vs. superlogarithmic scaling) is determined by the sign of $N_n(t)$4.

Empirical attention score statistics from trained transformers validate this decomposition, with the empirically optimally tuned $N_n(t)$5 closely matching the gap-counting prediction $N_n(t)$6 across all tested datasets and models.

Specializations: Equicorrelated and Block-Structured Configurations

The gap-counting abstraction encompasses classical attention configurations and structured extensions. In the equicorrelated ("simplex") case, $N_n(t)$7 jumps sharply at a unique gap, recapitulating $N_n(t)$8 ($N_n(t)$9). In hierarchical block-constant matrices, this framework interpolates between $t$0 and other exponents via block size and gap parameters, offering a unified lens on disparate theoretical models and their scaling phenomenology.

Empirical Validation: Critical Exponent Recovery from Attention Scores

Gap-counting diagnostics applied to both inference-time (Qwen-7B-Chat at $t$1) and training-time (nanoGPT with learned $t$2) settings demonstrate that the critical scaling exponent $t$3 is recoverable from the attention scores alone. The slope of $t$4 versus $t$5 (i.e., $t$6) aligns with optimization-sourced exponents $t$7 to within grid or measurement error.

Figure 4: Loss landscape for the bias-free post-hoc least-squares fit of the learned temperature vector $t$8 as a function of scaling exponent $t$9, demonstrating optimal fit at sublogarithmic scaling in the nanoGPT regime.

Calibration via $N_n(t)$0 demystifies why prior heuristics (Gaussian, equicorrelated, doubled-rescaling/YaRN) yield differing scaling exponents: these are not universal laws, but are driven by dataset/model-specific score gap statistics.

Figure 5: Two equivalent renderings of $N_n(t)$1. (Right) Gap-counting curve and exponential envelope; (Left) sorted competitor gaps and their log-jump form.

Implications and Future Directions

This framework has both immediate practical and theoretical consequences. Practically, it prescribes a direct diagnostic: sample attention scores at representative context lengths, obtain $N_n(t)$2, and fit $N_n(t)$3 to determine the critical $N_n(t)$4 scaling for model stability and optimal performance on long contexts. This unification obviates blind adherence to canonical scaling heuristics, which may be suboptimal or unstable, especially on extrapolation workloads.

Theoretically, the work raises the question of endogenizing the gap-counting statistics: can $N_n(t)$5 be predicted from training dynamics, architecture, positional encoding, or input distribution? A promising direction is to combine recent inference-time and training-time analyses of attention geometry—such as models of token clustering and low-dimensional structure, or those tracing hierarchical structure formation in embeddings—into the present gap-counting paradigm. Doing so would allow predictive control of critical rescaling schedules as a function of model depth, loop time, or data domain.

Conclusion

This study provides a gap-counting-based deterministic framework that unifies and extends prior scaling rules for self-attention inverse-temperature calibration. The critical scaling exponent can be directly diagnosed from empirical gap statistics in attention scores, and is neither universal nor solely a function of context length. This advances both principled model calibration and theoretical understanding of softmax regimes in long-context transformers, setting the stage for adaptive, data-driven scaling rules responsive to actual attention map statistics, and opening avenues for further integration with representation learning dynamics.