Super Activations in Neural and Physical Systems

• Super Activations are a class of potent activation functions and statistics in both neural and physical models, characterized by extreme magnitudes and adaptive mechanisms.
• They encompass phenomena such as massive activations, superexpressive functions, and regularization strategies that stabilize networks and improve convergence rates.
• Recent research highlights their practical use in architectures like transformers and diffusion models, emphasizing enhanced interpretability and performance.

Super Activations refer to a diverse collection of phenomena, methodologies, and theoretical constructs centered on activation functions or activation statistics in neural and physical systems, wherein either the expressivity, magnitude, trainability, interpretability, or functional utility of activations is found to be atypically strong, concentrated, or critical for the system’s performance. Recent literature encompasses theoretical advances in neural expressivity, architectural mechanisms in deep networks, interpretability tools, and physically-inspired models, but converges on the critical importance of the activation function and the quantitative or qualitative regime in which activations are “super” relative to baseline expectations. The following sections detail core concepts, mathematical formalisms, empirical observations, and major subfields in which super activations play a central role.

1. Definition and Taxonomy of Super Activations

Super activations encompass phenomena where:

• Individual activation values are orders of magnitude larger than their peers in the same layer (massive activations: (Sun et al., 27 Feb 2024, Yu et al., 11 Nov 2024, Gan et al., 24 May 2025, Gallego-Feliciano et al., 5 Aug 2025)).
• Activation functions or families yield vastly enhanced expressivity, convergence, or theoretical approximation guarantees (“superexpressivity”: (Yarotsky, 2021); super-convergence: (Yang et al., 7 Aug 2025)).
• Adaptive, dynamic, or ensemble-based designs empower neurons or layers to select/weight multiple activation types to optimize learning (activation ensembles: (Harmon et al., 2017); adaptive activation: (Lee et al., 2022, Rane et al., 2023)).
• Activation statistics, regularized via explicit penalties, enforce stability and prevent pathologies such as exploding/vanishing states or poor generalization (norm stabilization: (Krueger et al., 2015); smoothness for reproducibility: (Shamir et al., 2020)).
• Mechanisms in physical and chemical models where the dynamics of “activation events” have non-standard, rare, or entropy-driven characteristics (‘super-activated’ transitions in glassy systems: (Baity-Jesi et al., 2021)).

The major axes of classification, illustrated by the research literature, may be tabulated as follows:

Category	Mechanism/Property	Main References
Massive/super outlier activations	Extreme, rare, input-invariant	(Sun et al., 27 Feb 2024, Yu et al., 11 Nov 2024, Gan et al., 24 May 2025, Gallego-Feliciano et al., 5 Aug 2025)
Superexpressive activation families	Constant-size universal approximation	(Yarotsky, 2021)
Ensemble/Adaptive activations	Dynamic mixtures/context adaptation	(Harmon et al., 2017, Lee et al., 2022, Rane et al., 2023)
Regularization/stabilization	Norm control/smoothness/reproducibility	(Krueger et al., 2015, Shamir et al., 2020)
Physical activation (glassiness)	Entropic rarity, not energy barrier	(Baity-Jesi et al., 2021)
Theoretical super-convergence	Superior rates in Sobolev norms	(Yang et al., 7 Aug 2025)

2. Mathematical Formulations and Theoretical Principles

Massive Activations in Transformers and ViTs

Let $h \in \mathbb{R}^d$ be the hidden state; “massive activations” are outliers for which $|h_j| \gg \operatorname{median}_{i} |h_i|$. In practice, only a handful of $j$ produce $h_j$ exceeding $10^3$ or $10^4$ times the typical value (Sun et al., 27 Feb 2024, Yu et al., 11 Nov 2024). These values are largely fixed across sequence inputs and strongly concentrated in particular channels, mathematically acting as implicit bias terms in attention computations.

When $Y = XW^\top$, as in the projection of an MLP block, if $(X_{ik}, W_{jk})$ are both extreme in magnitude, then $Y_{ij} \approx X_{ik} W_{jk}$ is a massive activation. Preservation of these during quantization—by holding out $Y_{ij}$, quantizing the residuals, and restoring—is effective: $$Ȧ = \operatorname{Restore}(Q^{-1}(Q(\operatorname{Replace}(A))))$$ (Yu et al., 11 Nov 2024).

In Diffusion Transformers, massive activations are further shown to be correlated with AdaLN-zero normalization and can be suppressed via adaptive per-channel normalization and selective channel discarding (Gan et al., 24 May 2025).

Superexpressive Families and Super-Convergence

A family $\mathcal{A}$ of activations is “superexpressive” for $d$ if a constant-architecture neural network using $\mathcal{A}$ can approximate any $f \in C([0,1]^d)$ to arbitrary uniform accuracy via merely adjusting the weights, not growing the network (Yarotsky, 2021). This is formally distinct from the standard universal approximation theorem. For example, $\{\sin, \arcsin\}$ is superexpressive, as proven via density arguments related to irrational windings on the torus.

For super-convergence, the error in Sobolev spaces for a target $f\in W^{n,\infty}(\Omega)$ and a $\sigma$–network $\phi$ is

$$\|f - \phi\|_{W^{m,\infty}(\Omega)} \leq C_0 \|f\|_{W^{n,\infty}(\Omega)} N^{-2(n-m)/d} L^{-2(n-m)/d},$$

surpassing classical methods by a factor of two in convergence rate (Yang et al., 7 Aug 2025).

Ensembles, Adaptive, and Smooth Activations

An “activation ensemble” uses a convex combination of normalized candidate activation functions $h_i^j(z)$ at each neuron $i$:

$y_i(z) = \sum_{j=1}^m \alpha_i^j h_i^j(z),$

where $\alpha^j \ge 0$, $\sum_j \alpha^j = 1$, and all parameters are learned per neuron or per layer (Harmon et al., 2017). Similar adaptive mechanisms, e.g., ASH (“Adaptive SwisH”), condition thresholds on local statistics:

$$\operatorname{ASH}(x^{(i)}) = x^{(i)} \cdot S\big(-2\alpha(x^{(i)} - \mu_X - z_k \sigma_X)\big),$$

with $z_k$ trainable for percentile selection (Lee et al., 2022).

Smooth activations (e.g., SmeLU, RESCU) yield continuous derivatives with tunable smoothness, leading to improved reproducibility and accuracy-reproducibility tradeoffs (Shamir et al., 2020).

Regularization/Stabilization

Norm stabilization in RNNs imposes a cost $$\beta (1/T) \sum_{t=1}^T (\|h_t\|_2 - \|h_{t-1}\|_2)^2$$, directly penalizing norm fluctuations over time and preventing both exponential growth and decay in hidden state norms—especially critical for ReLU-based RNNs where eigenvalues of the transition matrix may induce instability (Krueger et al., 2015).

3. Empirical Observations and Architecture-Specific Dynamics

Emergence and Dynamics of Massive Activations

In transformer pretraining, massive activation trajectories follow predictable patterns modeled by

$$f(t) = A \exp(-\lambda x_t) \log(x_t) + K, \quad x_t = \gamma t + t_0,$$

with a five-parameter fit controlling amplitude, decay, timescale, offset, and steady-state value. Shallow and deep layers show early peaking; middle layers exhibit steady logarithmic growth. The emergence and magnitude can be predicted from architectural parameters, enabling proactive design (Gallego-Feliciano et al., 5 Aug 2025).

Massive activations concentrate in specific dimensions and tokens (e.g., start-of-sequence or delimiter tokens), often acting as dominant contributors to the attention pattern. Their near-invariant magnitudes across different inputs supports their interpretation as hard-coded biases required for proper model function (Sun et al., 27 Feb 2024, Gallego-Feliciano et al., 5 Aug 2025).

Role in Quantization, Efficiency, and Debugging

Super activations dominate the numerical scale and, if not properly managed, force an increase in quantization scale, harming inlier representation. Strategies such as outlier-aware quantization—isolating and restoring massive activations or super weights—substantially improve quantized LLM performance, reducing the need for complex per-channel scaling (Yu et al., 11 Nov 2024).

In analog and mixed-signal AI accelerators, as in MACAM-based architectures, super activations correspond to nonlinear analog-to-digital conversion intervals. The SuperMixer framework lets the assignment of analog vs digital activation dataflow be optimized per-channel for energy and accuracy (Zhu et al., 2022).

Adaptive and PWL Activations in Practice

Learned piecewise-linear activations (AdAct) and adaptive thresholding (e.g., ASH) show enhanced approximation of smooth or oscillatory functions relative to classical ReLU, as demonstrated both theoretically and empirically (Lee et al., 2022, Rane et al., 2023).

4. Interpretability, Explanations, and Physical Analogues

Neuro-Activated Superpixels (NAS)

Leveraging the activations of deep image classifiers, superpixels representing semantic structures as identified by the network's hierarchical activations can be generated directly from upsampled and normalized activation maps, avoiding reliance on low-level cues. NAS is shown to enhance weakly-supervised localization performance and enables region-based evaluation of saliency methods, exposing discrepancies in pixel-aggregation metrics (Boubekki et al., 7 Jun 2024).

Physics of Super Activation in Glassy Systems

In supercooled liquids, “super activation” refers to rare, entropic rather than energetic transitions between metabasins in the energy landscape. Inter-metabasin transitions are controlled by the exponential of an entropic barrier,

$\tau \sim \exp(\Delta S_\text{eff}),$

rather than the classical Arrhenius energy barrier. This mechanism dominates glassy slowing down and macroscopic aging due to broad, heavy-tailed trapping times in a few deeply trapped metabasins (Baity-Jesi et al., 2021).

5. Algorithmic and Theoretical Applications

Generalized-Activated Operators in DRL

Generalized weighting in policy-value updates employs non-decreasing activation functions $g$ to “activate” $Q$-values in double critic estimation. The operator

$$GA_g(Q(s, \pi(s));\psi) = \int_{a \in \mathcal{A}} \frac{g(Q(s,a);\psi) Q(s,a)}{\int_{a'} g(Q(s,a');\psi) da'} da$$

allows for parameterized bias correction, interpolating between mean and max operators with theoretical guarantees on operator distance. Task-specific selection of $g$ provides bias correction and accelerates convergence (Lyu et al., 2021).

Fast Kernel Methods and Dual Activations

Super activations also arise in the theory and computation of neural kernels. When exact dual activation expressions are unknown, truncated Hermite series expansions can efficiently approximate any smooth activation’s corresponding NNGP or NTK kernel. With PolySketch-based subspace embedding, 100× speedups in kernel matrix computation are achievable for a wide range of activation families, broadening applicability of kernel-based analysis to diverse architectures (Han et al., 2022).

Universal Approximation and Sobolev Super-Convergence

Construction of fixed-width, deep, general-activation DNNs enables O((NL)^{-2(n-m)/d}) super-convergence in Sobolev norms, suitable for PDEs and scientific computing (Yang et al., 7 Aug 2025). The framework relies critically on the ability of chosen activations to approximate ReLU^k and step functions both globally (quasi-decay) and locally (non-affine smoothness).

6. Implications, Limitations, and Future Directions

Super activations redefine both the theoretical and empirical boundaries of activation design and usage:

• Predictive frameworks now enable architectural preselection to modulate, suppress, or encourage massive activations based on layer depth, attention head density, width/depth ratios, and other design choices (Gallego-Feliciano et al., 5 Aug 2025).
• The synergy between explicit bias mechanisms, normalization schemes, and activation design in transformers and diffusion architectures indicates that model stability and efficiency may be greatly enhanced by systematic activation control (Sun et al., 27 Feb 2024, Gan et al., 24 May 2025).
• In interpretability and evaluation, activation-driven explanations such as NAS provide semantically meaningful structure for saliency analysis, with implications for explainable AI and reliability (Boubekki et al., 7 Jun 2024).
• The identification of superexpressive families and super-convergence rates focuses future research on both new activation designs and theoretical guarantees for DNN-based scientific computing, especially for high-accuracy PDE solvers (Yarotsky, 2021, Yang et al., 7 Aug 2025).

While super activations frequently encode essential computational or representational properties, unchecked or poorly controlled extremes may hinder quantization, optimization, and robustness. Understanding and harnessing super activations thus continues to be central in the ongoing development of scalable, interpretable, and high-performing neural and physical models.