Plateau Neurons: Mechanisms in AI and Physiology

• Plateau neurons are regimes with nearly constant influence or regenerative activity, defined differently in transformer models, network loss landscapes, cortical physiology, and respiratory dynamics.
• In transformer models, a small subset of final-layer MLP neurons exhibit high, nearly constant ΔLoss, playing a critical role in rare-token prediction.
• In cortical and respiratory systems, plateau phenomena emerge from dendritic Ca²⁺ events and slow ionic feedback, respectively, which alter neuronal spiking patterns.

“Plateau neurons” denotes different but technically precise objects across contemporary literature. In transformer analysis, the term refers to a small “plateau” of final-layer MLP neurons whose mean-activation ablation has nearly constant, exceptionally high influence on rare-token loss. In cortical neurophysiology, it refers to layer-5 pyramidal neurons whose backpropagation-activated calcium firing evokes a regenerative dendritic Ca²⁺ plateau in the apical tuft. Closely related plateau phenomena also arise in two-layer neural-network loss landscapes, where neuron splitting creates an affine family of stationary points, and in respiratory neuron models, where slow ionic feedback generates a plateau potential during ramping bursts (Liu et al., 25 Sep 2025, Rodriguez-Garcia et al., 3 Jul 2025, Ding et al., 3 Jun 2026, Abdulla et al., 2021).

1. Terminological scope

The literature uses “plateau” in several non-equivalent ways. In the transformer setting, the defining observation is a leftmost flat region in a log-log influence plot: a small set of neurons has nearly constant, exceptionally high $\Delta Loss$, and these are formally defined as plateau neurons. In cortical physiology, the defining event is a dendritic membrane potential jump above approximately $-30$ mV that lasts tens of milliseconds and converts single somatic spikes into high-frequency bursts. In two-layer network geometry, the plateau is not a neuron type but an affine family of stationary points created by duplicating a hidden neuron and redistributing its output weights. In respiratory dynamics, the plateau is a stable slow-manifold segment reached after a SNIC-to-homoclinic bifurcation switch, producing ramping spiking on a depolarized plateau (Liu et al., 25 Sep 2025, Rodriguez-Garcia et al., 3 Jul 2025, Ding et al., 3 Jun 2026, Abdulla et al., 2021).

Context	“Plateau” denotes	Principal object
Transformer MLP analysis	Leftmost flat region in neuron influence hierarchy	Rare-token neurons
Layer-5 cortical physiology	Regenerative dendritic Ca²⁺ plateau	BAC-firing pyramidal neurons
Two-layer loss landscape	Affine family of stationary points	Split hidden neuron manifold
Respiratory neuron dynamics	Depolarized plateau during ramping burst	Slow ionic-feedback state

A common source of confusion is to treat these usages as interchangeable. The cited works instead attach the term to distinct geometric, functional, and biophysical structures.

2. Stationary plateaus from neuron splitting

In the loss-landscape setting, the relevant model is a two-layer network

$$f(x;\theta)=v^\top \sigma(Wx), \qquad \theta=(v,W),$$

trained by the squared loss

$L(\theta)=\tfrac12 \|\hat y(\theta)-y\|^2.$

Starting from a narrow network of width $r$ at a stationary $\underline{\theta}$, neuron splitting selects a parent neuron $(\underline{w}_r,\underline{v}_r)$ and replaces it by $m-r+1$ daughter neurons with the same input weight and redistributed output weights. If $\lambda=(\lambda_0,\ldots,\lambda_{m-r})$ with $\sum_{j=0}^{m-r}\lambda_j=1$, then $-30$0 satisfies $-30$1 and $-30$2. The resulting affine family

$-30$3

is the stationary plateau (Ding et al., 3 Jun 2026).

The local geometry of this plateau is controlled by the per-neuron “inner Hessian” matrix. For the $-30$4-th hidden neuron,

$-30$5

with matrix form

$-30$6

Its relation to the full Hessian with respect to $-30$7 is

$-30$8

so $-30$9 captures the “second-derivative” term coming from the network’s nonlinearity, and its eigenvalues govern local curvature induced by asymmetric perturbations of the split daughters (Ding et al., 3 Jun 2026).

For a nondegenerate local minimum of the narrow net with $$f(x;\theta)=v^\top \sigma(Wx), \qquad \theta=(v,W),$$0, the paper classifies all points on the plateau. Writing

$$f(x;\theta)=v^\top \sigma(Wx), \qquad \theta=(v,W),$$1

the conclusions are:

• If $$f(x;\theta)=v^\top \sigma(Wx), \qquad \theta=(v,W),$$2: $$f(x;\theta)=v^\top \sigma(Wx), \qquad \theta=(v,W),$$3 is a local minimum iff $$f(x;\theta)=v^\top \sigma(Wx), \qquad \theta=(v,W),$$4, and a saddle otherwise.
• If $$f(x;\theta)=v^\top \sigma(Wx), \qquad \theta=(v,W),$$5: $$f(x;\theta)=v^\top \sigma(Wx), \qquad \theta=(v,W),$$6 is a local minimum iff $$f(x;\theta)=v^\top \sigma(Wx), \qquad \theta=(v,W),$$7, and a saddle otherwise.
• If $$f(x;\theta)=v^\top \sigma(Wx), \qquad \theta=(v,W),$$8 is indefinite: $$f(x;\theta)=v^\top \sigma(Wx), \qquad \theta=(v,W),$$9 is a saddle for all $L(\theta)=\tfrac12 \|\hat y(\theta)-y\|^2.$0.

The same section identifies a sure-saddle region

$L(\theta)=\tfrac12 \|\hat y(\theta)-y\|^2.$1

which has positive measure whenever $L(\theta)=\tfrac12 \|\hat y(\theta)-y\|^2.$2, and every $L(\theta)=\tfrac12 \|\hat y(\theta)-y\|^2.$3 in it yields a saddle on the plateau. Under the weaker assumption that the parent neuron is “locally effective,” there exists at least one $L(\theta)=\tfrac12 \|\hat y(\theta)-y\|^2.$4 producing a saddle; if the neuron is “locally constant,” splitting preserves local minimality everywhere on the plateau. By contrast, splitting a saddle point always produces a plateau of saddle points (Ding et al., 3 Jun 2026).

These results make the term “plateau neuron” misleading in this context: the primary object is an affine stationary manifold. A plausible implication is that width expansion should be analyzed neuronwise through $L(\theta)=\tfrac12 \|\hat y(\theta)-y\|^2.$5 and through the coefficient region in which $L(\theta)=\tfrac12 \|\hat y(\theta)-y\|^2.$6 is chosen, rather than by width alone.

3. Plateau neurons as rare-token neurons in LLMs

In transformer interpretability, plateau neurons are identified in the last MLP layer by targeted mean-activation ablation. For neuron $L(\theta)=\tfrac12 \|\hat y(\theta)-y\|^2.$7 with pre-activation component $L(\theta)=\tfrac12 \|\hat y(\theta)-y\|^2.$8 and output weight vector $L(\theta)=\tfrac12 \|\hat y(\theta)-y\|^2.$9, the ablated residual-stream input is

$r$0

where $r$1 is neuron $r$2’s mean activation over a reference corpus. The neuron-level influence on rare-token cross-entropy is then

$r$3

Large values of $r$4 indicate neurons whose ablation most degrades rare-token prediction (Liu et al., 25 Sep 2025).

When neurons are sorted in descending order of $r$5 and $r$6 is plotted against $r$7, three regimes emerge for rare tokens:

• Plateau regime: a small plateau of size $r$8 of all neurons with nearly constant, exceptionally high $r$9.
• Power-law regime: an intermediate region in which

$\underline{\theta}$0

with typical fitted exponents $\underline{\theta}$1–$\underline{\theta}$2 across GPT-2/Pythia.

• Rapid-decay tail: $\underline{\theta}$3 falls off much faster than the power-law prediction.

Across model families, the final MLP layer consistently shows roughly $\underline{\theta}$4–$\underline{\theta}$5 of neurons in the plateau, $\underline{\theta}$6–$\underline{\theta}$7 in the power-law regime, and the remainder in the rapid tail. For common tokens, only a smooth power-law is observed and no plateau appears. Plateau neurons are therefore formally defined as those whose $\underline{\theta}$8 exceeds the upper continuity threshold of the power-law regime, that is, the leftmost flat region in the log-log plot (Liu et al., 25 Sep 2025).

The same work tests whether these neurons form a discrete module or a coordinated subspace. With activation matrix $\underline{\theta}$9 over rare-token contexts, covariance $(\underline{w}_r,\underline{v}_r)$0, and PCA-based effective dimensionality

$(\underline{w}_r,\underline{v}_r)$1

using $(\underline{w}_r,\underline{v}_r)$2, plateau neurons have lower $(\underline{w}_r,\underline{v}_r)$3 than random controls across all reported models (Liu et al., 25 Sep 2025).

Model	Plateau $(\underline{w}_r,\underline{v}_r)$4	Random $(\underline{w}_r,\underline{v}_r)$5
Pythia 1B	0.79	0.88
Pythia 1.4B	0.74	0.89
Pythia 2.8B	0.72	0.89
GPT-2 774M	0.77	0.82
GPT-2 1.5B	0.81	0.90

This lower-dimensional organization is accompanied by heavy-tailed weight correlation spectra. For the plateau-neuron submatrix $(\underline{w}_r,\underline{v}_r)$6 of the last-layer MLP weight matrix, the empirical correlation matrix is

$(\underline{w}_r,\underline{v}_r)$7

and tail-heaviness is quantified by the Hill estimator

$(\underline{w}_r,\underline{v}_r)$8

Plateau neurons exhibit consistently lower $(\underline{w}_r,\underline{v}_r)$9 (approximately $m-r+1$0–$m-r+1$1) than random controls (approximately $m-r+1$2–$m-r+1$3), and during training $m-r+1$4 decreases monotonically from about $m-r+1$5 in early iterations to about $m-r+1$6 at convergence, while controls remain near $m-r+1$7 (Liu et al., 25 Sep 2025).

The same paper rejects a hard modular interpretation. Louvain community detection on a mutual-information graph shows no consistent elevation in modularity, average community sizes are similar to random sets, and plateau neurons are spatially interspersed rather than forming discrete clusters. Attention routing is likewise distributed: the Gini coefficient comparison gives mean $m-r+1$8 versus $m-r+1$9 with $\lambda=(\lambda_0,\ldots,\lambda_{m-r})$0, single-head ablations change plateau activations by only approximately $\lambda=(\lambda_0,\ldots,\lambda_{m-r})$1–$\lambda=(\lambda_0,\ldots,\lambda_{m-r})$2, and ablating all heads in a layer causes an approximately $\lambda=(\lambda_0,\ldots,\lambda_{m-r})$3–$\lambda=(\lambda_0,\ldots,\lambda_{m-r})$4 drop, with both $\lambda=(\lambda_0,\ldots,\lambda_{m-r})$5 (Liu et al., 25 Sep 2025).

4. Layer-5 pyramidal plateau neurons and BAC firing

In cortical physiology, plateau neurons are specifically layer-5 pyramidal neurons that exhibit backpropagation-activated calcium firing. When a distal dendritic excitatory drive coincides with a back-propagating action potential, a regenerative dendritic Ca²⁺ plateau is evoked in the apical tuft. The core signatures are a dendritic membrane potential jump above approximately $\lambda=(\lambda_0,\ldots,\lambda_{m-r})$6 mV lasting tens of milliseconds, conversion of single somatic spikes into high-frequency bursts of $\lambda=(\lambda_0,\ldots,\lambda_{m-r})$7–$\lambda=(\lambda_0,\ldots,\lambda_{m-r})$8 spikes at approximately $\lambda=(\lambda_0,\ldots,\lambda_{m-r})$9–$\sum_{j=0}^{m-r}\lambda_j=1$0 Hz, and a supra-linear increase in the somatic input-output curve once plateaus become probable (Rodriguez-Garcia et al., 3 Jul 2025).

The mechanism is formulated in a two-compartment Izhikevich model. The soma is a quadratic integrate-and-fire unit with adaptation,

$\sum_{j=0}^{m-r}\lambda_j=1$1

with reset at $\sum_{j=0}^{m-r}\lambda_j=1$2. The apical dendritic compartment obeys

$\sum_{j=0}^{m-r}\lambda_j=1$3

$\sum_{j=0}^{m-r}\lambda_j=1$4

where

$\sum_{j=0}^{m-r}\lambda_j=1$5

A somatic spike injects a back-propagating action-potential current into the apical compartment with probability equal to the coupling strength $\sum_{j=0}^{m-r}\lambda_j=1$6, and if $\sum_{j=0}^{m-r}\lambda_j=1$7 crosses $\sum_{j=0}^{m-r}\lambda_j=1$8 mV, the somatic reset switches from regular-spiking values to burst-mode values for the next spike (Rodriguez-Garcia et al., 3 Jul 2025).

The regular-spiking somatic parameter set reported in Table 1 is $\sum_{j=0}^{m-r}\lambda_j=1$9 mV, $-30$00 mV, $-30$01 mV, $-30$02 mV, $-30$03, $-30$04 pF, $-30$05, $-30$06, and $-30$07. For burst mode, the reset shifts to $-30$08 mV and $-30$09. Typical dendritic parameters are $-30$10 mV, $-30$11 ms, $-30$12 pF, $-30$13 pS, $-30$14 pS, $-30$15 ms, and $-30$16; the external apical drive $-30$17 is an Ornstein-Uhlenbeck process with mean $-30$18 pA (Rodriguez-Garcia et al., 3 Jul 2025).

Neuromodulation and inhibition shape the probability of entering the plateau regime. Acetylcholine and noradrenaline act by increasing $-30$19 and increasing coupling $-30$20. SOM cells target the apical compartment and selectively reduce plateau probability, flattening the gain curve, whereas PV cells target the soma and raise the firing threshold, shifting the input-output curve laterally. Once a plateau arises, the soma bursts instead of firing isolated spikes, and because bursting elevates post- and presynaptic spike rates, the absolute rate of STDP updates $-30$21 increases nearly linearly with both $-30$22 and $-30$23 (Rodriguez-Garcia et al., 3 Jul 2025).

The voltage-trace descriptions make the operational definition concrete: somatic $-30$24 shows isolated spikes when $-30$25 mV, but $-30$26–$-30$27-spike bursts when a dendritic plateau is evoked; the apical trace $-30$28 jumps from approximately $-30$29 mV to approximately $-30$30 mV for approximately $-30$31 ms during a plateau. The authors interpret brief gain pulses that raise $-30$32 and $-30$33 as an adaptive two-timescale plasticity mechanism, with a transient high-gain, burst-dominated regime accelerating STDP updates and a baseline regime maintaining slower synaptic change (Rodriguez-Garcia et al., 3 Jul 2025).

5. Plateau potentials in respiratory neuron models

A related but distinct physiological literature analyzes plateau potentials in respiratory neurons in the pre-Bötzinger Complex. In the model of Abdulla et al., adding extracellular potassium dynamics to an existing bursting model, together with parameter updates, is sufficient to induce pre-inspiratory ramping, in which relatively slow tonic spiking gradually progresses to faster spiking and a full blown burst, with a corresponding gradual development of an underlying plateau potential (Abdulla et al., 2021).

The full system has seven ODEs: membrane voltage, five gating variables, and extracellular potassium concentration. The fast subsystem consists of $-30$34 on a timescale of at most about $-30$35 ms, while the slow variables are $-30$36 with $-30$37 s and $-30$38 with $-30$39 s plus glial uptake. The model uses default maximal conductances $-30$40 nS, $-30$41 nS, $-30$42 nS, $-30$43 nS, and $-30$44 nS, with $-30$45 ms, $-30$46 mM, $-30$47 mM/s, $-30$48 mM, $-30$49, $-30$50, and $-30$51 (Abdulla et al., 2021).

Its fast subsystem exhibits an S-shaped critical manifold of equilibria with a lower stable branch, a middle unstable branch, and an upper stable branch, together with a saddle-node at the lower knee and a supercritical Andronov-Hopf bifurcation. For small $-30$52, stable periodic orbits terminate in a SNIC and the system shows tonic spiking. As $-30$53 rises, the periodic orbit instead terminates in a homoclinic, yielding square-wave bursting. The approximate transition values reported are a SNIC-to-homoclinic switch near $-30$54 mV, corresponding to $-30$55 mM, and a burst-to-spike transition near $-30$56 mV, corresponding to $-30$57 mM (Abdulla et al., 2021).

In the phase-plane description, the $-30$58-nullcline is cubic-shaped and the $-30$59-nullcline is monotonic. As $-30$60 slowly increases, the lower branch of the $-30$61-nullcline shifts upward, the intersection moves toward the left knee, and the system first slips into small-amplitude slow spiking on a depolarized plateau. The plateau corresponds to the trajectory evolving near the SNIC bifurcation curve of the fast subsystem, and when $-30$62 falls to $-30$63, a homoclinic jump ends the plateau and returns the trajectory to quiescence (Abdulla et al., 2021).

This work does not designate a neuron class as “plateau neurons,” but it is central for understanding plateau potentials as slow-manifold states created by ionic positive feedback. The reported dynamic range includes burst frequencies from approximately $-30$64 Hz to approximately $-30$65 Hz, burst durations from approximately $-30$66 s to over $-30$67 s, and duty cycles from approximately $-30$68 up to approximately $-30$69 (Abdulla et al., 2021).

6. Comparative interpretation and recurring misconceptions

A recurring misconception is that “plateau neurons” always implies a localized module. The rare-token study explicitly finds the opposite: plateau neurons are functionally coordinated but spatially distributed, with no dedicated routing circuit and no discrete clustering. A second misconception is that widening a network by splitting a hidden neuron merely preserves the local nature of a stationary point. The geometric analysis shows instead that splitting a local minimum can yield a mixture of local minima and saddles or an all-saddle plateau, while splitting a saddle always yields saddles. A third misconception is that a physiological plateau is merely prolonged depolarization. In the layer-5 pyramidal model, the plateau is a coincidence-dependent dendritic Ca²⁺ event that switches firing mode from isolated spikes to bursting; in the respiratory model, the plateau is a slow-manifold segment generated by extracellular potassium feedback and terminated by a homoclinic bifurcation (Liu et al., 25 Sep 2025, Ding et al., 3 Jun 2026, Rodriguez-Garcia et al., 3 Jul 2025, Abdulla et al., 2021).

Taken together, the cited literature suggests a family resemblance rather than a single ontology. In every case, “plateau” identifies a regime with constrained local variation: a flat influence hierarchy in the transformer case, an affine stationary family in the loss-landscape case, a regenerative depolarized state in the cortical case, and a stable slow-manifold segment in the respiratory case. A plausible implication is that the term is best understood operationally, with the underlying mechanism specified explicitly: ablation-defined influence, neuron-splitting geometry, BAC-triggered dendritic bistability, or slow ionic feedback.