---
title: 'Plateau Neurons: Mechanisms in AI and Physiology'
url: https://www.emergentmind.com/topics/plateau-neurons
type: topic
---

# Plateau Neurons: Mechanisms in AI and Physiology

“Plateau neurons” denotes different but technically precise objects across contemporary literature. In transformer analysis, the term refers to a small “plateau” of final-layer MLP neurons whose mean-activation ablation has nearly constant, exceptionally high influence on rare-token loss. In cortical neurophysiology, it refers to layer-5 pyramidal neurons whose backpropagation-activated calcium firing evokes a regenerative dendritic Ca²⁺ plateau in the apical tuft. Closely related plateau phenomena also arise in two-layer neural-network loss landscapes, where neuron splitting creates an affine family of stationary points, and in respiratory neuron models, where slow ionic feedback generates a plateau potential during ramping bursts [2509.21163; 2507.03222; 2606.04327; 2105.03548].

## 1. Terminological scope

The literature uses “plateau” in several non-equivalent ways. In the transformer setting, the defining observation is a leftmost flat region in a log-log influence plot: a small set of neurons has nearly constant, exceptionally high $\Delta Loss$, and these are formally defined as plateau neurons. In cortical physiology, the defining event is a dendritic membrane potential jump above approximately $-30$ mV that lasts tens of milliseconds and converts single somatic spikes into high-frequency bursts. In two-layer network geometry, the plateau is not a neuron type but an affine family of stationary points created by duplicating a hidden neuron and redistributing its output weights. In respiratory dynamics, the plateau is a stable slow-manifold segment reached after a SNIC-to-homoclinic bifurcation switch, producing ramping spiking on a depolarized plateau [2509.21163; 2507.03222; 2606.04327; 2105.03548].

| Context | “Plateau” denotes | Principal object |
|---|---|---|
| Transformer MLP analysis | Leftmost flat region in neuron influence hierarchy | Rare-token neurons |
| Layer-5 cortical physiology | Regenerative dendritic Ca²⁺ plateau | BAC-firing pyramidal neurons |
| Two-layer loss landscape | Affine family of stationary points | Split hidden neuron manifold |
| Respiratory neuron dynamics | Depolarized plateau during ramping burst | Slow ionic-feedback state |

A common source of confusion is to treat these usages as interchangeable. The cited works instead attach the term to distinct geometric, functional, and biophysical structures.

## 2. Stationary plateaus from neuron splitting

In the loss-landscape setting, the relevant model is a two-layer network
$$
f(x;\theta)=v^\top \sigma(Wx), \qquad \theta=(v,W),
$$
trained by the squared loss
$$
L(\theta)=\tfrac12 \|\hat y(\theta)-y\|^2.
$$
Starting from a narrow network of width $r$ at a stationary $\underline{\theta}$, neuron splitting selects a parent neuron $(\underline{w}_r,\underline{v}_r)$ and replaces it by $m-r+1$ daughter neurons with the same input weight and redistributed output weights. If $\lambda=(\lambda_0,\ldots,\lambda_{m-r})$ with $\sum_{j=0}^{m-r}\lambda_j=1$, then $\theta=\mathrm{split}(\underline{\theta},\lambda)$ satisfies $f(x;\theta)=f(x;\underline{\theta})$ and $L(\theta)=L(\underline{\theta})$. The resulting affine family
$$
P_{\underline{\theta}}=\{\mathrm{split}(\underline{\theta},\lambda):\lambda\in\Lambda\}, \qquad
\Lambda=\{\lambda\in\mathbb{R}^{m-r+1}:\sum \lambda_j=1\},
$$
is the stationary plateau [2606.04327].

The local geometry of this plateau is controlled by the per-neuron “inner Hessian” matrix. For the $i$-th hidden neuron,
$$
(H_i)_{j,j'} = v_i \langle \hat y-y,\; z''_i \odot x_j \odot x_{j'} \rangle,
$$
with matrix form
$$
H_i = v_i \cdot [\langle \hat y-y,\; z''_i\odot x_j\odot x_{j'}\rangle]_{j,j'}.
$$
Its relation to the full Hessian with respect to $w_i$ is
$$
\frac{\partial^2 L}{\partial w_i^2}=J^\top J + H_i,
$$
so $H_i$ captures the “second-derivative” term coming from the network’s nonlinearity, and its eigenvalues govern local curvature induced by asymmetric perturbations of the split daughters [2606.04327].

For a nondegenerate local minimum of the narrow net with $L(\underline{\theta})\neq 0$, the paper classifies all points on the plateau. Writing
$$
\Lambda_{PD}=\{\lambda:\lambda_j>0\ \forall j\}, \qquad
\Lambda_{ND}=\{\lambda:\text{exactly one }\lambda_j>0\text{ and the rest }<0\},
$$
the conclusions are:

- **If $H_r \succ 0$**: $\theta=\mathrm{split}(\underline{\theta},\lambda)$ is a local minimum iff $\lambda\in\Lambda_{PD}$, and a saddle otherwise.
- **If $H_r \prec 0$**: $\theta$ is a local minimum iff $\lambda\in\Lambda_{ND}$, and a saddle otherwise.
- **If $H_r$ is indefinite**: $\theta$ is a saddle for all $\lambda\in\Lambda$.

The same section identifies a sure-saddle region
$$
\Lambda \setminus (\Lambda_{PD}\cup \Lambda_{ND}),
$$
which has positive measure whenever $H_r\neq 0$, and every $\lambda$ in it yields a saddle on the plateau. Under the weaker assumption that the parent neuron is “locally effective,” there exists at least one $\lambda$ producing a saddle; if the neuron is “locally constant,” splitting preserves local minimality everywhere on the plateau. By contrast, splitting a saddle point always produces a plateau of saddle points [2606.04327].

These results make the term “plateau neuron” misleading in this context: the primary object is an affine stationary manifold. A plausible implication is that width expansion should be analyzed neuronwise through $H_r$ and through the coefficient region in which $\lambda$ is chosen, rather than by width alone.

## 3. Plateau neurons as rare-token neurons in large language models

In transformer interpretability, plateau neurons are identified in the last MLP layer by targeted mean-activation ablation. For neuron $i$ with pre-activation component $n_i$ and output weight vector $w_{\mathrm{out}}^{(i)}$, the ablated residual-stream input is
$$
\tilde x^{(i)} = x + (\bar n_i - n_i) w_{\mathrm{out}}^{(i)},
$$
where $\bar n_i$ is neuron $i$’s mean activation over a reference corpus. The neuron-level influence on rare-token cross-entropy is then
$$
\Delta Loss(i)=\mathbb{E}_{x\sim D_{\mathrm{rare}}}
\left|
\mathcal{L}(LM(x),x)-\mathcal{L}(LM(\tilde x^{(i)}),x)
\right|.
$$
Large values of $\Delta Loss(i)$ indicate neurons whose ablation most degrades rare-token prediction [2509.21163].

When neurons are sorted in descending order of $\Delta Loss(i)$ and $\log \Delta Loss$ is plotted against $\log(\mathrm{rank})$, three regimes emerge for rare tokens:

- **Plateau regime**: a small plateau of size $p\approx 1\%$ of all neurons with nearly constant, exceptionally high $\Delta Loss$.
- **Power-law regime**: an intermediate region in which
  $$
  \log \Delta Loss(r)\approx -\kappa \log r + \beta
  $$
  with typical fitted exponents $\kappa\approx 0.8$–$1.1$ across GPT-2/Pythia.
- **Rapid-decay tail**: $\Delta Loss$ falls off much faster than the power-law prediction.

Across model families, the final MLP layer consistently shows roughly $0.8$–$1.2\%$ of neurons in the plateau, $8$–$15\%$ in the power-law regime, and the remainder in the rapid tail. For common tokens, only a smooth power-law is observed and no plateau appears. Plateau neurons are therefore formally defined as those whose $\Delta Loss$ exceeds the upper continuity threshold of the power-law regime, that is, the leftmost flat region in the log-log plot [2509.21163].

The same work tests whether these neurons form a discrete module or a coordinated subspace. With activation matrix $A$ over rare-token contexts, covariance $\Sigma=(1/M)A^\top A$, and PCA-based effective dimensionality
$$
d_{\mathrm{eff}}=\min\left\{d:\sum_{j=1}^d \lambda_j \big/ \sum_{j=1}^p \lambda_j \ge \tau\right\},
$$
using $\tau=0.95$, plateau neurons have lower $d_{\mathrm{eff}}/p$ than random controls across all reported models [2509.21163].

| Model | Plateau $d_{\mathrm{eff}}/p$ | Random $d_{\mathrm{eff}}/p$ |
|---|---:|---:|
| Pythia 1B | 0.79 | 0.88 |
| Pythia 1.4B | 0.74 | 0.89 |
| Pythia 2.8B | 0.72 | 0.89 |
| GPT-2 774M | 0.77 | 0.82 |
| GPT-2 1.5B | 0.81 | 0.90 |

This lower-dimensional organization is accompanied by heavy-tailed weight correlation spectra. For the plateau-neuron submatrix $W_G$ of the last-layer MLP weight matrix, the empirical correlation matrix is
$$
\Xi_G=\frac1d W_G W_G^\top,
$$
and tail-heaviness is quantified by the Hill estimator
$$
\alpha_{\mathrm{Hill}}
=
\left[
\frac1k \sum_{i=1}^k \ln(\lambda_i/\lambda_k)
\right]^{-1}.
$$
Plateau neurons exhibit consistently lower $\alpha_{\mathrm{Hill}}$ (approximately $1.2$–$1.8$) than random controls (approximately $2.5$–$3.2$), and during training $\alpha$ decreases monotonically from about $2.5$ in early iterations to about $1.3$ at convergence, while controls remain near $3$ [2509.21163].

The same paper rejects a hard modular interpretation. Louvain community detection on a mutual-information graph shows no consistent elevation in modularity, average community sizes are similar to random sets, and plateau neurons are spatially interspersed rather than forming discrete clusters. Attention routing is likewise distributed: the Gini coefficient comparison gives mean $0.34$ versus $0.32$ with $p=0.43$, single-head ablations change plateau activations by only approximately $0.28$–$0.34$, and ablating all heads in a layer causes an approximately $45$–$50\%$ drop, with both $p<10^{-3}$ [2509.21163].

## 4. Layer-5 pyramidal plateau neurons and BAC firing

In cortical physiology, plateau neurons are specifically layer-5 pyramidal neurons that exhibit backpropagation-activated calcium firing. When a distal dendritic excitatory drive coincides with a back-propagating action potential, a regenerative dendritic Ca²⁺ plateau is evoked in the apical tuft. The core signatures are a dendritic membrane potential jump above approximately $-30$ mV lasting tens of milliseconds, conversion of single somatic spikes into high-frequency bursts of $2$–$5$ spikes at approximately $200$–$300$ Hz, and a supra-linear increase in the somatic input-output curve once plateaus become probable [2507.03222].

The mechanism is formulated in a two-compartment Izhikevich model. The soma is a quadratic integrate-and-fire unit with adaptation,
$$
\frac{dv_s}{dt}
=
\frac{1}{C_s}
\left[
k_s(v_s-v_r)(v_s-v_t)-u_s+I_s
\right],
\qquad
\frac{du_s}{dt}
=
a_s[b_s(v_s-v_r)-u_s],
$$
with reset at $v_s\ge v_p$. The apical dendritic compartment obeys
$$
\frac{dv_d}{dt}
=
\frac{E_{L_d}-v_d}{\tau_d}
+
\frac{1}{C_d}
\left[
g_d F_d(v_d)+w_d+m_d P_{\rm BAP} H(t-t^{(s)})+I_d
\right],
$$
$$
\frac{dw_d}{dt}
=
\frac{-w_d+a_d(v_d-E_{L_d})}{\tau_{w_d}},
$$
where
$$
F_d(v_d)=\frac{1}{1+e^{-(v_d+38)/6}}.
$$
A somatic spike injects a back-propagating action-potential current into the apical compartment with probability equal to the coupling strength $\delta$, and if $v_d$ crosses $-30$ mV, the somatic reset switches from regular-spiking values to burst-mode values for the next spike [2507.03222].

The regular-spiking somatic parameter set reported in Table 1 is $v_r=-75$ mV, $v_t=-45$ mV, $v_p=50$ mV, $c_s=-65$ mV, $d_s=250$, $C_s=150$ pF, $k_s=2.5$, $a_s=0.01$, and $b_s=5$. For burst mode, the reset shifts to $c_s=-55$ mV and $d_s=150$. Typical dendritic parameters are $E_{L_d}=-75$ mV, $\tau_d=30$ ms, $C_d=150$ pF, $g_d=20$ pS, $m_d=5$ pS, $\tau_{w_d}=200$ ms, and $a_d=0.01$; the external apical drive $I_d$ is an Ornstein-Uhlenbeck process with mean $\mu_d\in[200,800]$ pA [2507.03222].

Neuromodulation and inhibition shape the probability of entering the plateau regime. Acetylcholine and noradrenaline act by increasing $\mu_d$ and increasing coupling $\delta$. SOM cells target the apical compartment and selectively reduce plateau probability, flattening the gain curve, whereas PV cells target the soma and raise the firing threshold, shifting the input-output curve laterally. Once a plateau arises, the soma bursts instead of firing isolated spikes, and because bursting elevates post- and presynaptic spike rates, the absolute rate of STDP updates $\lvert dw/dt\rvert$ increases nearly linearly with both $\delta$ and $\mu_d$ [2507.03222].

The voltage-trace descriptions make the operational definition concrete: somatic $v_s(t)$ shows isolated spikes when $v_d<-30$ mV, but $3$–$4$-spike bursts when a dendritic plateau is evoked; the apical trace $v_d(t)$ jumps from approximately $-75$ mV to approximately $0$ mV for approximately $50$ ms during a plateau. The authors interpret brief gain pulses that raise $\delta$ and $\mu_d$ as an adaptive two-timescale plasticity mechanism, with a transient high-gain, burst-dominated regime accelerating STDP updates and a baseline regime maintaining slower synaptic change [2507.03222].

## 5. Plateau potentials in respiratory neuron models

A related but distinct physiological literature analyzes plateau potentials in respiratory neurons in the pre-Bötzinger Complex. In the model of Abdulla et al., adding extracellular potassium dynamics to an existing bursting model, together with parameter updates, is sufficient to induce pre-inspiratory ramping, in which relatively slow tonic spiking gradually progresses to faster spiking and a full blown burst, with a corresponding gradual development of an underlying plateau potential [2105.03548].

The full system has seven ODEs: membrane voltage, five gating variables, and extracellular potassium concentration. The fast subsystem consists of $\{V,m_{Na},h_{Na},m_{NaP},n\}$ on a timescale of at most about $10$ ms, while the slow variables are $h_{NaP}$ with $\tau\sim 5$ s and $[K^+]_{out}$ with $\tau_{diff}\sim 0.75$ s plus glial uptake. The model uses default maximal conductances $\bar g_{Na}=150$ nS, $\bar g_K=160$ nS, $\bar g_{NaP}=5.0$ nS, $\bar g_L=2.5$ nS, and $\bar g_{Syn}=0.365$ nS, with $\tau_{diff}=750$ ms, $K_{bath}=4$ mM, $\bar G=10$ mM/s, $\bar K=5$ mM, $z_k=6\ \mathrm{(mM)^{-1}}$, $\gamma=7.214\times 10^{-3}$, and $\beta=14.555$ [2105.03548].

Its fast subsystem exhibits an S-shaped critical manifold of equilibria with a lower stable branch, a middle unstable branch, and an upper stable branch, together with a saddle-node at the lower knee and a supercritical Andronov-Hopf bifurcation. For small $E_K$, stable periodic orbits terminate in a SNIC and the system shows tonic spiking. As $E_K$ rises, the periodic orbit instead terminates in a homoclinic, yielding square-wave bursting. The approximate transition values reported are a SNIC-to-homoclinic switch near $E_K\approx -90$ mV, corresponding to $[K^+]_{out}\approx 5$ mM, and a burst-to-spike transition near $E_K\approx -83$ mV, corresponding to $[K^+]_{out}\approx 6.7$ mM [2105.03548].

In the phase-plane description, the $V$-nullcline is cubic-shaped and the $h_{NaP}$-nullcline is monotonic. As $[K^+]_{out}$ slowly increases, the lower branch of the $V$-nullcline shifts upward, the intersection moves toward the left knee, and the system first slips into small-amplitude slow spiking on a depolarized plateau. The plateau corresponds to the trajectory evolving near the SNIC bifurcation curve of the fast subsystem, and when $h_{NaP}$ falls to $h_{NaP}^{HC}$, a homoclinic jump ends the plateau and returns the trajectory to quiescence [2105.03548].

This work does not designate a neuron class as “plateau neurons,” but it is central for understanding plateau potentials as slow-manifold states created by ionic positive feedback. The reported dynamic range includes burst frequencies from approximately $0.2$ Hz to approximately $0.6$ Hz, burst durations from approximately $0.5$ s to over $4$ s, and duty cycles from approximately $0.1$ up to approximately $0.9$ [2105.03548].

## 6. Comparative interpretation and recurring misconceptions

A recurring misconception is that “plateau neurons” always implies a localized module. The rare-token study explicitly finds the opposite: plateau neurons are functionally coordinated but spatially distributed, with no dedicated routing circuit and no discrete clustering. A second misconception is that widening a network by splitting a hidden neuron merely preserves the local nature of a stationary point. The geometric analysis shows instead that splitting a local minimum can yield a mixture of local minima and saddles or an all-saddle plateau, while splitting a saddle always yields saddles. A third misconception is that a physiological plateau is merely prolonged depolarization. In the layer-5 pyramidal model, the plateau is a coincidence-dependent dendritic Ca²⁺ event that switches firing mode from isolated spikes to bursting; in the respiratory model, the plateau is a slow-manifold segment generated by extracellular potassium feedback and terminated by a homoclinic bifurcation [2509.21163; 2606.04327; 2507.03222; 2105.03548].

Taken together, the cited literature suggests a family resemblance rather than a single ontology. In every case, “plateau” identifies a regime with constrained local variation: a flat influence hierarchy in the transformer case, an affine stationary family in the loss-landscape case, a regenerative depolarized state in the cortical case, and a stable slow-manifold segment in the respiratory case. A plausible implication is that the term is best understood operationally, with the underlying mechanism specified explicitly: ablation-defined influence, neuron-splitting geometry, BAC-triggered dendritic bistability, or slow ionic feedback.

Source: https://www.emergentmind.com/topics/plateau-neurons