---
title: Neural Uncertainty Principle
url: https://www.emergentmind.com/topics/neural-uncertainty-principle-nup
type: topic
---

# Neural Uncertainty Principle

Searching arXiv for current papers on “Neural Uncertainty Principle” and closely related formulations.
Attempting arXiv lookup for exact phrase matches and cited IDs.
The Neural Uncertainty Principle (NUP) is a family of research proposals that use the language of “uncertainty” to describe structural tradeoffs in neural systems and neural-network models. Across the current arXiv literature, the term does not denote a single canonical theorem. Instead, it names several non-equivalent complementarity claims: expressivity versus optimization degeneracy in Heaviside/sigmoid networks, accuracy versus adversarial robustness in classifiers, temporal localization versus finite recurrent order in linear RNNs, dual-channel allocation of information across firing and co-firing spike-train codes, and threshold-versus-latency variability in retinal transduction [2603.06634], [2205.01493], [2502.09287], [1912.11126], [2603.19562], [2507.22785].

## 1. Terminology and scope

In present usage, NUP is best understood as an umbrella label for complementarity relations in neural representation, training geometry, or neural coding. The common pattern is that increased sharpness, localization, or discriminability in one domain is accompanied by redundancy, dispersion, or degraded conditioning in another domain.

| Formulation | Core tradeoff | Representative paper |
|---|---|---|
| Heaviside/sigmoid learning | “the sharper the minimum, the smoother the canyons” | [2603.06634] |
| Adversarial classification | \(\sigma_{p_i}\sigma_{x_i}\ge \frac{1}{2}\) | [2205.01493] |
| Linear recurrent copy task | temporal width proportional to \(K/S\) | [2502.09287] |
| Spike-train coding | information in \(R\) comes at the expense of information in \(R'\) | [1912.11126] |
| Vision–LLM reliability | \(V_c(u)\ge \kappa\) under input–gradient coupling | [2603.19562] |
| Retinal probabilistic transduction | \(\Delta \alpha \cdot \Delta t \geq \eta\) | [2507.22785] |

Two adjacent literatures use the word “uncertainty” differently and are therefore relevant mainly as contrast classes. The neuroscience review "Studying the neural representations of uncertainty" distinguishes code-driven and correlational approaches to uncertainty as a property of an observer’s belief, organized by sensitivity, specificity, invariance, and functionality, but does not propose an NUP [2202.04324]. The Bayesian-neural-network study of quantum contextuality treats uncertainty as prediction confidence derived from a posterior over weights rather than as a structural complementarity principle [2212.13564].

## 2. Heaviside–sigmoid expressivity and canyon geometry

"A new Uncertainty Principle in Machine Learning" formulates NUP as a statement about representability versus trainability in two-layer Heaviside and sigmoid networks [2603.06634]. Its central architectural claim is that arbitrary polynomials of arbitrary degree and number of variables can be represented by one and the same two-layer Heaviside formula,
\[
{\rm Pol}(\vec x) =y(\vec x):= \int_I w_2^I\cdot\theta\left(\int_J w_1^{IJ}\cdot \theta(\vec w_0^J \vec x+b_0^J) + b_1^I\right),
\]
with sigmoids treated as nearly singular approximants to Heavisides. The paper states the principle in several equivalent verbal forms: “The more pronounced is the minimum of the functional, the more uncertain is its Heaviside approximation,” “the sharper the minimum, the smoother the canyons,” and “if we want a broadly applicable ansatz/Heavisidization, we get the canyon problem” [2603.06634].

The mechanism is explicit. Enlarging the Heaviside or sigmoid ansatz introduces many non-identifiable parameters. Exact Heaviside representations then produce valleys of equivalent solutions, while smoothing, discretization, or bias terms lift those valleys into canyons: steep descent across the canyon and slow drift along its bottom. The identity-function example makes the point analytically. For
\[
x=\int_0^\infty W\theta(wx-z)\,dz,
\]
all parameter pairs satisfying
\[
Ww=1
\]
represent the same function, and the loss
\[
{\cal L}=\frac{X^3(Ww-1)^2}{3}
\]
vanishes on the entire hyperbola \(Ww=1\). Reintroducing a bias-type parameter lifts the exact degeneracy only weakly, producing a deep canyon whose bottom height scales much more weakly than the typical loss scale [2603.06634].

The same paper gives a second, spectrum-oriented formulation for one-layer sigmoids: “The more ‘oscillating/wavy/irregular’ is the target function \(y\) on a given interval, the wider is the spectrum (the set of relatively large coefficients of \(w_1\)) of its \(\sigma\)-transform.” The analogy to Fourier uncertainty is deliberate but structural rather than literal. The intended correspondence is between concentration in function space versus spread in spectral space for Fourier analysis, and concentration in representational accuracy versus spread or degeneracy in parameter space for Heaviside/sigmoid networks [2603.06634].

This formulation is not presented as a universal lower-bound theorem analogous to \(\Delta x\,\Delta p \ge \hbar/2\). It is presented as a mathematically motivated combination of exact representational formulas, analytic toy derivations, semi-rigorous asymptotics, and numerical illustrations. Its importance lies in making optimization degeneracy, not lack of expressivity, the central obstruction.

## 3. Input–gradient conjugacy in robustness and hallucination

A distinct line of work uses NUP to interpret adversarial fragility through an operator calculus on the loss. "On the uncertainty principle of neural networks" normalizes the loss into a “neural packet”
\[
\psi_Y(X)=\frac{l(f(X,\theta),Y)}{\beta^{1/2}}, \qquad \int \psi_Y(X)^2\, dX = 1,
\]
defines the pixel operator \(\hat x_i\psi_Y(X)=x_i\psi_Y(X)\) and the attack operator \(\hat p_i\psi_Y(X)=\frac{\partial}{\partial x_i}\psi_Y(X)\), and derives
\[
\sigma_{p_i}\sigma_{x_i}\ge \frac{1}{2}
\]
from the commutator \([\hat p_i,\hat x_i]\psi_Y=\psi_Y\) [2205.01493]. In that formalism, the inequality is rigorous at the operator level, but the mapping from \(\sigma_{x_i}\) and \(\sigma_{p_i}\) to empirical clean accuracy and adversarial robustness is interpretive rather than a standard learning-theoretic theorem. The paper explicitly frames the claimed accuracy–robustness tradeoff as a Heisenberg-style analogy built on the normalized loss representation [2205.01493].

"Neural Uncertainty Principle: A Unified View of Adversarial Fragility and LLM Hallucination" retains the input–gradient conjugacy motif but substantially formalizes it [2603.19562]. For a unit direction \(u\), it defines
\[
(\hat{x}_u g)(x):=(u^\top x)\,g(x), \qquad (\hat{p}_u g)(x):=-i\,\partial_u g(x),
\]
with canonical commutator \([\hat{x}_u,\hat{p}_u]=i\mathbb I\), and constructs a loss-induced state
\[
\psi_c(x) = A_c(x)\exp\!\big(i\alpha\,\mathcal L_c(x)\big), \qquad
A_c(x):=\frac{|\mathcal L_c(x)|}{\sqrt{\beta_c}}.
\]
The resulting neural uncertainty relation is
\[
(\Delta \hat{x}_c(u))^2(\Delta \hat{p}_c(u))^2 \ge \frac14\big|\langle[\hat{x}_u,\hat{p}_u]\rangle_c\big|^2 +\mathrm{Cov}_c(\hat{x}_u,\hat{p}_u)^2,
\]
or, equivalently,
\[
V_c(u):=\Delta \hat{x}_c(u)\,\Delta \hat{p}_c(u)\,\sqrt{1-\rho_c(u)^2}\ge \kappa.
\]
A key exact reduction theorem states
\[
\mathrm{Cov}_c(\hat x_u,\hat p_u) = \alpha\,\mathrm{Cov}^{\mathrm{sc}}_c(x_u,p_u),
\]
which ties the operator covariance directly to the loss-weighted scalar covariance between projected input and projected input gradient [2603.19562].

This later formulation uses the same geometry to interpret two different failure modes. In vision, strong input–gradient coupling reduces the effective feasible conjugate volume and forces larger sensitivity dispersion, yielding “boundary stress” and adversarial fragility. In language, unusually weak prompt–gradient coupling is interpreted as slack or under-conditioning, which is used to predict hallucination risk before any answer token is generated. The practical surrogate is the single-backward CC-Probe, implemented as a cosine between input and gradient in vision and a mean-centered embedding–gradient cosine in language. Guided by that theory, the paper proposes ConjMask, which masks high-contribution input components during training, and LogitReg, an output-side regularization term [2603.19562].

The two operator-based NUP papers therefore share a common intuition—input coordinates and input-loss gradients behave as conjugate objects—while differing sharply in formal maturity. The earlier paper uses a physics-inspired construction to motivate an accuracy–robustness tradeoff; the later paper develops an operator theorem, a covariance channel, and single-backward diagnostics for both adversarial robustness and LLM hallucination [2205.01493], [2603.19562].

## 4. Temporal resolution limits in linear recurrent networks

A different and more narrowly delimited NUP appears in "An Uncertainty Principle for Linear Recurrent Neural Networks" [2502.09287]. Here the setting is a linear recurrent model
\[
x_n = A_n x_{n-1} + B_n u_n,
\]
specialized to the time-invariant diagonal case
\[
x_n = \operatorname{diag}(a)\,x_{n-1} + u_n b,
\]
with output
\[
y_n = (c*u)_n = \sum_{k=0}^\infty c_k u_{n-k}, \qquad c_k = \sum_{s=1}^S a_s^k b_s.
\]
The task is to approximate the shift-\(K\) filter \(d=\delta_K\), namely the delayed copy \(u_{n-K}\), using only recurrent order \(S\) [2502.09287].

The central theorem is a lower bound on approximation quality. For white noise,
\[
\mathcal L_{\mathrm{time}}(c,d)\ge 1-\frac{S}{K+1}.
\]
For autocorrelated inputs with \(\gamma(k)=\rho^{|k|}\),
\[
\mathcal L_{\mathrm{time}}(c,d)\ge \Big(1-\frac{3S}{K}\frac{1}{1-\rho}\Big)_+.
\]
The constructive upper bound then shows that the optimal filter approximates the shift only over a frequency window of width \(\asymp S/K\). By Fourier duality, the corresponding time-domain kernel must spread over a width of order
\[
\frac{K}{S}.
\]
The paper summarizes this as: “The optimal filter has to average values around the \(K\)-th time step in the past with a range (width) that is proportional to \(K/S\)” [2502.09287].

This is one of the cleanest NUP instances in the literature because the tradeoff is exact within the model class. The principle is not a vague “limited memory” claim. It is a resolution statement: a low-order linear RNN can still refer to a long delay \(K\), but only in a temporally smeared form. Long-range recall remains possible, yet temporal localization degrades as \(K/S\). The result is rigorous but specific to stable linear recurrent or linear time-invariant filters of the form \(c_k=\sum_{s=1}^S a_s^k b_s\), quadratic loss, and the copy-task approximation problem [2502.09287].

## 5. Dual-channel spike coding and retinal threshold–time proposals

In systems neuroscience, NUP has been proposed in two rather different forms. "An uncertainty principle for neural coding: Conjugate representations of position and velocity are mapped onto firing rates and co-firing rates of neural spike trains" argues that spike trains contain two coupled representational channels: a firing-rate code \(R\), derived from within-cell spike intervals,
\[
r_n = s_n * u,
\]
and a co-firing-rate code \(R'\), derived from between-cell spike intervals through pairwise chi-rates,
\[
\chi_{i,j} = (s_i \circ r_j) * v.
\]
The paper then postulates uncertainty-like relations such as
\[
\operatorname{var} q_\Sigma \, \operatorname{var} q_{\Sigma\chi} \ge C_q,
\]
\[
\operatorname{var} q_\Sigma \, \operatorname{var} \dot q_\Sigma \ge C_\Sigma,
\qquad
\operatorname{var} q_{\Sigma\chi} \, \operatorname{var} \dot q_{\Sigma\chi} \ge C_{\Sigma\chi}.
\]
Its central claim is that information conveyed in \(R\) comes at the expense of information in \(R'\), and vice versa, except when the two channels encode a conjugate pair of world variables such as position \(q\) and velocity \(\dot q\). Sigma decoding recovers information from \(R\), whereas sigma-chi decoding recovers information from \(R'\). Simulations of head-direction cells, grid cells, and theta-modulated speed cells are used to show both regimes \(q \to R,\ \dot q \to R'\) and \(\dot q \to R,\ q \to R'\) [1912.11126]. The paper explicitly treats the principle as a formal hypothesis supported by simulations rather than a closed-form theorem.

"An Uncertainty Principle for Probabilistic Computation in the Retina" relocates the discussion to early visual transduction and proposes a threshold–latency tradeoff
\[
\Delta \alpha \cdot \Delta t \geq \eta,
\]
with \(\Delta \alpha\) the variability in activation thresholds, \(\Delta t\) the temporal variability in onset time or first-spike latency across trials, and \(\eta\) a lower bound “analogous to Planck’s constant in physics” and “determined empirically” [2507.22785]. The modeling pipeline begins with a probabilistic light intensity field
\[
I(x_i, t) = I_0 \cdot \exp\left( -\frac{(|x_i| - v t)^2}{2 \sigma^2} \right),
\]
passes through stochastic thresholding at photoreceptors,
\[
P_i(t) = \frac{1}{1 + \exp\left( -\frac{I(x_i, t) - \theta_i(t)}{\beta} \right)},
\qquad
\theta_i(t) \sim \mathcal{N}(\bar{\theta}, \Delta \alpha^2),
\]
and continues through horizontal, bipolar, amacrine, and ganglion-cell stages to a symbolic spike code [2507.22785].

The retinal formulation is explicitly speculative. The paper states that the retina should be viewed as a “probabilistic measurement device” and that the “moment of first perception is not a sharply defined spatiotemporal event, but a probabilistic cloud shaped by physical and biological limits.” At the same time, it does not derive the inequality from first principles, several displayed equations are malformed, and the evidence cited supports intrinsic variability rather than the specific lower-bound product relation itself [2507.22785]. A plausible implication is that this work advances NUP primarily as a phenomenological research program.

## 6. Relation to broader neural uncertainty research and conceptual status

A recurring source of confusion is the difference between NUP and ordinary predictive uncertainty. The review "Studying the neural representations of uncertainty" emphasizes that uncertainty in neuroscience is “a property of an observer’s belief about the world,” formalized through quantities such as \(p(s \mid I_{\text{observed}})\), and that claims about neural uncertainty representations must be evaluated using sensitivity, specificity, invariance, and functionality [2202.04324]. This is a methodological framework for uncertainty representations, not an uncertainty principle in the NUP sense. Its main relevance is to show that any putative NUP in neuroscience is model-relative and method-dependent.

Likewise, "Uncertainty of Feed Forward Neural Networks Recognizing Quantum Contextuality" uses uncertainty in the Bayesian predictive sense. Standard networks estimate uncertainty from softmax entropy,
\[
E(p)=-\sum_{c=1}^d p_c \log p_c,
\]
whereas Bayesian neural networks place a posterior over weights,
\[
p(w\mid D)=\frac{p(D\mid w)p(w)}{p(D)},
\]
and use the posterior predictive distribution
\[
p(y\mid x,D)=\int p(y\mid x,w)p(w\mid D)\,dw.
\]
That paper shows that BNN uncertainty is an independent and informative indicator of possible misclassification, especially under biased training coverage, but it does not formulate a tradeoff of the NUP type [2212.13564].

Taken together, the literature suggests that NUP is presently a plural concept. Some formulations are rigorous within sharply defined model classes, as in the linear-RNN copy task and the operator-theoretic loss-induced state for input–gradient conjugacy [2502.09287], [2603.19562]. Others are structurally motivated analogies, as in Heaviside canyon geometry and normalized-loss attack operators [2603.06634], [2205.01493]. Still others are simulation-supported coding hypotheses or phenomenological proposals, as in spike-train conjugate channels and retinal threshold–latency variability [1912.11126], [2507.22785].

The most defensible general characterization is therefore narrow. NUP does not currently designate a universal law for all neural systems. It designates a family of complementarity principles in which improved localization, expressivity, or discriminability in one neural domain is accompanied by dispersion, redundancy, ambiguity, or ill-conditioning in a conjugate domain. The specific conjugate pair depends on the paper: function space versus parameter space, input versus input gradient, delay horizon versus temporal width, firing-rate versus co-firing-rate code, or threshold variability versus response-time variability. The unifying theme is structural complementarity rather than probabilistic prediction uncertainty.

Source: https://www.emergentmind.com/topics/neural-uncertainty-principle-nup