Perceptron Impossibility Results

• Perceptron impossibility results are a set of negative theorems that define the limits of perceptron-type classifiers, demonstrating failures in convergence, feasibility, and robust learning.
• They cover diverse challenges including strategic online manipulation, high-dimensional minimax limits, and precise storage capacity bounds in spherical and binary perceptron models.
• The results also expose representational and verification barriers in one-layer feedforward networks, underscoring obstacles in realizing exact neural codes and certifying robustness via interval methods.

Searching arXiv for the cited perceptron impossibility papers to ground the article in current bibliographic records. arxiv_search(query="Perceptron impossibility results strategic perceptron negative spherical perceptron binary perceptron one-layer feedforward networks high dimensional supervised learning", max_results=10) Perceptron impossibility results are negative statements about perceptron-type classifiers, perceptron-style learning rules, or one-layer threshold networks under regimes where classical guarantees, existence claims, or certification procedures break down. In the literature considered here, impossibility takes several distinct forms: the standard online Perceptron can cycle indefinitely under strategic manipulation even when a perfect large-margin separator exists; random spherical and binary perceptron instances become infeasible above explicit load thresholds; high-dimensional Gaussian classification can be minimax-impossible for every supervised learner and therefore for perceptrons in particular; some exact neural codes cannot be realized by one-layer feedforward threshold networks; and interval arithmetic cannot provide complete robustness proofs even for very simple networks (Ahmadi et al., 2020, Stojnic, 2013, Altschuler et al., 2024, Rohban et al., 2013, Giusti et al., 2013, Mirman et al., 2021).

1. Meanings of impossibility in perceptron theory

The phrase “perceptron impossibility results” does not refer to a single theorem or even to a single model. In the works considered here, impossibility appears at multiple levels. At the algorithmic level, the classical Perceptron update can lose its finite-mistake guarantee because the learner updates on manipulated observations rather than on true features (Ahmadi et al., 2020). At the feasibility level, one asks whether there exists any weight vector satisfying all random constraints, as in the spherical and binary perceptrons; impossibility then means infeasibility with overwhelming probability beyond a load threshold (Stojnic, 2013, Altschuler et al., 2024). At the statistical level, impossibility can be minimax and information-theoretic: in a high-dimensional Gaussian model with fixed Bayes error and $n/d \to 0$, every supervised classifier has asymptotic worst-case error at least $1/2$, so perceptrons are ruled out as a special case (Rohban et al., 2013).

A different strand concerns representational limits of one-layer threshold networks when the object of interest is a combinatorial neural code rather than a real-valued input-output map. There, impossibility means that some exact families of output activity patterns are not realizable by any one-layer feedforward network because such codes must be convex (Giusti et al., 2013). Still another strand is verification-theoretic: interval analysis or interval bound propagation may fail to certify robustness even when robust classifiers exist, so the impossibility is a limitation of the proof method rather than of the classifier class itself (Mirman et al., 2021).

This taxonomy suggests that perceptron impossibility results are best understood as a family of negative theorems indexed by what is being denied: convergence, feasibility, learnability, realizability, or certifiability.

2. Strategic online classification and failure of the classical Perceptron

In online strategic classification, the learner does not observe the true feature vector $\mathbf{z}_t$, but rather a manipulated vector $\mathbf{x}_t$ chosen by an agent who wants positive classification and trades off value against manipulation cost. The paper "The Strategic Perceptron" formalizes this through

$$\mathbf{x}_t=\arg\max_{\mathbf{x}} \big[\mathrm{value}(\mathbf{x})-\mathrm{cost}(\mathbf{z}_t,\mathbf{x})\big],$$

with value equal to $1$ for positive classification and $0$ otherwise, and with either $\ell_2$ or weighted $\ell_1$ movement costs. The true data are assumed margin-separable through the origin by some target $\mathbf{w}^*$, with $1/2$0 on positives and $1/2$1 on negatives, so the margin is $1/2$2. In the non-strategic setting, the standard Perceptron predicts $1/2$3, updates by $1/2$4 on mistaken positives and $1/2$5 on mistaken negatives, and has mistake bound $1/2$6. The negative result is that this guarantee can fail completely once $1/2$7 depends on the current classifier (Ahmadi et al., 2020).

The central counterexample for $1/2$8 manipulation uses three true points in $1/2$9,

$\mathbf{z}_t$0

with labels $\mathbf{z}_t$1 negative, $\mathbf{z}_t$2 positive, $\mathbf{z}_t$3 negative, manipulation radius $\mathbf{z}_t$4, and arrival order $\mathbf{z}_t$5, then $\mathbf{z}_t$6. Starting from $\mathbf{z}_t$7, the first mistake on $\mathbf{z}_t$8 yields $\mathbf{z}_t$9. Point $\mathbf{x}_t$0 is then classified positive without manipulating. Point $\mathbf{x}_t$1, however, can move horizontally by $\mathbf{x}_t$2 to $\mathbf{x}_t$3, be observed at the same location as $\mathbf{x}_t$4, and be misclassified as positive, producing update $\mathbf{x}_t$5. Under $\mathbf{x}_t$6, point $\mathbf{x}_t$7 lies at distance $\mathbf{x}_t$8 from the decision boundary, which exceeds $\mathbf{x}_t$9, so it cannot manipulate enough to become positive; it is then misclassified as negative, sending the learner back to $$\mathbf{x}_t=\arg\max_{\mathbf{x}} \big[\mathrm{value}(\mathbf{x})-\mathrm{cost}(\mathbf{z}_t,\mathbf{x})\big],$$0. The resulting two-state oscillation

$$\mathbf{x}_t=\arg\max_{\mathbf{x}} \big[\mathrm{value}(\mathbf{x})-\mathrm{cost}(\mathbf{z}_t,\mathbf{x})\big],$$1

yields an unbounded number of mistakes even though a perfect classifier exists in the strategic environment, namely $$\mathbf{x}_t=\arg\max_{\mathbf{x}} \big[\mathrm{value}(\mathbf{x})-\mathrm{cost}(\mathbf{z}_t,\mathbf{x})\big],$$2, which “works perfectly for the three points as $$\mathbf{x}_t=\arg\max_{\mathbf{x}} \big[\mathrm{value}(\mathbf{x})-\mathrm{cost}(\mathbf{z}_t,\mathbf{x})\big],$$3 can manipulate to be classified positive but $$\mathbf{x}_t=\arg\max_{\mathbf{x}} \big[\mathrm{value}(\mathbf{x})-\mathrm{cost}(\mathbf{z}_t,\mathbf{x})\big],$$4 and $$\mathbf{x}_t=\arg\max_{\mathbf{x}} \big[\mathrm{value}(\mathbf{x})-\mathrm{cost}(\mathbf{z}_t,\mathbf{x})\big],$$5 cannot” (Ahmadi et al., 2020).

The same construction also breaks the Perceptron under weighted $$\mathbf{x}_t=\arg\max_{\mathbf{x}} \big[\mathrm{value}(\mathbf{x})-\mathrm{cost}(\mathbf{z}_t,\mathbf{x})\big],$$6 costs when $$\mathbf{x}_t=\arg\max_{\mathbf{x}} \big[\mathrm{value}(\mathbf{x})-\mathrm{cost}(\mathbf{z}_t,\mathbf{x})\big],$$7, so the failure is not peculiar to Euclidean manipulation. A separate example shows that the usual bias-coordinate reduction also fails strategically. With

$$\mathbf{x}_t=\arg\max_{\mathbf{x}} \big[\mathrm{value}(\mathbf{x})-\mathrm{cost}(\mathbf{z}_t,\mathbf{x})\big],$$8

labels $$\mathbf{x}_t=\arg\max_{\mathbf{x}} \big[\mathrm{value}(\mathbf{x})-\mathrm{cost}(\mathbf{z}_t,\mathbf{x})\big],$$9 negative, $1$0 positive, $1$1 negative, and $1$2, the standard nonzero-threshold extension cycles forever between the separators

$1$3

The underlying reason is that the learner updates on endogenous observations: a negative point can strategically move onto the location of a positive point, the update then changes future best responses discontinuously, and the classical Perceptron proof no longer has its two key ingredients—monotone progress toward $1$4 and controlled norm growth on fixed examples. The impossibility is therefore not a failure of strategic learnability per se; it is a failure of the original Perceptron update rule under strategic observation geometry (Ahmadi et al., 2020).

3. High-dimensional supervised learning limits and perceptrons as a special case

A stronger kind of impossibility arises in high-dimensional supervised classification. In the Gaussian model studied in "An Impossibility Result for High Dimensional Supervised Learning," the label is balanced,

$1$5

and the class-conditional distribution is

$1$6

with unknown means $1$7 and common covariance $1$8. A classifier is any measurable map

$1$9

constructed from $0$0 iid labeled samples $0$1, knowledge of the model family, and a feasible parameter set $0$2, but with no direct knowledge of the true $0$3. The asymptotic regime is

$0$4

and the Bayes difficulty is held constant through

$0$5

Thus the task remains intrinsically nontrivial but does not become easier with dimension (Rohban et al., 2013).

The decisive lower bound is proved already on the spherical subclass

$0$6

equivalently

$0$7

where the only unknown parameter is the direction $0$8. The Bayes classifier is linear,

$0$9

so this is precisely a regime in which one might expect a perceptron-like linear method to succeed. Theorem 1 states that for any sequence of classifiers $\ell_2$0,

$\ell_2$1

Corollary 1 extends the same lower bound to any larger parameter family containing the spherical subclass. Since error $\ell_2$2 is attained by random guessing, the asymptotic minimax performance of every supervised learning rule collapses to chance (Rohban et al., 2013).

For perceptron theory, the implication is direct. The impossibility is not algorithm-specific, not a failure of covariance estimation alone, and not a byproduct of difficult Bayes geometry: the Bayes rule is linear, the Bayes error $\ell_2$3 can be made arbitrarily close to $\ell_2$4, and yet every supervised learner still has worst-case asymptotic error at least $\ell_2$5. The paper further states that “nontrivial asymptotic minimax error probability is attainable only for parametric subsets of zero Haar measure,” and highlights sparsity as the type of strong structural prior required to escape the lower bound. This suggests that in the regime $\ell_2$6, the relevant impossibility is fundamentally informational rather than computational (Rohban et al., 2013).

4. Negative spherical perceptron and infeasibility above storage capacity

The spherical perceptron poses a random feasibility problem rather than an online learning problem. One seeks a unit vector $\ell_2$7 such that

$\ell_2$8

where $\ell_2$9 is an $\ell_1$0 random matrix with $\ell_1$1 and i.i.d. standard normal entries in the asymptotic analysis. The paper "Negative spherical perceptron" focuses on the regime $\ell_1$2. Feasibility is characterized by the min-max quantity

$\ell_1$3

with $\ell_1$4 corresponding to feasibility and $\ell_1$5 to infeasibility. The benchmark threshold throughout is Gardner’s formula

$\ell_1$6

equivalently $\ell_1$7, where

$\ell_1$8

For $\ell_1$9, this is already rigorously exact; for $\mathbf{w}^*$0, the paper establishes it only as an upper bound on capacity (Stojnic, 2013).

The first impossibility statement is therefore one-sided but rigorous: for any fixed $\mathbf{w}^*$1, if

$\mathbf{w}^*$2

then with overwhelming probability the spherical perceptron is infeasible. This confirms Talagrand’s conjecture that the Gardner curve remains an upper bound even in the negative-threshold regime. The paper then develops a stronger Gaussian-comparison or lifting method. After introducing a Gordon-type exponential comparison, it derives a lower bound on $\mathbf{w}^*$3 involving

$\mathbf{w}^*$4

and proves the theorem that if $\mathbf{w}^*$5 and

$\mathbf{w}^*$6

then the feasibility problem

$\mathbf{w}^*$7

is infeasible with overwhelming probability (Stojnic, 2013).

The strengthened condition yields a potentially smaller rigorous upper bound $\mathbf{w}^*$8. Numerically, the improvement is negligible near $\mathbf{w}^*$9 but more visible for larger negative thresholds; for example, at $1/2$00 the paper reports

$1/2$01

The exact threshold for $1/2$02 remains open because the paper provides only infeasible regions, not a matching achievability theorem. The impossibility contribution is therefore an upper-bound theory for storage capacity in the negative spherical regime, together with evidence that the true capacity may lie strictly below the naive Gardner continuation (Stojnic, 2013).

5. Binary perceptron capacity as a non-existence theorem

The binary perceptron asks whether a random Gaussian constraint matrix

$1/2$03

admits a sign vector

$1/2$04

such that

$1/2$05

If $1/2$06 denotes the $1/2$07-th row, feasibility is equivalent to

$1/2$08

This is a pure existence question: the issue is not whether any algorithm can find $1/2$09, but whether such a binary weight vector exists at all. The note "A note on the capacity of the binary perceptron" proves the rigorous upper bound that, with high probability, the capacity of the binary perceptron is at most $1/2$10; equivalently, at $1/2$11,

$1/2$12

This sharpens earlier rigorous upper bounds $1/2$13 and $1/2$14, while remaining above the Krauth–Mézard prediction $1/2$15 (Altschuler et al., 2024).

The ordinary first moment is too weak. Writing

$1/2$16

one has

$1/2$17

so only the trivial upper bound $1/2$18 follows immediately. The note instead conditions on the spherical perceptron free energy

$1/2$19

Using rotational invariance, a fixed binary vector rotated by a Haar orthogonal matrix becomes uniform on the sphere of radius $1/2$20, so its conditional success probability is bounded by $1/2$21. The imported Shcherbina–Tirozzi theorem gives concentration of $1/2$22 around the Gardner–Derrida formula

$1/2$23

with $1/2$24 and $1/2$25 for $1/2$26. The proof reduces to showing

$1/2$27

which is done by evaluating at $1/2$28 and using the exact identity

$1/2$29

Then the feasible spherical volume fraction is typically small enough to beat the hypercube entropy $1/2$30, and a Markov argument yields unsatisfiability with high probability (Altschuler et al., 2024).

This is a canonical perceptron impossibility result in the strict storage-capacity sense: above a specified load, no binary solution exists with high probability. The theorem is asymptotic, specific to random Gaussian constraints and zero margin, and unconditional as stated in the note, although the proof depends on previously established spherical-perceptron results (Altschuler et al., 2024).

6. One-layer feedforward networks and combinatorial code obstructions

In "A no-go theorem for one-layer feedforward networks," a one-layer feedforward network consists of uncoupled output neurons driven directly by nonnegative input firing rates: $1/2$31 with $1/2$32 for $1/2$33 and $1/2$34 for $1/2$35. Each output neuron is therefore a threshold unit, and the induced combinatorial neural code is

$1/2$36

where

$1/2$37

Because such codes are convex, every non-convex code is automatically unrealizable by a one-layer feedforward network. The paper strengthens this by proving that if $1/2$38 is convex, $1/2$39, and

$1/2$40

then $1/2$41 must be contractible. Hence any missing codeword whose localization is noncontractible yields an impossibility theorem for one-layer feedforward threshold networks (Giusti et al., 2013).

A concrete unrealizable example is

$1/2$42

which is non-convex and therefore cannot be realized by a one-layer feedforward network. More generally, Theorem $1/2$43 identifies a large forbidden family via subcodes satisfying three explicit combinatorial conditions, again concluding non-convexity and thus unrealizability (Giusti et al., 2013).

The paper simultaneously establishes a sharp boundary on these impossibility statements. If one keeps only maximal codewords,

$1/2$44

then the no-go theorem runs in the opposite direction: for every collection $1/2$45 of maximal patterns, there exists a one-layer feedforward network such that

$1/2$46

Thus exact code realization is obstructed, but realization of maximal patterns is universal. Under Dale’s law, however, expressivity collapses again: the paper proves that a one-layer feedforward network respecting Dale’s law has exactly one maximal pattern,

$1/2$47

The resulting impossibility theory is therefore unusually nuanced: one-layer perceptron-like networks are severely limited at the level of fine combinatorics, unrestricted at the level of maximal patterns, and sharply restricted again under biologically motivated sign constraints (Giusti et al., 2013).

7. Verification-theoretic limits of interval methods for simple networks

The paper "The Fundamental Limits of Interval Arithmetic for Neural Networks" does not state a standalone impossibility theorem for a single linear threshold unit, but it is directly relevant to perceptron-style simplicity because it establishes negative results for interval analysis on feed-forward and one-hidden-layer networks. The central general theorem states that if a feed-forward $1/2$48-network is non-injective at some output value $1/2$49, in the sense that $1/2$50 is compact and non-empty, then there exists a box $1/2$51 such that

$1/2$52

where $1/2$53 is the interval transformer. This yields the corollary that there is no feed-forward $1/2$54-network that is a completely $1/2$55-provable classifier for the three-point dataset

$1/2$56

The mechanism is geometric: any continuous classifier with labels $1/2$57 on those three points must cross zero at least twice, and non-injectivity then forces interval imprecision (Mirman et al., 2021).

The stronger theorem concerns one-hidden-layer networks

$1/2$58

For any $1/2$59, define flips by

$1/2$60

Then no single-layer $1/2$61-network can provably $1/2$62-robustly classify

$1/2$63

or more flips. The points are pairwise distance $1/2$64 apart, so for $1/2$65 their robust intervals are disjoint. The theorem does not assert nonexistence of robust classifiers. On the contrary, the paper also proves that for any finite one-dimensional dataset there exists a one-hidden-layer $1/2$66-network that completely robustly classifies it. The impossibility is therefore about interval analysis or interval bound propagation as a certification method, not about expressive power (Mirman et al., 2021).

Taken together with the earlier strands, these results mark a final extension of perceptron impossibility theory into proof systems. Algorithmic impossibility denies convergence, statistical impossibility denies learnability from limited samples, feasibility impossibility denies the existence of satisfying weights, representational impossibility denies realization of exact codes, and verification impossibility denies complete certification even when robust solutions exist. This suggests that the term “perceptron impossibility results” names a layered negative theory rather than a single obstruction, with each layer tracking a different interface between linear-threshold structure and the regime in which it is deployed.