Linear-Readout Floors and Threshold Recovery in Computation in Superposition

Abstract: Two recent approaches to computation in superposition reach different recursive capacity regimes: Hänni et al. certify $\tilde{O}(d<sup>{3/2})$ computable features in width $d$ via an approximate-linear recursive template, while Adler and Shavit reach near-quadratic capacity (up to logarithmic factors) using thresholded Boolean recovery. The main contribution of this paper is conceptual: we argue these results are not contradictory because they maintain different interface invariants, and we formalize the distinction. As a tool, we record a rank-trace Welch-type lower bound for biorthogonal linear readouts: for $F \gg d$, the worst-case off-diagonal cross-talk of any unit-diagonal linear readout is $Ω(d<sup>{-1/2})$, and the bound is tight on average for unit-norm tight frames. At quadratic feature load $F=d<sup>2$, random-support threshold recovery succeeds for sparsities $s=O(d/\log d)$, while linear readouts still incur $Ω(s/d)$ average per-coordinate squared error on Bernoulli sparse states. Matching the Welch floor against the published tolerance of the Hänni correction layer explains the $d<sup>{3/2}$ scale as a compatibility threshold for that template, not a universal upper bound. Robust nonlinear reset beyond the Hänni template is left open.

• The paper establishes the Biorthogonal Welch Floor, a geometric lower bound that limits linear readouts to O(d^(3/2)) capacity due to unavoidable cross-talk.
• It contrasts linear interfaces with threshold recovery, demonstrating that threshold-based mechanisms bypass strict geometric constraints to achieve nearly quadratic feature loads.
• It synthesizes concepts from compressed sensing and mechanistic interpretability to propose a taxonomy of recursive capacity regimes and outlines open problems in nonlinear reset design.

Geometric Bounds and Interface Invariants in Computation in Superposition

Introduction

"Linear-Readout Floors and Threshold Recovery in Computation in Superposition" (2605.01192) investigates the computational capacity of neural networks operating in the superposition regime, focusing on the distinction between linear and threshold-based interfaces for recursive computation with sparse Boolean features. The paper rigorously analyzes and formalizes the geometric constraints imposed by linear readouts, derives tight lower bounds for cross-talk, and contrasts these with thresholded Boolean recovery mechanisms, clarifying the compatibility scales of several prominent recent constructions. The work synthesizes results from compressed sensing, feature geometry, and mechanistic interpretability to establish a taxonomy of capacity regimes available to neural architectures.

Superposition and Recent Capacity Regimes

The superposition hypothesis posits that neural networks represent or compute more conceptual features than neurons, making superposition a central concept in mechanistic interpretability. Two approaches to recursive computation in superposition have produced divergent capacity regimes:

• Hänni et al.: Their $\varepsilon$-linear recursive template achieves $O(d^{3/2})$ computable features in width-$d$ networks, relying on linear decodability with error-correction layers.
• Adler and Shavit: Their threshold-based Boolean recovery leverages margin-based thresholding to enable near-quadratic capacity, $O(d^2/\log d)$, avoiding the accumulation of linear error.

The resulting gap spans a factor of $O(d^{1/2})$ between active computable feature loads.

Biorthogonal Welch Floor: Linear Readout Bound

The primary theoretical contribution is a geometric constraint on linear readout interfaces, termed the Biorthogonal Welch Floor. For a feature code $\Psi \in \mathbb{R}^{d \times F}$ and a readout $G \in \mathbb{R}^{F \times d}$ with unit diagonal, the worst-case off-diagonal cross-talk is bounded below:

$$\frac{1}{F(F-1)} \sum_{i \neq j} |(G\Psi)_{ij}|^2 \geq \frac{F-d}{d(F-1)}$$

When $F \gg d$, this implies unavoidable cross-talk of order $\Omega(d^{-1/2})$. The bound is shown to be tight for unit-norm tight frames. As a corollary, small-error $O(d^{3/2})$0-linear reset with $O(d^{3/2})$1 is impossible for superlinear feature loads, explaining why linear recursive templates (as in Hänni et al.) naturally stop at the $O(d^{3/2})$2 scale.

Threshold Recovery and Interface Distinction

Threshold recovery mechanisms, such as those used by Adler and Shavit, are not obstructed by the Welch floor. Coherence-based threshold recovery is analyzed: for a unit-norm code with coherence $O(d^{3/2})$3, exact recovery for $O(d^{3/2})$4-sparse states by thresholding is feasible if $O(d^{3/2})$5 for noise $O(d^{3/2})$6. Random feature codes with $O(d^{3/2})$7 yield coherence $O(d^{3/2})$8, so threshold recovery succeeds for $O(d^{3/2})$9 even at quadratic feature loads.

Crucially, linear readout interfaces require every inactive score to be $d$0, whereas threshold recovery only demands that aggregate interference across the sparse support remains below a fixed margin. This interface distinction is formally separated, demonstrating that thresholded Boolean recovery can coexist with geometric constraints that linear readout faces.

Distributional Separation of Recovery Criteria

A distributional analysis quantifies the separation:

• Linear Readout Error: Any unit-diagonal linear readout at $d$1 must incur average squared per-coordinate error $d$2 for Bernoulli $d$3-sparse states.
• Threshold Recovery Success: Random-support threshold recovery in the same regime succeeds with probability $d$4 for supports of size $d$5.

This demonstrates non-equivalence between linear readout and threshold recovery invariants and clarifies why threshold-based frameworks escape linear interface floor constraints.

Interpolation and Open Problems

An open-problem framework is formulated around robust nonlinear or threshold resets. Any reset mechanism attaining input tolerance scaling as $d$6 would achieve feature load $d$7, for $d$8. However, such resets necessarily leave the $d$9-linear class to avoid contradictions with the Welch floor. This interpolation formalizes the gap and identifies the construction of such resets as a central open problem in recursive computation in superposition.

Taxonomy of Capacity Regimes

The paper synthesizes capacity scales from different domains:

• Compressed Sensing: Passive storage, $O(d^2/\log d)$0, ($O(d^2/\log d)$1 from compressed sensing theory).
• Recursive Linear Computation: Hänni-compatible, $O(d^2/\log d)$2.
• Thresholded Recovery: Adler–Shavit, between $O(d^2/\log d)$3 and $O(d^2/\log d)$4.
• Passive Representation: Johnson–Lindenstrauss, $O(d^2/\log d)$5.

The hierarchy is not monotonic for all tasks; recursive computation, passive representation, and linear decodability measure distinct capacities.

Implications and Future Directions

The geometric floor derived for linear interfaces mathematically limits the recursive capacity attainable by linear decoders in superposition. This necessitates either a change in recursion invariants or the introduction of nonlinear interfaces for scaling capacity beyond $O(d^2/\log d)$6. Threshold recovery mechanisms, leveraging margin separation and sparse interference, exemplify such invariants and are compatible with quadratic-scale feature loads in randomized settings.

The distinction between passive storage and active recursive computation is formalized, and implications extend to neural architecture design: error-correction, thresholding, and nonlinear reset modules may become indispensable for exploiting higher-dimensional superpositional capacities. Future theoretical work is required to construct explicit robust threshold resets, establish universal recursive capacity bounds, and explain empirical scaling phenomena observed in sparse autoencoder studies.

Conclusion

This paper rigorously delineates the geometric obstruction imposed on linear interfaces in computation in superposition. The Biorthogonal Welch Floor shows that linear readouts at superlinear feature loads incur unavoidable cross-talk, explaining the compatibility scale of the Hänni recursive template. Threshold recovery, especially in random feature codes, escapes this obstruction, enabling near-quadratic capacity as achieved by Adler and Shavit. The distinction between linear and threshold invariants underpins both theoretical taxonomy and practical architecture design. The central open problem is the explicit construction of robust nonlinear or threshold reset mechanisms that extend recursive computable capacity beyond the linear floor, potentially unlocking new scaling laws and interpretability regimes for large neural systems.