Complete or Sparse: A Tale of Two Identifiabilities

Abstract: Understanding when neural networks learn interpretable representations is central to mechanistic interpretability. We study this through identifiability: when can supervised learning recover the true latent variables underlying the data? We introduce two data generating processes distinguished by their causal direction where either labels cause latents (generative) or latents cause labels (discriminative). When the representation has at least as many dimensions as the latent space, we prove that the optimal encoders are linear in the latents. Conversely, when dimensions are insufficient, identifiability generically fails and encoders become nonlinear. We then show that additional constraints yield recovery of individual latent variables, not just linear mixtures. In the generative setting, non-negative activations and energy efficiency force each neuron to encode exactly one latent, providing a precise account of selective grandmother cell-like responses. In the discriminative setting, when latents are sparse and the true measurement matrix satisfies the spark condition, the learned encoder inherits compressed sensing structure, enabling sparse dictionary learning to perfectly recover individual latents. This constitutes, to our knowledge, the first identifiability result for undercomplete nonlinear independent component analysis. Our framework unifies nonlinear ICA, compressed sensing, and neural network interpretability, delineating exactly when representations transition from nonlinear distortions to linear representations and, under additional constraints or steps, to clean, axis-aligned codes. Theoretical predictions are validated on synthetic benchmarks and on activations drawn from vision models, LLMs, and macaque inferotemporal cortex.

• The paper demonstrates that under complete identifiability ($m≥n'$), all task-relevant latent variables can be recovered up to a linear transformation, whereas under sparse identifiability ($m<n'$), the representation must be nonlinear, enforcing non-perfect recovery of latent variables.
• Nonlinear ICAs and supervised learning involve greatly varied assumptions for data generation, and the paper translates these into theoretical predictions about representation recovery. The connection between the capacity, data-generating process and identifiability of representation are elaborated with and proper implementation is required for obtaining the true latent variables.
• The discovery that intersystem unit correlation stabilizes with capacity levels provides opportunities to explore and better understand why and how biological neurons perform complex tasks through these processes and it illustrates necessity for evaluation with real-world data.

Problem formulation and central thesis

“Complete or Sparse: A Tale of Two Identifiabilities” (2610.20851) studies when supervised neural representations recover latent variables in an interpretable form. The paper frames monosemanticity, polysemanticity, and sparse-autoencoder recovery as consequences of identifiability under different data-generating processes and capacity regimes.

The analysis assumes an invertible nonlinear observation map $x=g(z)$ from latent variables $z\in\mathbb{R}^n$ to inputs, followed by a learned continuous encoder $f$ whose latent-to-representation composition is $h=f\circ g$. Classification uses a linear readout:

$q(y\mid z)=\operatorname{softmax}(Uh(z)).$

The ground-truth posterior is assumed to be log-linear,

$p(y\mid z)=\operatorname{softmax}(W^*z),$

with effective latent dimension $n'=\operatorname{rank}(W^*)$. The central capacity parameter is therefore not necessarily the full latent dimension $n$, but the dimension $n'$ visible to the supervised task.

Two causal data-generating processes are distinguished. In the generative setting, labels cause latents, $Y\rightarrow Z$, and the proposed latent distribution is a class-conditional product of nonnegative exponential variables. In the discriminative setting, latents cause labels, $z\in\mathbb{R}^n$0, and the classifier depends on a potentially lower-dimensional measurement $z\in\mathbb{R}^n$1, where $z\in\mathbb{R}^n$2.

The paper’s principal claim is a capacity-dependent dichotomy:

• If $z\in\mathbb{R}^n$3, every population cross-entropy optimum recovers the classifier-relevant latent variables up to a linear transformation.
• If $z\in\mathbb{R}^n$4, linear recovery is generically impossible; the optimal encoder must be nonlinear.
• Under additional structural assumptions, the residual linear ambiguity can be reduced to permutation and scaling. In the generative setting this produces axis-aligned neurons, whereas in the discriminative setting it produces a compressed-sensing geometry that permits recovery of sparse latent variables from distributed codes.

This organization gives the paper a unified account of apparently contradictory observations: individual neurons can be selective when the representation is complete, while mixed selectivity and sparse decoding arise when representations are constrained or task-relevant dimensions are compressed.

Linear identifiability at and above effective capacity

The first theorem establishes the structural result on which the rest of the paper depends. When $z\in\mathbb{R}^n$5, every global minimizer of population cross-entropy has the form

$z\in\mathbb{R}^n$6

where $z\in\mathbb{R}^n$7 has rank $z\in\mathbb{R}^n$8 and $z\in\mathbb{R}^n$9.

Thus, the classifier-visible component of the representation is linear in $f$0. Any nonlinear component must lie in the null space of the readout and is therefore invisible to the classification loss. This distinction is important: the theorem does not assert that the entire learned network is linear. It asserts that the task-relevant projection of its representation is linear, while unused representation dimensions can remain arbitrary, including nonlinear.

The proof relies on the log-ratio invariance of the softmax. At a zero-KL optimum, equality of the true and learned posteriors implies that every class-logit difference computed from $f$1 equals the corresponding linear function of $f$2. Since the class-difference vectors span an $f$3-dimensional subspace, these inner-product constraints force the readout-visible component of $f$4 to be linear.

The result formalizes a supervised analogue of nonlinear ICA identifiability. However, the equivalence class is initially broad: $f$5 can be any rank-$f$6 linear embedding compatible with the readout. Consequently, linear identifiability alone does not imply individual neurons, semantic axes, or permutation alignment. The classifier can represent a set of latent variables through arbitrary mixtures.

The experiments support this result across distinct latent distributions. In exact-capacity experiments, both linear-output and nonnegative-output encoders achieved near-perfect linear alignment, even when stricter permutation-based alignment remained poor. This separation is empirically important: a representation can be almost perfectly linearly equivalent to the latent variables while individual neurons remain strongly polysemantic.

Capacity insufficiency forces nonlinear representations

The second theoretical result concerns the undercomplete regime, $f$7. Under regularity conditions on the support and density of $f$8, the paper proves that the global cross-entropy optimum is generically nonlinear. A linear encoder cannot preserve all task-relevant logit variation through an $f$9-dimensional bottleneck when the effective latent dimension is larger than $h=f\circ g$0.

This result is stronger than the statement that linear representations are insufficient for exact prediction. It states that among continuous encoders of the available dimension, the optimum itself must distort the latent geometry nonlinearly. The bottleneck therefore changes the type of identifiable representation rather than merely increasing approximation error.

The experiments fix $h=f\circ g$1 and sweep $h=f\circ g$2. They compare linear encoders with MLPs across exponential, Gaussian, and hyperspherical latent distributions. For every distribution and every $h=f\circ g$3, the MLP achieves lower final KL divergence than the linear encoder. At and above the threshold, both architectures converge to near-zero KL divergence.

Figure 1: A capacity transition at $h=f\circ g$4, below which nonlinear MLP encoders outperform linear encoders and above which both achieve near-zero KL divergence.

The transition at $h=f\circ g$5 is consistent across all three distributions, supporting the claim that the phenomenon is governed primarily by effective task dimension rather than by the particular latent prior. Its implication is that undercomplete representations should not be interpreted as linearly mixed versions of the original latents in general. The nonlinear distortion is not merely an optimization artifact under the paper’s population-level assumptions; it is required by the geometry of the supervised bottleneck.

The theorem is nevertheless conditional on global optimization, continuity, a sufficiently rich encoder class, and a support condition ensuring that the null space of a candidate linear map intersects the interior of the latent support. These assumptions exclude several finite-sample and architectural effects that may alter the observed transition.

Generative identifiability and axis-aligned neurons

The paper’s first route from linear identifiability to individual latent variables uses a generative model with three additional ingredients:

1. Nonnegative latent variables and activations.
2. Exact capacity, $h=f\circ g$6.
3. A fixed within-class variance together with minimum activation energy.

The proposed conditional distribution is factorized exponential:

$h=f\circ g$7

with a class-independent product of rates, equivalent to constant conditional entropy. This construction yields a log-linear posterior while retaining nonnegative, sparse-like latents with varying norms.

Under exact capacity, the null-space residual $h=f\circ g$8 disappears because the readout has full column rank. The encoder is therefore an invertible linear map $h=f\circ g$9. Nonnegative activations imply that $q(y\mid z)=\operatorname{softmax}(Uh(z)).$0 is entrywise nonnegative. The energy objective then penalizes rows of $q(y\mid z)=\operatorname{softmax}(Uh(z)).$1 that combine multiple positive latent coordinates: cross terms in the expected squared activation are nonnegative, so the minimum is achieved only when each row contains a single nonzero entry. Invertibility then forces each column to be used exactly once.

The resulting identifiability class is

$q(y\mid z)=\operatorname{softmax}(Uh(z)).$2

where $q(y\mid z)=\operatorname{softmax}(Uh(z)).$3 is a permutation matrix and $q(y\mid z)=\operatorname{softmax}(Uh(z)).$4 is a positive diagonal matrix. Each neuron therefore corresponds to exactly one latent coordinate, up to ordering and positive scale.

Figure 2: Linear alignment is obtained broadly, whereas axis alignment emerges specifically from the combination of sparse nonnegative latents and nonnegative activations.

The experiments compare the proposed sparse nonnegative DGP with a hyperspherical DGP used in prior nonlinear-ICA analyses. All configurations reach near-perfect linear alignment. Only the combination of the sparse DGP and a nonnegative output converges toward permutation-and-scaling alignment. Hyperspherical latents and linear outputs retain mixed representations, despite achieving comparable linear recovery.

This contrast supports the paper’s central distinction between linear identifiability and axis identifiability. Nonnegativity alone is insufficient; it must interact with the latent support and the energy-selection principle. The result provides a precise statistical mechanism for selective, grandmother-cell-like responses, but only under a restrictive model of the latent distribution and optimization criterion.

Optimization behavior also exposes an important qualification. Axis alignment is considerably slower and less reliable than linear alignment. In a finite-budget sweep without weight decay, only 3 of 10 sparse-ReLU runs reached MCC $q(y\mid z)=\operatorname{softmax}(Uh(z)).$5 within $q(y\mid z)=\operatorname{softmax}(Uh(z)).$6 updates, with median final MCC $q(y\mid z)=\operatorname{softmax}(Uh(z)).$7 and interquartile range $q(y\mid z)=\operatorname{softmax}(Uh(z)).$8. Adding weight decay improved both reliability and speed:

Weight decay	Runs reaching MCC $q(y\mid z)=\operatorname{softmax}(Uh(z)).$9	Median final MCC	Median time to threshold
$p(y\mid z)=\operatorname{softmax}(W^*z),$0	$p(y\mid z)=\operatorname{softmax}(W^*z),$1	$p(y\mid z)=\operatorname{softmax}(W^*z),$2	$p(y\mid z)=\operatorname{softmax}(W^*z),$3 updates
$p(y\mid z)=\operatorname{softmax}(W^*z),$4	$p(y\mid z)=\operatorname{softmax}(W^*z),$5	$p(y\mid z)=\operatorname{softmax}(W^*z),$6	$p(y\mid z)=\operatorname{softmax}(W^*z),$7 updates
$p(y\mid z)=\operatorname{softmax}(W^*z),$8	$p(y\mid z)=\operatorname{softmax}(W^*z),$9	$n'=\operatorname{rank}(W^*)$0	$n'=\operatorname{rank}(W^*)$1 updates
$n'=\operatorname{rank}(W^*)$2	$n'=\operatorname{rank}(W^*)$3	$n'=\operatorname{rank}(W^*)$4	$n'=\operatorname{rank}(W^*)$5 updates

Figure 3: Axis-aligned recovery is substantially more sensitive to initialization and finite optimization budgets than linear recovery, while moderate weight decay improves convergence.

The optimization findings matter theoretically because the population theorem identifies the selected optimum, not the time required for gradient-based training to reach it. In practice, a model may exhibit strong linear disentanglement while failing to display axis-aligned units within a feasible training budget.

Discriminative identifiability and compressed sensing

The second route applies to discriminative tasks in which labels depend on a compressed measurement of a higher-dimensional sparse latent:

$n'=\operatorname{rank}(W^*)$6

Here $n'=\operatorname{rank}(W^*)$7 is a measurement matrix, with $n'=\operatorname{rank}(W^*)$8. The supervised task does not identify the full latent vector directly; it identifies the row space, or equivalently the null space, of the measurement operator.

The paper assumes that the latents are $n'=\operatorname{rank}(W^*)$9-sparse and that $n$0 satisfies the spark condition,

$n$1

This condition ensures that no nonzero $n$2-sparse vector lies in the null space of $n$3, so distinct $n$4-sparse latent vectors cannot produce the same measurement.

The theorem states that the classifier-visible component of the learned representation, $n$5, satisfies

$n$6

Consequently, $n$7 and $n$8 have the same row space and the same sparse injectivity properties. The learned representation may differ from the ground-truth measurement by an invertible transformation in measurement space, but this transformation does not affect sparse recovery.

Figure 4: As training approaches zero KL divergence, the learned measurement operator aligns with the true row space, enabling sparse recovery from compressed activations.

The synthetic experiments use $n$9, $n'$0, and sparsity levels $n'$1. As the training KL approaches zero, row-space alignment undergoes a sharp transition toward 1. Supervised Lasso achieves near-perfect $n'$2 for all tested sparsity levels, demonstrating recovery of the full 16-dimensional latent despite an 8-dimensional representation. Dense OLS instead plateaus at approximately

$n'$3

as expected from a dense linear decoder that cannot exploit sparse injectivity. Unsupervised sparse dictionary learning reaches MCC approximately $n'$4 in the 1-sparse condition, with performance declining as $n'$5 increases.

The distinction between supervised and unsupervised recovery is consequential. The theorem guarantees injectivity of the learned measurement on the sparse latent set, but it does not guarantee that an unsupervised dictionary-learning algorithm will estimate the original coordinate system perfectly from finite data. The decline in MCC with increasing sparsity level reflects this gap between information-theoretic recoverability and practical axis recovery.

The result provides a formal explanation for why sparse autoencoders can recover interpretable features from representations that are themselves distributed or polysemantic. The original network need not dedicate one neuron to each latent. It is sufficient that the learned representation preserve the relevant sparse measurement geometry.

Cross-system preservation in models and cortex

The paper extends the identifiability framework to real activations by testing whether interpretability predicts cross-system feature preservation. If individual units correspond to identifiable latent variables, then highly disentangled units should have counterparts in independently trained systems exposed to similar data.

Because ground-truth latents are unavailable in real systems, neuron interpretability is quantified with the Mechanistic Interpretability Score (MIS). Cross-system correspondence is quantified with the Maximum Matching Unit (MMU), defined as the maximum absolute correlation between a unit in one system and any unit in another. The prediction is a positive MIS–MMU association.

Figure 5: More interpretable units exhibit stronger cross-system correspondence in vision models, LLMs, and macaque inferotemporal cortex.

Across four ImageNet-trained vision systems—VGG-19, ResNet-50, ViT-B/16, and DINOv2 ViT-S/14—MIS predicts MMU positively for all 12 directed model pairs. The same relationship appears more weakly across GPT-2, Pythia-70M, and TinyLlama, evaluated on frequent Wikipedia unigrams.

The biological analysis uses recordings from 58 neurons in one macaque and 110 neurons in another, each responding to 3,200 natural images. The directional correlations are positive but modest: Tito-to-Chabo gives $n'$6 with $n'$7, while Chabo-to-Tito gives $n'$8 with $n'$9. These values do not provide conventional evidence against a zero-correlation null at standard significance thresholds. The paper appropriately attributes the weaker and asymmetric result partly to the unequal candidate pools, but the biological result remains substantially less decisive than the synthetic and artificial-model analyses.

The real-data findings are therefore consistent with, but do not establish, the proposed identifiability mechanism. MIS is only a proxy for latent disentanglement, MMU is sensitive to unit scaling and representational transformations, and the compared systems may differ in architecture, training objective, and data distribution.

Limitations and open questions

The strongest theoretical claims depend on assumptions that are idealized and not generally satisfied by natural data.

The axis-alignment theorem requires constant conditional entropy across classes. In the ablations, increasing the across-class standard deviation of the conditional entropy to 2 nats reduced MCC to $Y\rightarrow Z$0 with a bias-free readout, although an affine readout restored MCC to $Y\rightarrow Z$1. Thus, class-dependent normalization terms can substantially disrupt axis alignment even when linear alignment remains high.

Nonnegative latent support is also central. Introducing signed latents reduced MCC from the reference level near $Y\rightarrow Z$2 to $Y\rightarrow Z$3 when signs were flipped with probability $Y\rightarrow Z$4, and to $Y\rightarrow Z$5 when the probability was $Y\rightarrow Z$6. The theorem therefore does not directly extend to signed sparse sources, where positive and negative mixtures create additional symmetries.

The global-optimality assumption is particularly strong. The theoretical results concern population cross-entropy minima, whereas the experiments use finite datasets and gradient-based optimization. The pronounced seed dependence and the large improvement from weight decay show that optimization is not a secondary detail. The selected finite-time solution can differ materially from the population optimum.

The discriminative compressed-sensing result also assumes exact sparse support, a known effective factorization, and a spark condition on the true measurement matrix. Real neural features may be approximately sparse, structured rather than coordinate-sparse, or distributed on a manifold. The experiments test only small synthetic systems with $Y\rightarrow Z$7, $Y\rightarrow Z$8, and $Y\rightarrow Z$9; they do not establish that the same recovery guarantees apply to large-scale model activations.

Finally, the paper’s real-data analysis is correlational. Positive MIS–MMU associations are compatible with shared latent structure, but they could also arise from common preprocessing, dataset statistics, architectural inductive biases, or low-dimensional covariance structure. The macaque correlations are positive but statistically weak. An open question is whether an intervention that selectively increases identifiability—rather than merely interpretability as measured by MIS—causally increases cross-system feature preservation.

Conclusion

The paper develops a unified identifiability framework in which representation geometry is controlled by effective capacity, causal direction, latent sparsity, and architectural constraints. With $z\in\mathbb{R}^n$00, supervised cross-entropy identifies the task-relevant representation up to a linear transformation. With $z\in\mathbb{R}^n$01, the optimum is generically nonlinear. Additional structure resolves the linear ambiguity in two different ways: nonnegative sparse generative coding plus energy minimization yields axis-aligned neurons, while sparse discriminative coding plus a spark condition yields compressed-sensing recovery from distributed representations.

The empirical results support the predicted capacity transition, the distinction between linear and axis alignment, the compressed-sensing interpretation of sparse autoencoders, and positive cross-system correspondence of interpretable units. At the same time, the results show that axis alignment is highly dependent on latent support, normalization assumptions, regularization, and optimization time. The paper’s main contribution is therefore not a general guarantee of monosemanticity, but a precise characterization of the assumptions under which neural representations are identifiable as linear mixtures, individual axes, or sparse recoverable codes.