SOSAE: Self-Organizing Sparse AutoEncoder

• SOSAE is a self-organizing sparse autoencoder that dynamically determines the effective bottleneck size through a position-weighted sparsity regularizer.
• The method organizes latent features by penalizing later coordinates, ensuring that non-active dimensions can be truncated without loss of essential information.
• Empirical results on image datasets demonstrate that SOSAE achieves significant compression and robustness improvements over conventional sparse autoencoders.

Searching arXiv for the cited papers and closely related sparse autoencoder work. Self-Organizing Sparse AutoEncoder (SOSAE) denotes a class of autoencoder methods in which the effective latent dimensionality or active sparse support is not fixed entirely by a manually chosen global hyperparameter, but emerges during training from an explicit organizing mechanism. In the strict sense, the name refers to the model introduced in "SOSAE: Self-Organizing Sparse AutoEncoder" (Modi et al., 7 Jul 2025), where a position-weighted sparsity regularizer causes useful information to concentrate in early latent coordinates and non-active dimensions to be pushed toward the end of the feature vector, enabling truncation of the tail without loss of information. In a broader, conceptual sense, later sparse autoencoder research has developed closely related adaptive-sparsity mechanisms—most notably sparsemax-based dynamic attention SAEs (Wang et al., 16 Apr 2026), adaptive allocation SAEs (Ayonrinde, 2024), and identifiable SAE variants (Nelson et al., 29 May 2026)—that are SOSAE-like in spirit because they replace rigid, globally fixed sparsity patterns with input-dependent or training-emergent organization.

1. Definition and conceptual scope

In (Modi et al., 7 Jul 2025), SOSAE is introduced to address a practical bottleneck in autoencoder design: choosing the hidden-layer size. The paper states that standard practice treats bottleneck dimensionality as a fixed hyperparameter and commonly relies on grid search, random search, mutual-information-based methods, elbow analysis, or similar iterative procedures. SOSAE replaces that external search with a training objective that dynamically adapts the dimensionality of the feature space to the optimal size. Its defining claim is that sparsity is induced in a structured way that permits truncation of the non-active part of the feature vector without loss of information (Modi et al., 7 Jul 2025).

The distinguishing feature is not sparsity alone, but ordered sparsity. Conventional $L_1$, $L_2$, and K-sparse penalties can produce scattered zeros, so latent coordinates cannot be safely truncated merely because they appear late in the vector. SOSAE instead imposes a positional ordering: later latent coordinates are penalized more heavily than earlier ones, so inactive or weakly useful components are driven toward the end of the latent representation (Modi et al., 7 Jul 2025).

A broader literature extends this organizing principle beyond latent-width adaptation. "Improving Sparse Autoencoder with Dynamic Attention" (Wang et al., 16 Apr 2026) does not use the name SOSAE, but it is explicitly described as very close in spirit because sparsity is determined per input by sparsemax thresholding rather than by a fixed $K$ or an auxiliary sparsity penalty. "Adaptive Sparse Allocation with Mutual Choice & Feature Choice Sparse Autoencoders" (Ayonrinde, 2024) likewise formulates sparse coding as a resource-allocation problem in which sparsity is distributed unevenly across tokens and features. "Toward Identifiable Sparse Autoencoders" (Nelson et al., 29 May 2026) addresses a different axis—stability and identifiability across training runs—but supports a related interpretation of self-organization through stable concept discovery.

2. Core mechanism of the named SOSAE model

The central mechanism in the original SOSAE paper is the self-organization regularization, also called the push loss (Modi et al., 7 Jul 2025). It combines activation magnitude with positional index so that sparsity is both encouraged and spatially organized within the latent vector. The recursive form given in the paper is

$$L_N(H) = \sum_{k=1}^{d_N}(1+\alpha_N)^k \cdot L_{N-1}(H)$$

with base level

$$L_1(H) = \sum_{k=1}^{d_1}(1+\alpha_1)^k \cdot |h_k|.$$

The full objective is written as

$$\mathbf{L(x,g(f(\hat{x})))} = \mathbf{|x - g(f(\hat{x}))| + \sum_{i = 1}^{N} (1 + \alpha)^i \cdot h_i}.$$

The paper also states that $\alpha_k$ follows the proportionality condition

$$\alpha_k \propto \frac{\sum_{n = 1}^k d_n}{\sum_{n = 1}^N d_n}.$$

The two factors have distinct functions. The magnitude term penalizes large activations and therefore induces sparsity. The positional term grows exponentially with coordinate index, making later dimensions progressively more expensive. According to the paper, this pushes zero-activation dimensions to higher-loss regions and toward the end of the feature space. The latent representation thereby acquires an ordered structure in which earlier coordinates are more likely to contain informative content and later coordinates are more likely to be inactive (Modi et al., 7 Jul 2025).

The same paper additionally mentions a contractive regularization term based on the Frobenius norm of the Jacobian of the encoder activations,

$$\| Jf(x) \|^2 = \sum_{j=1}^{d_h} \|h_j (1 - h_j) W_j\|^2,$$

which is described as helping local space contraction, improving robustness, and stabilizing training (Modi et al., 7 Jul 2025).

3. Dynamic adaptation and truncatable representations

The defining operational consequence of SOSAE is that hidden-layer size becomes effectively trainable. During training, useful information concentrates toward one side of the latent vector, while non-active dimensions are pushed to the other side. Because the tail becomes mostly zeros or inactive units, it can be truncated after training without destroying information (Modi et al., 7 Jul 2025).

This directly addresses a limitation of ordinary sparse autoencoders. Standard sparse penalties yield unstructured sparsity, meaning that zero and nonzero coordinates can be interleaved arbitrarily. In such a representation, removing the tail may delete essential coordinates. SOSAE's ordered latent structure is meant to avoid that failure mode by turning the coordinate order itself into a meaningful ranking of usefulness (Modi et al., 7 Jul 2025).

The truncation experiments reported in (Modi et al., 7 Jul 2025) are presented as the central validation of this claim. The authors progressively remove the tail of the latent vector and evaluate downstream accuracy. The reported outcome is that SOSAE is nearly immune to truncation, while K-sparse loses some accuracy, CAE loses more, and $L_1$ and $L_2$0 sparse autoencoders degrade sharply once many tail dimensions are removed. The interpretation offered in the paper is that the exponentially increasing positional penalty makes later dimensions progressively more expensive, so dimensions that do not contribute enough to reconstruction are incentivized to remain inactive and to cluster toward the end (Modi et al., 7 Jul 2025).

This suggests a broader criterion for calling a method SOSAE-like: the sparse support is not merely small, but organized so that effective capacity can be inferred from the representation itself. Under that interpretation, dynamic sparsemax thresholding in (Wang et al., 16 Apr 2026) and adaptive token-feature budget allocation in (Ayonrinde, 2024) are plausible members of the same conceptual family, even though they solve different problems.

4. Architecture, training procedure, and empirical profile of SOSAE

The implementation described in (Modi et al., 7 Jul 2025) is intentionally shallow: a single fully connected encoder layer and a single fully connected decoder layer, trained with Adam on images flattened into vectors. The training objective is reconstruction loss plus push loss, with optional Jacobian-based contractive regularization. The paper emphasizes that the autoencoder is shallow so that the observed gains are attributable to the loss function and self-organization mechanism rather than deep architectural capacity (Modi et al., 7 Jul 2025).

The experiments use four image datasets: MNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet. The baselines include Contractive Autoencoder (CAE), K-sparse autoencoder, Contractive K-sparse, $L_2$1 sparse autoencoder, $L_2$2 sparse autoencoder, and, for dimensionality tuning comparison, Grid Search and Random Search. For baseline autoencoder experiments, fixed hidden sizes are used: 400 for MNIST, 800 for CIFAR-10, 800 for CIFAR-100, and 1024 for Tiny ImageNet (Modi et al., 7 Jul 2025).

The reported metrics are Compressed Length, Classification Accuracy, FLOPs Usage, and Memory Usage. The main results given in the paper are summarized below.

Dataset	CAE	SOSAE
MNIST	compressed length 400, accuracy 96.75%, FLOPs 100%, memory 96 MB	compressed length 94, accuracy 97.58%, FLOPs 23.5%, memory 22.56 MB
CIFAR-10	800, 45.62%, 100%, 192 MB	208, 46.79%, 26%, 49.92 MB
CIFAR-100	800, 18.23%, 100%, 192 MB	256, 24.05%, 32%, 61.44 MB
Tiny ImageNet	1024, 8.26%, 100%, 409.6 MB	239, 8.46%, 23.3%, 95.44 MB

The paper summarizes these results as $L_2$3 to $L_2$4 more compression than the baselines while maintaining or improving classification accuracy (Modi et al., 7 Jul 2025). It also reports a dimensionality-search comparison in which Grid Search uses 784 iterations, tuned value 88, and total 8.7 PFLOPs; Random Search uses 400 iterations, tuned value 92, and total 2.27 PFLOPs; and SOSAE uses 1 training run, tuned value 94, and total 0.066 PFLOPs. From this, the paper claims about $L_2$5 fewer FLOPs than grid search and $L_2$6 fewer FLOPs than random search for reaching an optimal dimensionality (Modi et al., 7 Jul 2025).

The same work includes denoising experiments with zero masking and Gaussian noise. It reports that SOSAE remains robust and often outperforms traditional DAE, with compressed length much smaller than DAE and classification accuracy higher across noise levels; for Gaussian noise the reported improvement is roughly $L_2$7 to $L_2$8, and for zero masking about $L_2$9 to $K$0 (Modi et al., 7 Jul 2025).

5. SOSAE-like adaptive sparsity in later sparse autoencoder research

Although (Modi et al., 7 Jul 2025) defines SOSAE in terms of ordered latent truncation, later work has advanced a different but related idea: sparsity that is self-organized per sample rather than fixed globally.

In (Wang et al., 16 Apr 2026), the proposed model is best understood as a dynamic-attention sparse autoencoder. It retains the SAE objective of learning an overcomplete dictionary of interpretable concepts, but replaces the usual MLP encoder/decoder with a cross-attention block and uses sparsemax instead of dense softmax or fixed-$K$1 selection. In the paper's transformer form,

$K$2

with reconstruction

$K$3

The sparse operator is sparsemax, defined as projection onto the probability simplex:

$K$4

Its closed form is

$K$5

with $K$6 chosen so that active outputs sum to one. The support size

$K$7

is not fixed in advance. The paper explicitly states that simple query features yield a small active set while complex queries containing multiple concepts select more active concepts. This makes sparsity sample-dependent and is the reason the model is described as analogous to a self-organizing sparse autoencoder (Wang et al., 16 Apr 2026).

That paper contrasts sparsemax with several SAE variants. ReLU SAEs often require a sparsity-regularized objective such as

$K$8

typically with $K$9 sparsity, which can cause feature shrinkage. JumpReLU and GatedReLU also need additional tuning or regularization, while TopK and BatchTopK require a hand-chosen $$L_N(H) = \sum_{k=1}^{d_N}(1+\alpha_N)^k \cdot L_{N-1}(H)$$0. Sparsemax replaces those external choices with a sample-dependent threshold (Wang et al., 16 Apr 2026).

The empirical results reported in (Wang et al., 16 Apr 2026) support the adaptive-sparsity interpretation. On text reconstruction, Sparsemax SAE achieves the lowest NMSE in every reported setting on OpenWebText and WikiText-103 across dictionary sizes. On OpenWeb with $$L_N(H) = \sum_{k=1}^{d_N}(1+\alpha_N)^k \cdot L_{N-1}(H)$$1, the paper reports NMSE values $$L_N(H) = \sum_{k=1}^{d_N}(1+\alpha_N)^k \cdot L_{N-1}(H)$$2; on WikiText-103, $$L_N(H) = \sum_{k=1}^{d_N}(1+\alpha_N)^k \cdot L_{N-1}(H)$$3. For interpretability, zero-shot image classification using SAE-reconstructed embeddings and only top-$$L_N(H) = \sum_{k=1}^{d_N}(1+\alpha_N)^k \cdot L_{N-1}(H)$$4 concepts is reported to be best on average across 11 datasets for all $$L_N(H) = \sum_{k=1}^{d_N}(1+\alpha_N)^k \cdot L_{N-1}(H)$$5, with examples including ImageNet $$L_N(H) = \sum_{k=1}^{d_N}(1+\alpha_N)^k \cdot L_{N-1}(H)$$6 and Food101 $$L_N(H) = \sum_{k=1}^{d_N}(1+\alpha_N)^k \cdot L_{N-1}(H)$$7 for top-1, top-5, top-10, and top-50 respectively (Wang et al., 16 Apr 2026).

A different SOSAE-like mechanism appears in (Ayonrinde, 2024), which reframes token-feature matching as a resource-allocation problem under a total sparsity upper bound. The paper proposes Feature Choice SAEs and Mutual Choice SAEs, both of which allow a variable number of active features per token rather than fixing exactly $$L_N(H) = \sum_{k=1}^{d_N}(1+\alpha_N)^k \cdot L_{N-1}(H)$$8 features per token as in TopK SAEs. The mask $$L_N(H) = \sum_{k=1}^{d_N}(1+\alpha_N)^k \cdot L_{N-1}(H)$$9 is selected either by feature-side constraints or by taking the top $$L_1(H) = \sum_{k=1}^{d_1}(1+\alpha_1)^k \cdot |h_k|.$$0 affinities globally. This enables adaptive computation in which easy tokens use fewer features and hard tokens use more (Ayonrinde, 2024).

The paper reports that Feature Choice and Mutual Choice reduce dead features and improve reconstruction loss at equivalent sparsity levels. Specific findings include 0% dead features for 16M and 34M latent Feature Choice runs, more than 10% better loss recovered from Zipf-distributed feature capacities than from uniform capacities, and two-stage training in which Mutual Choice is followed by optional Feature Choice finetuning (Ayonrinde, 2024). The authors explicitly characterize the method as close to a Self-Organizing Sparse AutoEncoder idea because the model self-allocates sparse capacity rather than enforcing a uniform per-token budget.

6. Stability, identifiability, and the organization of concepts across runs

A separate but consequential issue for any self-organizing sparse autoencoder is whether the organized structure is reproducible. "Toward Identifiable Sparse Autoencoders" (Nelson et al., 29 May 2026) argues that standard TopK SAEs are empirically and theoretically unstable: different runs can produce different concept dictionaries and different sparse codes, undermining interpretability and interventions.

The paper formalizes a standard SAE as

$$L_1(H) = \sum_{k=1}^{d_1}(1+\alpha_1)^k \cdot |h_k|.$$1

with encoder typically

$$L_1(H) = \sum_{k=1}^{d_1}(1+\alpha_1)^k \cdot |h_k|.$$2

and studies identifiability in a run-to-run sense. Two independently trained SAEs are nearly identifiable if there exists a signed permutation matrix $$L_1(H) = \sum_{k=1}^{d_1}(1+\alpha_1)^k \cdot |h_k|.$$3 such that

$$L_1(H) = \sum_{k=1}^{d_1}(1+\alpha_1)^k \cdot |h_k|.$$4

and

$$L_1(H) = \sum_{k=1}^{d_1}(1+\alpha_1)^k \cdot |h_k|.$$5

This definition permits only permutation and sign ambiguities (Nelson et al., 29 May 2026).

To address instability, the paper introduces iSAE and iSAE-ME as minimal modifications of standard TopK SAEs. The first key change is AbsTopK, which keeps the $$L_1(H) = \sum_{k=1}^{d_1}(1+\alpha_1)^k \cdot |h_k|.$$6 largest entries in absolute value:

$$L_1(H) = \sum_{k=1}^{d_1}(1+\alpha_1)^k \cdot |h_k|.$$7

This allows signed coefficients and avoids duplicating a concept and its negation as separate atoms. The second change is approximate RIP regularization; the third, in iSAE-ME, is a multi-step LISTA-style encoder,

$$L_1(H) = \sum_{k=1}^{d_1}(1+\alpha_1)^k \cdot |h_k|.$$8

with final output $$L_1(H) = \sum_{k=1}^{d_1}(1+\alpha_1)^k \cdot |h_k|.$$9 and $$\mathbf{L(x,g(f(\hat{x})))} = \mathbf{|x - g(f(\hat{x}))| + \sum_{i = 1}^{N} (1 + \alpha)^i \cdot h_i}.$$0 in experiments (Nelson et al., 29 May 2026).

The paper's significance for SOSAE is conceptual rather than nominal. It does not introduce a model called SOSAE, nor does it focus on latent-width self-tuning. Instead, it argues that self-organized concept learning is meaningful only if the resulting dictionary and sparse codes are stable across runs. It reports empirical metrics including DCS, IoU, normalized code $$\mathbf{L(x,g(f(\hat{x})))} = \mathbf{|x - g(f(\hat{x}))| + \sum_{i = 1}^{N} (1 + \alpha)^i \cdot h_i}.$$1 error, MSE, and an aRIP proxy $$\mathbf{L(x,g(f(\hat{x})))} = \mathbf{|x - g(f(\hat{x}))| + \sum_{i = 1}^{N} (1 + \alpha)^i \cdot h_i}.$$2, and states that iSAE-ME attains the lowest reconstruction error on real LLM activations, while stability gains are often improved though still limited by optimization (Nelson et al., 29 May 2026). This suggests that an encyclopedia treatment of SOSAE should distinguish between adaptive sparsity and identifiable sparsity: a model may self-organize a sparse structure, yet still fail to organize it reproducibly.

7. Interpretation, limitations, and scope of the term

The name SOSAE has a precise and a loose usage. In the precise usage, it refers to the shallow autoencoder with push loss introduced in (Modi et al., 7 Jul 2025). Its central contribution is replacing the conventional two-step workflow—first choose bottleneck size, then train the autoencoder—with a single end-to-end training procedure in which latent dimensionality emerges from a position-weighted sparsity regularizer. The strongest evidence in that paper concerns image datasets, shallow fully connected architectures, truncation robustness, memory savings, FLOPs savings, and noise robustness (Modi et al., 7 Jul 2025).

In the looser usage, SOSAE functions as an umbrella label for sparse autoencoders that let sparse support emerge adaptively from data rather than from a globally fixed $$\mathbf{L(x,g(f(\hat{x})))} = \mathbf{|x - g(f(\hat{x}))| + \sum_{i = 1}^{N} (1 + \alpha)^i \cdot h_i}.$$3, fixed latent width, or explicit sparsity penalty. On that reading, the sparsemax-based cross-attention SAE of (Wang et al., 16 Apr 2026) is a strong precursor or near-instantiation because it infers the active subset $$\mathbf{L(x,g(f(\hat{x})))} = \mathbf{|x - g(f(\hat{x}))| + \sum_{i = 1}^{N} (1 + \alpha)^i \cdot h_i}.$$4 through projection geometry. The Feature Choice and Mutual Choice SAEs of (Ayonrinde, 2024) are also close conceptual variants because they allocate sparse budget unevenly across tokens and features. The identifiable SAE line (Nelson et al., 29 May 2026) is less directly about adaptive sparsity, but is relevant to whether self-organized concepts remain stable enough to support mechanistic interpretation.

Several limitations follow directly from the cited papers. The original SOSAE paper does not present a deep theoretical convergence analysis and evaluates primarily shallow fully connected autoencoders on image datasets (Modi et al., 7 Jul 2025). The adaptive allocation paper notes that its Monotonic Importance Heuristic is only an approximation and that generalization beyond language is unknown (Ayonrinde, 2024). The dynamic-attention work reports occasional unclear or redundant concepts, attributed to feature absorption or fine-grained concept overlap (Wang et al., 16 Apr 2026). The identifiability paper shows that good reconstruction alone does not guarantee unique sparse codes and that optimization remains a limitation even when architecture and regularization are improved (Nelson et al., 29 May 2026).

Taken together, these works locate SOSAE at the intersection of three research threads: structured latent truncation, adaptive sparse support selection, and stable dictionary learning. The named SOSAE model (Modi et al., 7 Jul 2025) is the clearest statement of self-organization as automatic bottleneck discovery. Subsequent SAE variants extend the same underlying intuition: sparse representations are most useful when their support is not only small, but also data-adaptive, semantically coherent, and, ideally, identifiable across runs (Wang et al., 16 Apr 2026, Ayonrinde, 2024, Nelson et al., 29 May 2026).