SGATLIN: Sparsely Gated Linear Neurons

• SGATLIN is a transformer feedforward design that replaces dense MLPs with a sparse, token-dependent router selecting a minimal subset of rank-1 linear experts.
• It leverages conditional linearity, where top-k routing provides the sole nonlinearity, simplifying expert computation and enhancing model interpretability.
• Empirical results show SGATLIN matches or outperforms traditional dense and MoE models under compute constraints, offering scalable sparsity with clear circuit tracing.

Sparsely gated linear neurons, introduced under the name “sgatlin”, are a transformer feedforward-layer design in which a token-dependent sparse router selects a very small top-$k$ subset from a very large bank of single-neuron linear experts (Schug, 5 Jun 2026). The defining departure from dense MLPs, SwiGLU blocks, and coarse-grained MoE layers is that the experts are reduced to rank-1 linear operators and the usual expert nonlinearity is removed; the principal nonlinearity is instead supplied by the top-$k$ routing itself. Once routing is fixed for a token, the active feedforward computation is a low-rank linear circuit, which is central both to the paper’s compute-efficiency claims and to its interpretability program (Schug, 5 Jun 2026).

1. Definition and formal construction

In its basic form, a SGATLIN layer takes an input residual-stream vector $z \in \mathbb{R}^{d_{\text{model}}}$, computes sparse gating weights $g$, and then aggregates the selected linear neurons. The routing rule is

$$g = \operatorname{product\text{-}top\text{-}k}(W_{\text{key}} W_{\text{query}} z),$$

with $$W_{\text{query}} \in \mathbb{R}^{d_{\text{key}} \times d_{\text{model}}}$$ and $$W_{\text{key}} \in \mathbb{R}^{2\sqrt{d_{\text{ffw}}} \times d_{\text{key}}}$$. The operator $\operatorname{product\text{-}top\text{-}k}$ maps a vector in $\mathbb{R}^{2\sqrt{d_{\text{ffw}}}}$ to a sparse vector in $\mathbb{R}^{d_{\text{ffw}}}$ with exactly $k$0 nonzero entries. The feedforward computation is then

$k$1

where each neuron $k$2 is parameterized by $k$3 and $k$4 (Schug, 5 Jun 2026).

The full layer uses $k$5 parallel channels. For channel $k$6,

$k$7

and the output is

$k$8

A key architectural choice is that $k$9 is shared across channels, while the keys and neuron parameters are channel-specific. This makes each selected neuron a single linear expert and the whole active circuit a sum of rank-1 contributions (Schug, 5 Jun 2026).

2. Conditional linearity and the source of nonlinearity

Each expert defines a rank-1 matrix

$z \in \mathbb{R}^{d_{\text{model}}}$0

so for a fixed routing pattern the token-specific feedforward circuit is

$z \in \mathbb{R}^{d_{\text{model}}}$1

and the layer acts as

$z \in \mathbb{R}^{d_{\text{model}}}$2

Since only $z \in \mathbb{R}^{d_{\text{model}}}$3 neurons are selected, the resulting matrix is at most rank-$z \in \mathbb{R}^{d_{\text{model}}}$4 in the single-channel exposition. This is the exact sense in which SGATLIN is linear only conditionally: once the gate is fixed, the active subnetwork is linear; globally, the layer is nonlinear because the selected expert set changes with the input (Schug, 5 Jun 2026).

This places SGATLIN in a distinct position relative to standard transformer FFNs. A conventional dense feedforward block is typically written as

$z \in \mathbb{R}^{d_{\text{model}}}$5

or in SwiGLU form,

$z \in \mathbb{R}^{d_{\text{model}}}$6

Those architectures activate all hidden units for every token and place the nonlinearity inside the hidden layer. SGATLIN instead uses a token-dependent sparse mask $z \in \mathbb{R}^{d_{\text{model}}}$7 and no expert activation function. The paper states this directly: apart from the top-$z \in \mathbb{R}^{d_{\text{model}}}$8 selection, it applies no nonlinearities, neither to the gating weights nor to the neurons. A common misconception is therefore to equate “linear experts” with a globally linear model; in SGATLIN, the top-$z \in \mathbb{R}^{d_{\text{model}}}$9 router itself is the nonlinearity (Schug, 5 Jun 2026).

The same comparison clarifies the relation to sparse MoE. A generic MoE layer has the form

$g$0

with each $g$1 usually a nonlinear MLP. SGATLIN is the special case in which the experts are single-neuron linear maps,

$g$2

and $g$3, with exactly $g$4 active experts. In the paper’s terminology, it is the endpoint of shrinking MoE granularity down to individual neurons and then removing the usual expert nonlinearity (Schug, 5 Jun 2026).

3. Position within the gated-linear literature

The explicit acronym SGATLIN belongs to “Sparsely gated tiny linear experts” (Schug, 5 Jun 2026), but the design sits within a broader lineage of gated-linear models and sparse gating mechanisms.

Model	Core mechanism	Relation to SGATLIN
GaLU	Binary gate and linear response use different vectors	Decouples gating from linearity (Fiat et al., 2019)
GLN	Context-selected logit-linear weights	Conditional linearity with local convex learning (Veness et al., 2019)
GGDLN	Globally shared input-dependent gates over deep linear branches	Finite-width theory for gated linear networks (Li et al., 2022)
MGLU	Shared weight matrix with binary gate/value masks	Efficient gated linear units in LLM FFNs (Tajima et al., 29 Jun 2025)
Linear inhibition	Subtractive learned suppression after an encoder	Soft sparse gating by lateral inhibition (Le, 2015)
Gate-preactivation sparsification	Some recurrent gates become constant	Structured simplification of gated computation (Lobacheva et al., 2019)

GaLU is perhaps the cleanest earlier formal precursor at the neuron level. It rewrites a ReLU neuron as a product of a linear term and a binary gate, then explicitly decouples those roles by defining

$g$5

with standard ReLU recovered as $g$6. The payoff in that paper is analytic tractability: once gates are fixed, shallow GaLU networks reduce to linear regression over a random nonlinear feature map, yielding memorization guarantees of width $g$7 rather than the $g$8 guarantee cited for comparable ReLU analyses (Fiat et al., 2019). This suggests a direct conceptual bridge to SGATLIN: explicit separation between sparse selection and linear response can materially change both theory and scaling.

GLNs and GGDLNs generalize the same principle in different directions. GLNs use context functions to select one weight row per neuron, making each neuron conditionally linear in logit space and trained with local online convex optimization rather than backpropagation (Veness et al., 2019). GGDLNs instead use gates shared across all units in a layer, keeping the processing motifs linear while the input-dependent gates provide nonlinearity; their central theoretical result is that finite-width learning induces kernel shape renormalization, and with enough gating units the resulting networks behave similarly to standard ReLU networks (Li et al., 2022). Relative to SGATLIN, these works reinforce the idea that strong expressivity can arise from conditional linearity plus structured gating, without requiring dense pointwise nonlinear experts.

More recent LLM-oriented work explores a different axis of the same design space. MGLU replaces the usual pair of gate and value matrices in a GLU with a single shared matrix partitioned by learned binary masks, achieving a reported 47% memory-efficiency gain and 34% speed gain over standard GLUs on an RTX5090 GPU while matching or surpassing a SwiGLU baseline in downstream accuracy (Tajima et al., 29 Jun 2025). That is not sparse token routing in the SGATLIN sense, but it is a concrete example of how gated linear computation can be reparameterized around sparsity and hardware constraints.

4. Training setup, scaling law, and computational profile

The SGATLIN paper evaluates decoder-only causal transformers on SlimPajama 627B with a GPT-2 BPE tokenizer of vocabulary size 50,257, context length 2048, and global batch size 128 sequences, for $g$9 tokens per batch. Training uses bfloat16, AdamW, peak learning rate $$g = \operatorname{product\text{-}top\text{-}k}(W_{\text{key}} W_{\text{query}} z),$$0, weight decay $$g = \operatorname{product\text{-}top\text{-}k}(W_{\text{key}} W_{\text{query}} z),$$1 excluding scalar parameters, and gradient clipping with max global norm $$g = \operatorname{product\text{-}top\text{-}k}(W_{\text{key}} W_{\text{query}} z),$$2. The learning-rate schedule is warmup-stable-square-root-decay, with 1000 warmup steps and final square-root decay over the last 20% of training tokens. Hyperparameters were selected by a grid search on a 12-layer dense SwiGLU transformer at $$g = \operatorname{product\text{-}top\text{-}k}(W_{\text{key}} W_{\text{query}} z),$$3 FLOPs over learning rates $$g = \operatorname{product\text{-}top\text{-}k}(W_{\text{key}} W_{\text{query}} z),$$4, weight decays $$g = \operatorname{product\text{-}top\text{-}k}(W_{\text{key}} W_{\text{query}} z),$$5, and warmup steps $$g = \operatorname{product\text{-}top\text{-}k}(W_{\text{key}} W_{\text{query}} z),$$6 (Schug, 5 Jun 2026).

Model scaling is defined through an integer scale $$g = \operatorname{product\text{-}top\text{-}k}(W_{\text{key}} W_{\text{query}} z),$$7 with

$$g = \operatorname{product\text{-}top\text{-}k}(W_{\text{key}} W_{\text{query}} z),$$8

For SGATLIN specifically,

$$g = \operatorname{product\text{-}top\text{-}k}(W_{\text{key}} W_{\text{query}} z),$$9

The main routing hyperparameters are fixed at $$W_{\text{query}} \in \mathbb{R}^{d_{\text{key}} \times d_{\text{model}}}$$0 channels, top-$$W_{\text{query}} \in \mathbb{R}^{d_{\text{key}} \times d_{\text{model}}}$$1, and $$W_{\text{query}} \in \mathbb{R}^{d_{\text{key}} \times d_{\text{model}}}$$2. The paper emphasizes that in the largest models less than 0.1% of feedforward parameters are active per token. This is the central operational notion of “sparse gating” in SGATLIN: a huge feedforward parameter pool with a very small active subset per token (Schug, 5 Jun 2026).

The computational argument hinges on the product-key router. Because $$W_{\text{query}} \in \mathbb{R}^{d_{\text{key}} \times d_{\text{model}}}$$3 is kept small and constant, the cost of increasing the number of available neurons scales sublinearly, as $$W_{\text{query}} \in \mathbb{R}^{d_{\text{key}} \times d_{\text{model}}}$$4, rather than linearly in the number of experts. The appendix gives the asymptotic SGATLIN time complexity as

$$W_{\text{query}} \in \mathbb{R}^{d_{\text{key}} \times d_{\text{model}}}$$5

and memory complexity as

$$W_{\text{query}} \in \mathbb{R}^{d_{\text{key}} \times d_{\text{model}}}$$6

Unlike Switch-style MoE formulations, the reported SGATLIN formulation does not introduce a separate load-balancing auxiliary loss, dispatch/combine matrices, or explicit capacity constraints. A plausible implication is that the architecture trades off some of the familiar MoE balancing machinery for a much finer-grained routing problem over single-neuron experts (Schug, 5 Jun 2026).

5. Empirical performance and interpretability

The main empirical claim is an isoflop one: replacing all transformer FFNs with SGATLIN improves or matches test perplexity across compute budgets $$W_{\text{query}} \in \mathbb{R}^{d_{\text{key}} \times d_{\text{model}}}$$7 FLOPs, while also increasing compute sparsity relative to dense FFNs and coarse MoE baselines (Schug, 5 Jun 2026). The comparison is explicitly compute-matched rather than parameter-matched, which is important because sparse architectures can expose far more total parameters for the same active FLOPs.

The most concrete quantitative evidence in the excerpt is the ablation at $$W_{\text{query}} \in \mathbb{R}^{d_{\text{key}} \times d_{\text{model}}}$$8 FLOPs with 1.2B-parameter models. The reported test perplexities are: SGATLIN $$W_{\text{query}} \in \mathbb{R}^{d_{\text{key}} \times d_{\text{model}}}$$9; $$W_{\text{key}} \in \mathbb{R}^{2\sqrt{d_{\text{ffw}}} \times d_{\text{key}}}$$0 ReLU activation $$W_{\text{key}} \in \mathbb{R}^{2\sqrt{d_{\text{ffw}}} \times d_{\text{key}}}$$1; $$W_{\text{key}} \in \mathbb{R}^{2\sqrt{d_{\text{ffw}}} \times d_{\text{key}}}$$2 GeLU activation $$W_{\text{key}} \in \mathbb{R}^{2\sqrt{d_{\text{ffw}}} \times d_{\text{key}}}$$3; $$W_{\text{key}} \in \mathbb{R}^{2\sqrt{d_{\text{ffw}}} \times d_{\text{key}}}$$4 Swish activation $$W_{\text{key}} \in \mathbb{R}^{2\sqrt{d_{\text{ffw}}} \times d_{\text{key}}}$$5; and a router swap to a PEER-style router $$W_{\text{key}} \in \mathbb{R}^{2\sqrt{d_{\text{ffw}}} \times d_{\text{key}}}$$6 (Schug, 5 Jun 2026). These numbers are central because they isolate the paper’s strongest claim: once experts are shrunk to single neurons, removing the expert nonlinearity improves rather than hurts language-modeling performance. The authors summarize the result as evidence that top-$$W_{\text{key}} \in \mathbb{R}^{2\sqrt{d_{\text{ffw}}} \times d_{\text{key}}}$$7 gating by itself suffices to enable nonlinear processing in SGATLIN.

The interpretability program depends directly on conditional linearity. Because the gating weights uniquely identify the active feedforward circuit, the paper constructs a database of circuits by collecting gating vectors across many contexts and treating them as points in a cosine-distance space. In a small TinyStories study using a 4-layer model with $$W_{\text{key}} \in \mathbb{R}^{2\sqrt{d_{\text{ffw}}} \times d_{\text{key}}}$$8, 4 attention heads, context length 1024, and vocabulary size 8192, the authors project gating vectors with UMAP and report semantically structured clusters: punctuation tokens lie on the outer perimeter, different forms of the verb “to be” cluster together, names cluster together, and pronouns lie near names. In later layers, pronouns and names are reported to be processed with similar circuits, suggesting clustering by semantic role rather than exact token identity (Schug, 5 Jun 2026).

The paper also probes causal involvement in factual recall. From TinyStories it extracts noun-adjective pairs of the form “The [noun] was [adjective]”, builds clean and counterfactual prompts, and patches in counterfactual gating weights at specific layers and token positions. Using a normalized indirect effect based on the difference between target and counterfactual logits, it finds moderate causal effects, strongest at the noun position, strongest in the second layer, and largest when patching all layers. The authors interpret this as evidence that feedforward circuits are causally implicated in factual retrieval, while also noting that the effects are not overwhelming and may coexist with knowledge stored elsewhere, such as embeddings (Schug, 5 Jun 2026).

6. Limitations, scope, and conceptual significance

Several caveats in the paper constrain how SGATLIN should be interpreted. First, the efficiency claims are in FLOPs, not guaranteed wall-clock speed. Sparse routing can be hardware-unfriendly, and current accelerators are optimized for dense matrix multiplications. Second, only the feedforward layers are sparsified; attention remains dense. Third, the interpretability evidence is small-scale and confined to TinyStories, so it does not by itself establish that the same circuit clarity persists in substantially larger LLMs. Fourth, the evaluation focuses on autoregressive perplexity rather than downstream instruction tuning, long-context behavior, or multimodal settings (Schug, 5 Jun 2026).

There is also a narrower architectural caution. SGATLIN’s “sparse gating” is not the same thing as conventional expert load balancing or unit sparsification in every other literature. The active set is sparse per token, but the mechanism is a product-key top-$$W_{\text{key}} \in \mathbb{R}^{2\sqrt{d_{\text{ffw}}} \times d_{\text{key}}}$$9 router over a bank of rank-1 linear experts, not a standard top-$\operatorname{product\text{-}top\text{-}k}$0 router over large nonlinear experts, not a static pruning mask, and not merely a GLU-style gate/value factorization. This distinction matters when comparing SGATLIN to other gated-linear systems. Inhibitory sparse coding methods, for example, implement soft subtractive competition after an encoder rather than sparse expert routing (Le, 2015); recurrent gate-sparsification methods simplify the gate-generating computation itself by making some gates constant (Lobacheva et al., 2019); and MGLU uses learned binary masks to repartition gate and value roles within a shared projection matrix rather than selecting a token-specific subset of neurons (Tajima et al., 29 Jun 2025).

What makes SGATLIN notable within this broader landscape is the combination of three properties: extreme sparsity of active feedforward computation, linearity of individual experts, and a per-token circuit that is explicitly representable as a low-rank linear operator. This suggests a specific endpoint of a longer design trajectory: from dense FFNs, to sparse routing over large nonlinear experts, to single-neuron experts, and finally to linear single-neuron experts whose nonlinearity is delegated entirely to routing (Schug, 5 Jun 2026). Earlier theory on decoupled gating and gated linear networks suggests why this endpoint is plausible: once gating and linear response are separated, both analysis and specialization can improve materially (Fiat et al., 2019, Li et al., 2022).