LightningLM 0.1V: Reversible, Scalable MoE

• LightningLM 0.1V is a four-stage language model family defined by a recurrence-first backbone and progressive growth from a dense 2B seed to a 120B sparse MoE.
• The model preserves learned interfaces across scale transitions via state-preserving growth, enabling controlled expansion without reinitializing optimization state.
• Techniques like reversibility and TQP with TurboQuant reduce memory footprint and optimizer state, making single-node training feasible even at the 120B scale.

Searching arXiv for the LightningLM 0.1V paper and closely related methods to ground the article. {"query":"LightningLM 0.1V arXiv (Shravan, 5 Jun 2026) reversible foundations training a 120B sparse MoE through state-preserving scaling", "max_results": 5} {"query":"Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling", "max_results": 10} LightningLM 0.1V is a four-stage language-model family built around a single recurrence-first backbone and trained by progressive expansion from a dense seed to a 120B-parameter-class sparse mixture-of-experts (MoE) model on a single eight-GPU node (Shravan, 5 Jun 2026). In the released lineage, a dense 2B seed is grown into a 5B MoE, then a 9B MoE, and finally a 120B sparse MoE with 460 routed experts. The paper presents the family primarily as a systems-and-experience report rather than a benchmark study, and organizes its contribution around three coupled disciplines: reversibility, state-preserving growth, and single-node economics. Across the lineage, active parameters increase monotonically from 1.78B to 5.93B, while the largest model stores 118.67B parameters but activates only about 5% of them per token.

1. Model lineage and scaling trajectory

The family is defined as one architecture grown repeatedly rather than a set of unrelated checkpoints (Shravan, 5 Jun 2026). The released stages are a dense 2B seed, a 5B MoE, a 9B MoE, and a 120B sparse MoE. Their active parameter counts are 1.78B, 2.24B, 3.92B, and 5.93B, respectively. At the largest stage, the model stores 118.67B parameters, uses 460 routed experts under top-12 routing, and operates at 8K context; the early stages use 4K context, and the 9B stage onward uses 8K.

This staged growth is central to the paper’s conception of scaling. The larger model is not trained from scratch after each transition; instead, each expansion preserves previously learned interfaces and then adds capacity in a controlled way. The paper frames this as “state-preserving growth,” with the practical aim of avoiding resets in optimization state and avoiding destructive changes to already trained functions. A plausible implication is that the family treats scale transitions as continuation problems rather than re-initialization problems.

The paper also emphasizes the distinction between stored and active parameters. For the 120B stage, only 5.93B parameters are active per token despite 118.67B stored parameters. This active-vs-stored separation is one of the numerical preconditions for the single-node training claim.

2. Recurrence-first architectural backbone

LightningLM is built on a recurrence backbone rather than a standard full-attention transformer (Shravan, 5 Jun 2026). The shared residual width is 4096, and the repeating layer motif is written as

$\text{DDDGDDDG}$

meaning two repeats of a four-layer pattern with three DeltaNet-style layers followed by one sparse-attention layer. The 2B model has 8 layers, while the 9B and 120B models deepen the same motif to 20 layers.

The token-level recurrent component is a gated DeltaNet update,

$$S_t = \alpha_t \left( S_{t-1} - \beta_t \, S_{t-1} k_t k_t^\top \right) + \beta_t \, v_t k_t^\top, \qquad y_t = S_t q_t.$$

Here, $S_t$ is a matrix-valued state and $q_t, k_t, v_t$ are token projections with content-dependent gates $\alpha_t$ and $\beta_t$. The stated purpose is linear-in-sequence-length computation rather than quadratic attention cost.

At the layer level, the model uses Manifold-Constrained Hyper-Connections (mHC). The residual stream is expanded into four parallel streams, mixed by a content-dependent $4 \times 4$ matrix, normalized with Sinkhorn-Knopp to make it doubly stochastic, and then collapsed back to the main width. The paper characterizes this as conservative and stable stream mixing as depth grows.

At the chunk level, the architecture carries a Memory Stream, one 4096-dimensional summary per sequence, into the next chunk through a gated injection:

$$x_{\text{inject}} = x + \lambda_r \cdot g_t \cdot \mathrm{LN}(m_{\text{prev}}),$$

with outgoing memory set from the last token of the chunk,

$$m_{\text{out}} = \mathrm{stop\_grad}\!\left(h_{\text{main}[:, T-1, :]}\right).$$

This preserves $O(1)$ inter-chunk state because backpropagation does not traverse chunk boundaries. The paper treats these three recurrences—token-level, layer-level, and chunk-level—as the architectural identity of the family.

3. Reversibility as the training enabler

The main systems device is a reversible stack that reconstructs activations in the backward pass instead of storing them during the forward pass (Shravan, 5 Jun 2026). The reversible midpoint update is given schematically as

$$S_t = \alpha_t \left( S_{t-1} - \beta_t \, S_{t-1} k_t k_t^\top \right) + \beta_t \, v_t k_t^\top, \qquad y_t = S_t q_t.$$0

The claim is explicitly not that reversibility improves loss. Its function is to make the memory regime feasible as depth and sequence length increase.

The operational constraints are strict. Dropout must be zero because recomputation must be deterministic. The recompute path also had to be sharding-aware under ZeRO-style partitioning, and the paper notes that allocator regressions and kernel dtype assertions had to be fixed to keep the reversible path stable.

The most concrete feasibility comparison is at 2B scale. On 8×A100-40GB, the non-reversible 2B model at sequence length 4096 did not fit above batch 8, whereas the reversible version fit batch 32 with 25–29GB free per GPU. On 8×A100-80GB, the non-reversible 2B also fit batch 32. The paper therefore presents reversibility as a selective systems tool: it is used when memory is binding and omitted where it is not.

This design choice underwrites the broader claim that activation memory can be held flat as the family grows. In the paper’s framing, reversibility is the first of the three disciplines that make the 120B stage tractable on a single node.

4. State-preserving growth from dense seed to 120B MoE

The 2B seed is dense and fully active, uses the 8-layer backbone, the 4096 residual width, the 131,072-token BrahmicTokenizer-131K vocabulary, and a Kronecker embedding construction rather than a standard lookup table (Shravan, 5 Jun 2026). The paper states that a standard embedding table at this vocabulary and width would be about 537M trainable parameters, whereas the Kronecker projection is about 33.55M trainable parameters. Token vectors are reconstructed from frozen byte and position bases through a trained projection.

The 2B-to-5B transition is a dense-to-MoE upcycling step. The dense feed-forward block is copied into an always-active shared expert, and 20 routed experts are initialized from overlapping partitions of the dense feed-forward neurons. The router is initialized from scratch because the dense model has no routing pattern to inherit. The 5B model remains 8 layers deep, uses top-2 routing, stores 4.96B parameters, and activates 2.24B.

The 5B-to-9B transition is a depth increase from 8 to 20 layers using “depth-mapped re-entry.” The selected mapping is

$$S_t = \alpha_t \left( S_{t-1} - \beta_t \, S_{t-1} k_t k_t^\top \right) + \beta_t \, v_t k_t^\top, \qquad y_t = S_t q_t.$$1

which preserves the interface between the final layer and the language-model head by ending on the original terminal layer. The paper reports that an alternative mapping ending on an interior layer was rejected even if it showed slightly better short-run loss, because it violated the learned terminal-layer-to-head contract. The 9B model keeps 20 routed experts per layer, uses top-2 routing, and stores 9.36B parameters with 3.92B active per token.

The 9B-to-120B transition expands expert count rather than sequential depth. Routed experts increase from 20 to 460, depth stays at 20 layers, and routing changes to top-12. The expansion uses drop-upcycling: each source expert is cloned into 23 variants, a fraction of weights is reinitialized to break symmetry, and the router is tiled and perturbed. The paper stresses that cloning alone is insufficient; if the router cannot separate near-duplicate experts early, many experts die before differentiation. Probabilistic selection is therefore used during the early differentiation window.

These transitions are presented not merely as scaling heuristics but as reproducible principles paired with specific failure modes, including silent ones.

5. TQP, quantized experts, and single-node economics

The largest stage depends on TQP, or TQ-PreTraining, which combines quantized base expert weights with trained low-rank adapters (Shravan, 5 Jun 2026). The stated bottleneck is optimizer state rather than parameter storage alone. Full Adam moments over the more than 100B routed expert parameters would be on the order of 800 GB. TQP reduces trainable expert-path parameters to about 2.26B and cuts optimizer state to about 18 GB, a reduction of roughly 45×.

The adapters are rank 16 for each of the three weight packs per expert across all 460 experts in each of the 20 layers. The base expert weights remain in an 8-bit quantized representation using TurboQuant as the underlying quantizer. The paper describes TurboQuant’s MSE path as separating norm from direction, applying a random orthogonal rotation, quantizing each rotated coordinate independently, and reconstructing via the codebook and stored norm. Its novelty claim is not a new primitive in isolation, but the use of TurboQuant as a trainable base representation inside large-scale pretraining.

The paper also reports an important negative result. ReLoRA-style merge-and-reset training worked in smaller-scale experiments but diverged at 120B. The production run is therefore flushless: adapters accumulate a single delta over a long span rather than periodic merges. This is presented as evidence that a technique that works below 1B does not necessarily transfer to 120B.

The single-node setup remains constant at the level of one eight-GPU node, even though later stages move to H200/B200/B300-class hardware. The production 120B stage runs at about 17k–18k ordinary tokens/s. Additional systems measures include on-the-fly reconstruction of the Kronecker embeddings, saving about 2.15 GB per GPU, custom kernels for linear attention and MoE computation, fused cross-entropy, and checkpoint/upload handling tuned to protect wall-clock throughput.

6. Data strategy, optimization trajectory, and capability profile

The paper uses loss rather than benchmark leaderboards as its primary evidence (Shravan, 5 Jun 2026). Across the lineage, training loss descends continuously from above 6 nats/token at the start of the 2B seed to 1.78 at the released 120B checkpoint. Reported harvest-point losses are about 3.29 for 2B, 2.24 for 5B, 2.05 for 9B, and 1.78 for 120B. For the 120B lineage specifically, the pretraining run goes to about step 3009, supervised training resumes from step 2909 because step 3009 was corrupted, the released checkpoint is step 5161, its instantaneous training loss is 1.790, its trailing-100-step mean is 1.780, and the run minimum is 1.520 at step 5059.

Held-out evaluation is reported by domain. Code is the strongest domain at every scale and improves steadily; STEM also improves steadily; Web remains the hardest domain; Indic scripts improve across the family. Example held-out losses are 2.07 $$S_t = \alpha_t \left( S_{t-1} - \beta_t \, S_{t-1} k_t k_t^\top \right) + \beta_t \, v_t k_t^\top, \qquad y_t = S_t q_t.$$2 1.72 $$S_t = \alpha_t \left( S_{t-1} - \beta_t \, S_{t-1} k_t k_t^\top \right) + \beta_t \, v_t k_t^\top, \qquad y_t = S_t q_t.$$3 1.54 for Code and 2.35 $$S_t = \alpha_t \left( S_{t-1} - \beta_t \, S_{t-1} k_t k_t^\top \right) + \beta_t \, v_t k_t^\top, \qquad y_t = S_t q_t.$$4 2.00 $$S_t = \alpha_t \left( S_{t-1} - \beta_t \, S_{t-1} k_t k_t^\top \right) + \beta_t \, v_t k_t^\top, \qquad y_t = S_t q_t.$$5 1.68 for STEM from 2B to 5B to 9B. The paper treats the Indic result as evidence that targeted capability protection succeeded.

That protection is built into the data pipeline. The corpus is divided into OPUS-eligible bulk data, Always-ON data, and a Golden proxy. The Always-ON tier injects a fixed floor of code, reasoning, and Indic data into every batch, bypassing OPUS selection. The paper states that this is why the model learns those capabilities “by construction rather than by chance.” A dead-token audit then revealed that many vocabulary tokens never appeared in the raw corpus, including native Indic numerals and certain code-formatting tokens. The fixes were corpus-side: swapping Arabic digits for native Indic digits in Indic shards, and adding code data with tabs and CRLF line endings.

7. Significance and limits

LightningLM 0.1V is presented as a proof-of-principle that a 120B-class sparse MoE can be trained end to end on a single eight-GPU node if reversibility, state-preserving growth, and quantized-expert economics are integrated into one lineage (Shravan, 5 Jun 2026). The paper is explicit that none of the underlying primitives is new in isolation. Its novelty lies in the integration of known components into a continuous growth process, with practitioner-level documentation of failure modes.

Several of those failure modes are structurally important. The paper highlights silent checkpoint-loading failures in which some target parameters remained random despite a successful load, and clone-expansion failures in which the router could not distinguish near-duplicate experts, causing expert death before specialization. The emphasis on such failures distinguishes the work from a purely architectural proposal: it is equally about preserving learned interfaces across growth events and preserving training viability at each transition.

Within that framing, LightningLM 0.1V occupies an unusual position in the large-language-model literature. It is neither a claim that recurrence alone supersedes transformer scaling nor a claim that any one memory-saving trick is sufficient. Instead, it argues for a coupled recipe: reversible recurrence to control activation memory, controlled upcycling to preserve function across scale transitions, and TQP to compress optimizer-state requirements in the expert path. The resulting family is released together with tokenizer and training code, making the lineage not only a model family but also a documented scaling protocol.