Reversible Foundations: Training a 120B Sparse MoE through State-Preserving Scaling

Abstract: This paper reports on training a hundred-billion-parameter sparse mixture of experts on a single eight-GPU node, end to end. LightningLM 0.1V is a recurrence-backbone LLM family grown in four stages from a small dense seed, through a 5B and a 9B mixture of experts, to a 120B model with 460 routed experts under top-12 routing. Each larger model is grown from the trained weights of the smaller one; active parameters rise monotonically from 1.78B at the dense seed to 5.93B at 120B (about 5% of the 118.67B stored). The full lineage runs on single nodes, the larger stages at 8K context, reaching a released training loss of 1.78 at 120B scale. This is a systems and experience report. It is organized around three disciplines. Reversibility: a reversible recurrence stack reconstructs activations in the backward pass instead of storing them, holding activation memory flat as the model grows. State-preserving growth: each expansion (dense to MoE, shallow to deep, few experts to many) is given as a reproducible principle paired with the failure that results from getting it wrong; several failures are silent. Single-node economics: the 120B trains through TQP, a strategy of quantized base expert weights and trained low-rank adapters that carries optimizer state on 2.26B adapter parameters rather than 100B+ resident in routed experts, cutting expert-path optimizer state by a factor of ~45. What is new is the integration of known primitives, not any primitive in isolation: one grown lineage running end to end on a single node, documented at practitioner level, with per-domain held-out loss as evidence that targeted capabilities (multilingual Indic competence, code) were learned by construction. Model family, tokenizer, and training code are released.

• The paper introduces a state-preserving method that grows a 120B sparse MoE from a 2B seed using reversible transformations.
• It employs memory-efficient reversible backbones and TQP quantization to ensure resource-friendly scaling on a single eight-GPU node.
• The methodology highlights reproducibility and failure mode analysis, ensuring operational transparency for large-scale language models.

Training a 120B Sparse MoE on a Single Node: The LightningLM State-Preserving Lineage

Introduction and Motivation

This work introduces LightningLM 0.1V, a 120B parameter class sparse Mixture-of-Experts (MoE) LLM, grown in four successive stages from a dense 2B seed to a 120B MoE, utilizing a recurrence-backbone architecture. The salient claim is the end-to-end training of this 120B MoE on a single eight-GPU node, achieved through reversibility, state-preserving architectural growth, and a quantized/adapted training regime (TQP). The paper makes explicit both the operational "recipe" for model growth, often omitted in the literature, and the failure modes encountered when violating key scaling invariants.

Architectural and Systems Innovations

LightningLM is architected around three scaling enablers:

1. Reversible Backbone for Memory-Efficient Deep Stacking

The model relies on a reversible transformer backbone, specifically a reversible midpoint stack. This allows activation memory to be held flat as model depth and number of experts grow, making it possible to scale up context length and parameter count without incurring infeasible activation memory demands. This enables all growth stages from 5B onwards to fit on a single node.

2. State-Preserving Growth Procedures

Model growth is implemented as a deterministic, interface-preserving sequence of transformations:

• Dense → MoE: The feed-forward network is copied into a shared expert, with new routed experts seeded by partitioning the dense FFN.
• Shallow → Deep: Depth increases are implemented by block repetition, always ending on the terminal (originally head-facing) layer.
• Expert Expansion: Each routed expert is expanded into many clones with drop-upcycling for symmetry breaking.
• Positional Encoding: DroPE-style removal and recalibration of rotary encodings is performed before the final learning rate anneal, preserving post-recalibration performance and supporting context scaling.

3. Training Economics via TQP

To train a 120B-scale MoE with >100B stored parameters under tight memory budgets:

• The expert weights are quantized to 8 bits using a data-oblivious analytic vector quantizer (TurboQuant), selected for trainability under a moving weight basis.
• Only small, low-rank adapters (rank-16) are trained in FP16/bfloat16, holding optimizer state on just 2.26B parameters instead of >100B.
• The periodic LoRA/adapter "flush" (adapter-into-base merge) method, effective at sub-1B scale, is shown to diverge at 120B, motivating the use of flushless, continuous adaptation.

  Figure 1: Effective rank at 90% spectral energy by weight class at 9B; routed experts (TQP's adaptation target) are not low-rank, inverting usual LoRA justification.

Data Pipeline and Curriculum

A three-tier data system enforces per-batch guarantees, circumventing proxy bias that would otherwise starve code and Indic-text exposure:

• Bulk data pools are OPUS-selected for efficiency.
• Always-ON data (code, Indic) is injected at a fixed rate into every batch, guaranteeing learned coverage.
• Pool mixture shifts over the training lineage implement a difficulty curriculum, with held-out evaluation rather than training loss as the progress instrument.

  Figure 2: Per-domain held-out loss per stage for code, STEM, Indic-script mean, and web text. Loss steadily drops; code consistently lowest.

  

  Figure 3: Indic script-wise held-out loss at 9B harvest. All protected scripts reach strong loss, with outlier harder scripts attributed to evaluation set composition.

Numerical Results and Empirical Claims

Loss Reduction and Scale-Up

The state-preserving training lineage results in monotonically decreasing held-out and training loss, with each expansion picking up near the previous minimum rather than re-initializing.

Figure 4: The full genealogy—continuous loss reduction from 2B dense to 120B MoE, with growth/harvest points.

Figure 5: Stage-wise loss curves, each stage truncated at its harvest checkpoint.

At 120B, the final released checkpoint achieves a training loss of 1.78 (trailing-100 step mean, step 5161).

Routing and MoE Health

With top-12 routing over 460 experts, expert utilization remains balanced and healthy throughout training, with no dead experts and modest concentration over the top-10 experts.

Figure 6: 120B expert routing share—no dead experts and healthy load balancing.

Rank Analysis and Adaptation Regime

Contrary to typical LoRA-based adaptation rationales, routed experts exhibit relatively high effective rank (0.74 at 90% spectral energy), with attention and memory-projection matrices more amenable to low-rank adaptation. Consequently, expert adaptation is driven by parameter mass, not spectral compressibility.

Figure 7: Effective rank by layer depth and weight class in 9B, justifying uniform adapter rank allocation.

Flushless Adapter Training at Scale

Flush-based methods effective at small scale are found to diverge at 120B, necessitating flushless continuous adapter accumulation for stability in large MoE settings.

Multilingual and Code Proficiency

Protected per-batch representation for code and Indic-script data yields strong per-domain held-out loss across the lineage, extending to all major Indic scripts and code modalities.

Practical Implications

Reproducibility

The paper's emphasis on documenting the operational pipeline, checkpoint lineage, settings, and invariant-preserving transforms addresses a longstanding transparency gap for large MoE training. The released codebase and checkpoints facilitate end-to-end reproduction.

Resource-Efficient Training

This work demonstrates that, with judicious systems design—favoring reversibility, sharding-aware recomputation, and quantized-adapter regimes—a 120B-parameter MoE can be grown and pretrained on a single node with industry-standard 8-GPU hardware. This has direct implications for democratizing access to very large LMs outside of hyperscale clusters.

Methodological Cautions

Multiple failure modes are catalogued, including silent misloads (target/source keyspace mismatch), expert collapse under naive top-$k$ routing after cloning, over-reliance on low-rank justifications for adaptation, and pipeline instabilities under abrupt distributional shifts. The paper advocates invariant-instrumented development rather than trusting plausibility checks.

Future Directions

LightningLM’s modular, reversible, and state-preserving scaling approach is well-suited for further research on:

• Efficient continued pretraining and expert specialization
• Rapid scaling to even larger sparse models with affordable hardware
• Robustification of adapter regimes and rank scheduling at different scales
• Application to other modalities and language families exhibiting severe proxy bias in traditional selection pipelines

Conclusion

"Reversible Foundations" provides a comprehensive systems and architectural methodology for growing a 120B-class sparse MoE on a single node via state-preserving scaling, reversibility, and memory-optimized training procedures. It distinguishes itself by operational transparency, careful tracing of invariants, and extensive documentation of both successful techniques and observed pitfalls. The integration of analytic, calibration-free quantization (TurboQuant), flushless adaptation, and three-tier data guarantees are significant contributions for scaling LLMs under constrained resources, providing a replicable and extensible template for future MoE-based scaling efforts in NLP and beyond.