Laguna M.1: Sparse MoE Transformer Model

• Laguna M.1 is a sparsely activated Mixture-of-Experts Transformer model featuring 225.8B total parameters with only 23.4B activated per token for efficient agentic coding.
• Its architecture employs dynamic expert routing, a dense stabilization layer, and a reproducible 'Model Factory' system for scalable, distributed training.
• Benchmark evaluations show state-of-the-art performance on execution-grounded, long-horizon coding tasks, outperforming comparable MoE models.

Laguna M.1 is a large-scale, sparsely activated Mixture-of-Experts (MoE) Transformer foundation model designed for high-performance, long-horizon agentic coding. With a total parameter count of 225.8 billion, but only 23.4 billion parameters activated per token, it delivers efficiency and competitive capability in execution-grounded coding benchmarks. Laguna M.1 was trained from scratch in a tightly integrated, versioned pipeline referred to as a “Model Factory,” which formalizes every aspect of large-model development—from data curation and ablation to distributed training orchestration—to enable a reproducible industrial process (Abadji et al., 26 May 2026).

1. Model Architecture and Sparsity Design

The core of Laguna M.1’s architecture is a Mixture-of-Experts Transformer, which deploys a routing mechanism to select a small subset of expert subnetworks at each MoE layer. The principal quantities are:

• Total parameters: $N_{\rm total}=225.8\times10^{9}$
• Activated per token: $N_{\rm active}=23.4\times10^{9}$

At each MoE layer, a token’s representation $x_\ell$ is mapped to $x_{\ell+1}$ via:

$$x_{\ell+1}=x_\ell+\sum_{j\in\mathrm{TopK}(g(x_\ell))} \bar g_j\,{\Phi}_j(x_\ell)$$

where $g(x)=\mathrm{sigmoid}(W_g x + b_g)$ produces gating scores, the top-8 ($k=8$) experts are selected from a pool of 256 ($E=256$), their scores $\bar g_j$ renormalized, and only the corresponding feed-forward sub-blocks $\Phi_j$ executed. One shared expert $N_{\rm active}=23.4\times10^{9}$0 is always active per token. The result is a substantial reduction in inference compute compared to a dense (fully-activated) parameter regime.

Architectural features include:

• Pre-norm Transformer layers with RMSNorm.
• Byte-Pair Encoding (BPE) tokenizer (vocab size 100,352).
• First transformer layer is dense to stabilize early training.
• Linear gating, sigmoid activation, and per-token load-balancing loss.

2. Model Factory: Industrial Pipeline for Foundation Model Training

Laguna M.1 was developed not through single-use scripts but through a modular, versioned system called the “Model Factory.” This stack enforces reproducibility and enables systematic scaling via several infrastructure principles:

• Experiments as Code: All data, hyperparameter sweeps, and ablations are tracked in a single Git repository, with Dagster orchestrating directed acyclic graphs (DAGs) for assets, runs, and checkpoints.
• Composable Components: A unified PyTorch codebase (“Titan”) underpins all phases—pretraining, SFT, RL, and inference. Data are ingested via Spark, stored/mixed in Blender (gRPC), and treated as assets in Dagster. Inference is managed by “Atlas” (vLLM-based), which maintains consistency across CI, evaluation, RL rollouts, and deployment.
• Human Attention Efficiency: A custom Kubernetes batch scheduler replaces Volcano, supports per-job preemption and container stickiness, features automatic failure recovery (e.g., NCCL errors), and employs cross-replica hash checks for silent corruption.

This tightly coupled system enables rapid scaling, asset tracking, and robust distributed job management at the scale required for models with hundreds of billions of parameters.

3. Data Pipeline and Distributed Training Regimen

Laguna M.1 was pretrained on approximately 30 trillion tokens, utilizing 6,144 NVIDIA H200 GPUs. The dataset comprised a curated mixture of high-quality web data, open-source code, scholarly, educational corpora, and minor early synthetic data components:

• Unique tokens pool: 27 trillion; training draw: ~30 trillion tokens.
• Preprocessing included snapshot-level fuzzy deduplication, boilerplate removal, language normalization, and conservative filtering.
• Mixed-precision BF16, FP32 master weights.
• Optimizer: Muon, with AdamW-scale weight decay (Moonlight rescaling), for optimized numerical stability.
• Cosine learning-rate schedule with an initial warmup and decay profile.
• Training system: PyTorch "Titan," distributed via DDP (data parallel), FSDP (fully sharded), tensor parallelism for dense layers, expert parallelism for MoE, and pipeline parallelism.
• Specialized system-level throughput optimizations—fused all-to-all and grouped GEMM kernels for MoE dispatch.

4. Post-Training: Imitation, Supervised Fine-Tuning, and Agentic RL

Post-pretraining, training proceeded through a multi-stage process in the same infrastructure:

• Mid-training (Imitation Stage): 1 epoch, ~60B tokens with long sequence-length (131,072). Task mix: 40% chain-of-thought reasoning, 30% agentic coding, 30% general dialogue, with cosine learning-rate decay.
• Supervised Fine-Tuning (SFT): 3 epochs × 40B tokens; 85% agentic coding trajectories packaged as verifiable git-based tasks, 15% non-agentic to maintain skill diversity. Special role and action tokens (<assistant>, > , <tool_call>) are initialized via subtoken averaging with warmup.

  • Reinforcement Learning (RL): CISPO (clipped REINFORCE with leave-one-out, group-relative advantage), with containerized code sandboxes as evaluation environments. Sparse loss penalties for syntax or tool-call errors and reward is assigned by unit test pass/fail.

  During RL, weights are broadcast every 2 steps via NCCL/RDMA and the KV cache is reset at each update.

  5. Quantization and Hardware Efficiency

  No quantized release is detailed for Laguna M.1, but techniques tested on XS.2 are described and applicable. These methods include:

  • FP8 weight quantization with amax-based scaling (per-tile granularity).

  • INT4 via AWQ, SpinQuant rotation, and mixed-precision for the final layers.
  • NVFP4 quantization-aware distillation.
  • FP8 storage for KV caches.

  These approaches target deployment flexibility across hardware profiles, maximizing memory efficiency and inference throughput.

  6. Evaluation: Agentic Coding Benchmarks

  Laguna M.1 was rigorously evaluated on four execution-grounded, multi-step agentic coding benchmarks: SWE-bench Verified, SWE-bench Multilingual, SWE-Bench Pro, and Terminal-Bench 2.0. Each evaluation launches its own isolated sandbox and samples with high temperature and beam width ($N_{\rm active}=23.4\times10^{9}$1), supporting 256K context windows and “thinking” mode activation.

  A summary of M.1's competitive standing is given in Table 1:

  Model	Params	Active	SWE-bench Verified	Multilingual	Pro	Terminal 2.0
  Laguna M.1	225.8B	23.4B	74.6%	63.1%	49.2%	45.8%
  Devstral 2 (dense)	123B	123B	72.2%	61.3%	—	32.6%
  GLM-4.7 (dense)	355B	32B	73.8%	66.7%	—	41.0%
  DeepSeek-V4 Flash	284B	13B	79.0%	73.3%	52.6%	56.9%
  Qwen 3.5-397B-A17B	397B	17B	76.2%	69.3%	50.9%	52.5%

  On SWE-bench (Verified, Multilingual), M.1 outperforms all open-weight MoEs of comparable scale, and remains state-of-the-art on Terminal-Bench 2.0 and SWE-Bench Pro in its class (Abadji et al., 26 May 2026).

  7. Design Insights and Challenges

  Laguna M.1’s development yielded several insights relevant to large-scale MoE foundation model engineering:

  • Numerical Stability: MoE-specific instabilities—such as expert collapse induced by optimizer misalignment (Muon vs. AdamW)—required scaling weight decay appropriately. BF16 all-reduce of LM-head gradients caused logit drift, corrected via FP32 collectives. Padding tokens saturated expert capacity, resolved by omitting them from routing and load-balancing loss.
  • System Reliability: Cross-replica hash checks were instituted to detect silent hardware errors. Custom scheduling (per-job eviction, pod stickiness) and overlapping MoE dispatch with fused GEMMs improve run turnaround and maximize NVLink utilization.
  • Data Efficiency: Manual high-precision filtering inadvertently promoted high repetition and data misallocation; this motivated development of automatic dataset remixing and synthetic rephrasing (“AutoMixer”) in subsequent models.
  • Scaling Limits: High recall, not aggressive filtering, is needed to supply sufficiently diverse patterns when pretraining at >trillion-token scale.

  A plausible implication is that large MoE models can sustain SOTA coding agent performance with optimized system design and distributed, versioned orchestration, without linear scaling of hardware requirements with model size.

  • (Abadji et al., 26 May 2026) “Laguna M.1/XS.2 Technical Report”