Laguna M.1/XS.2 Technical Report

Abstract: We present Laguna M.1 and Laguna XS.2, two Mixture-of-Experts foundation models built for long-horizon, agentic coding: M.1 has $225.8$B total parameters ($23.4$B activated per token) and XS.2 has $33.4$B total ($3$B activated). Both models were trained from scratch end-to-end inside the same internal system that we refer to as our Model Factory: a tightly-integrated stack of versioned data, training, evaluation, and inference components that turn model development into an industrial process. We describe the principles and design choices of the Model Factory and also detail the end-to-end training process of our models, throughout pre-training data and architecture, post-training stages, evaluation, and quantization. On agentic software engineering and terminal benchmarks (SWE-bench Verified, SWE-bench Multilingual, SWE-Bench Pro, and Terminal-Bench 2.0) M.1 and XS.2 are competitive with state-of-the-art open models in their respective weight classes. Laguna XS.2 weights are released under Apache~2.0 at https://huggingface.co/collections/poolside/laguna-xs2.

• The paper details the scalable development and deployment of two MoE models, emphasizing agentic coding workflows and long-context training.
• It outlines a Model Factory system that automates reproducible model development with rigorous ablation studies and distributed optimizations.
• The report demonstrates significant performance gains on coding benchmarks via innovative quantization, RL fine-tuning, and advanced optimizer techniques.

Detailed Technical Summary of "Laguna M.1/XS.2 Technical Report" (2605.27605)

Introduction and Model Overview

The "Laguna M.1/XS.2 Technical Report" details the design, implementation, training, evaluation, and deployment of two Mixture-of-Experts (MoE) models: Laguna M.1 ($225.8$B parameters, $23.4$B activated per token) and Laguna XS.2 ($33.4$B total, $3$B activated). Both models are optimized for long-horizon, agentic coding workflows, emphasizing multi-step reasoning and code manipulation tasks. The models were constructed within a specially engineered system—the Model Factory—which systematizes and accelerates foundation model development using industrial principles.

In terms of model architecture, both models employ pre-norm Transformers with RMSNorm and token-choice routing. Laguna XS.2 uses interleaved Sliding Window Attention (SWA)/Global Attention (GA) with 3:1 ratio, Grouped Query Attention (GQA) (8 KV heads), and RoPE positional encoding. The choice of attention mechanism, routed expert modulation, learning rate schedule, and layer design evolved based on systematic ablation studies conducted on proxy MoE models.

Model Factory Concept

The Model Factory operates as a unified, versioned, and highly automated pipeline for model development. All configurations, assets, and experiments are tracked as code, ensuring consistent reproducibility and lineage tracing for every artifact. Experiments leverage Dagster as a control plane for DAG navigation, coupled with a custom batch scheduler implemented atop FoundationDB. This infrastructure enables per-job eviction, topology awareness, and cache-retaining sticky pod respawn, yielding sub-minute placement times, crucial for high-throughput hyperparameter sweeps and model runs.

Researcher and agent workflows are agentic, with AI agents actively participating in ablation design, experiment management, and analysis. Components are fully composable and decoupled, promoting rapid iteration and deployment while minimizing human intervention except for genuinely novel research decisions.

Pre-training Methodology

Training Setup and Stability

Laguna XS.2 was trained on 2,048 H200 GPUs, Laguna M.1 on 6,144, both using the Muon optimizer (Moonlight variant) in mixed-precision and FP32 where numeric stability is critical. The training recipe features a Warmup-Stable-Decay (WSD) schedule, with peak learning rates set according to a scaling law fitted across multiple sizes and token budgets. The scaling law is:

$$\text{lr}^\star(N, D) = 10^{4.488} \cdot N^{-0.4639} \cdot D^{-0.2661}$$

where $N$ is active parameters, $D$ is total token budget. This yields robust performance across diverse setups.

Long-context training is accomplished via staged context extension from 4K up to 128K tokens, applying YaRN to GA layers and further extending to 256K by RoPE scaling. EMA checkpointing improves base model stability.

Training instabilities in M.1—including expert collapse, numerical drift from LM head input-gradient reduction, and padding-induced expert saturation—were systematically addressed in XS.2, with FP32 enforced in numerically sensitive reductions and routing masks applied to non-learnable padding tokens.

Distributed and Efficient Training

Distributed training leverages Titan, a PyTorch-based library supporting DDP, FSDP, TP, EP, and PP. For MoE layers, dispatch/combine collectives are fused into grouped GEMM kernels, overlapping compute and NVLink communication at the SM granularity. Additional SMs are reserved for NCCL collectives; all GPU communication is mapped to appropriate scale-up/scale-out networks.

Reliability is ensured through periodic cross-replica hash checks to catch silent data corruption and logical bugs. Data loading employs streaming via Blender, while checkpoint persistence is optimized for S3 via distributed writes and fast RDMA reads.

Figure 1: Fine-grained overlap of dispatch and grouped GEMM compute during MoE layer execution, optimizing GPU utilization for token routing.

Pre-training Data Composition and AutoMixer

Web and Synthetic Data

Data pipelines incorporate web and code corpora, synthetic traces, and grounded educational datasets, totaling more than 30T tokens. Web data processing employs multi-dimensional tagging and model-based ranking, sampled from quality buckets to maximize diversity while rejecting confirmed noise.

Figure 2: High-level workflow for web data ingestion, deduplication, multi-dimensional quality tagging, conservative filtering, ranking, and quota-controlled sampling.

Figure 3: Distribution of web document buckets pre- and post-sampling, showing prioritization of high-quality regions with sustained diversity.

Synthetic data accounts for $\sim$13\% of XS.2's mix, generated via modular pipelines within Hive—a declarative pipeline runner for agentic generation, filtering, and validation.

Figure 4: Hive runtime, illustrating pipeline orchestration for modular synthetic data generation and validation.

AutoMixer

AutoMixer automates data mixture optimization by training proxy models on mixture perturbations, fitting nonlinear surrogate regressors over observed metrics, then regularizing optimized mixtures toward a prior using KL-divergence constraints. This enables principled balancing among coding, math, web, synthetic, and knowledge domains.

Figure 5: AutoMixer pipeline: proxy models are trained over controlled mixtures, capability evaluations fit surrogate regressors, subsequently optimized for target capability trade-offs.

Strong numerical improvements are recorded: +43% on HumanEval+, +54% on Crux-I, +41% on GSM8K, with modest regression on commonsense tasks, demonstrating successful capability-specific allocation.

Post-training and RL Fine-tuning

Post-training proceeds through mid-training, SFT, and agentic RL stages. Mid-training leverages broad mixture ratios, balancing instruction-following, tool calls, reasoning, and long-horizon coding trajectories, with systematic tuning of tool call presence, reasoning length, and trajectory counts.

SFT targets agentic coding via tasks derived from real git commits and synthetic augmentation, with dedicated instruction-following judges ensuring behavioral adherence. Multi-harness training promotes generalization across agent orchestration frameworks.

Agentic RL employs token-level REINFORCE with importance-ratio clipping (CISPO), length-weighted leave-one-out baselines, and shaping penalties for parsing, tool errors, and trajectory degeneracy. All rollouts are executed in containerized, real-world sandboxes with deterministic reward chains.

Quantization Strategies

XS.2 is quantized to FP8, INT4, and NVFP4 for compatibility with low-VRAM devices. FP8 quantization utilizes SpinQuant R1 rotation for improved quality. INT4 employs AWQ with mixed-precision (first 30 layers INT4, last 10 INT8) to mitigate outlier activation errors, identified via layer-wise activation analysis.

Figure 6: Layer-wise accumulation of outlier activations in the residual stream, informing mixed-precision quantization boundaries.

NVFP4 uses quantization-aware distillation (QAD) and microscaling for robust conversion, leveraging hidden-state caching for efficient full-vocabulary KL distillation.

Evaluation Protocol and Benchmark Results

Base Model Benchmarks

Laguna XS.2 achieves top scores among comparable open-weight MoE models on coding benchmarks (e.g., LiveCodeBench v6: 29.3 vs Qwen3.5: 24.4; MultiPL-E: 58.4 vs Nemotron-3 Nano: 56.1). Performance closely approaches much larger dense and MoE models on select coding tasks.

Agentic Benchmarks

M.1 and XS.2 are evaluated on SWE-bench Verified, SWE-bench Multilingual, SWE-Bench Pro, and Terminal-Bench 2.0. M.1 (74.6 on SWE-bench Verified) and XS.2 (69.9 on SWE-bench Verified) are competitive with the highest-performing open and proprietary models in their parameter classes.

Figure 7: Comparative SWE-bench and terminal task scores for M.1 and XS.2 against Devstral, Qwen, GLM, DeepSeek, Gemma, Claude, and GPT baselines.

Implications and Future Directions

This report substantiates the argument that systematic, industrialized model development—embodied in the Model Factory—accelerates frontier model construction, enabling rapid deployment of high-capability MoE architectures. The Model Factory's reproducibility and automation foster scalable research velocity, crucial as models, pipelines, and post-training become increasingly complex and intertwined.

The successful deployment of AutoMixer for large-scale data mixture optimization and quantization schemes for low-resource inference are practical advances, demonstrating generalizable improvements across coding, reasoning, and agentic benchmarks.

Theoretically, the WSD learning-rate scaling law and distributed optimizer innovations further elucidate high-scale training dynamics, suggesting new research in optimizer design, distributed systems, and stability analysis for MoE models.

Continued development will likely see further automation, more agent-driven research flows, refinement of mixture optimization methodology, and advanced quantization schemes enabling efficient deployment of increasingly larger and more capable models. There is an expectation that teams leveraging industrialized processes will outpace craft-focused efforts as the field scales.

Conclusion

The "Laguna M.1/XS.2 Technical Report" comprehensively documents the assembly, training, evaluation, and deployment of MoE models optimized for agentic coding. The Model Factory approach validates scalable, reproducible model development, while AutoMixer, advanced quantization, and robust distributed training strategies underpin competitive benchmark results. Release of XS.2 weights under Apache 2.0 licensing further facilitates open research and community advancement. Strategic focus on process, automation, and data-driven design positions Laguna M.1/XS.2 as a blueprint for future foundation model development within increasingly complex research landscapes.