Arcee Trinity Large Technical Report

Abstract: We present the technical report for Arcee Trinity Large, a sparse Mixture-of-Experts model with 400B total parameters and 13B activated per token. Additionally, we report on Trinity Nano and Trinity Mini, with Trinity Nano having 6B total parameters with 1B activated per token, Trinity Mini having 26B total parameters with 3B activated per token. The models' modern architecture includes interleaved local and global attention, gated attention, depth-scaled sandwich norm, and sigmoid routing for Mixture-of-Experts. For Trinity Large, we also introduce a new MoE load balancing strategy titled Soft-clamped Momentum Expert Bias Updates (SMEBU). We train the models using the Muon optimizer. All three models completed training with zero loss spikes. Trinity Nano and Trinity Mini were pre-trained on 10 trillion tokens, and Trinity Large was pre-trained on 17 trillion tokens. The model checkpoints are available at https://huggingface.co/arcee-ai.

• The paper presents a sparse Mixture-of-Experts model with 400B total parameters and a novel SMEBU update to stabilize expert routing.
• It details architectural innovations including interleaved local/global attention and a custom 200k BPE tokenizer to boost performance.
• Evaluation shows competitive benchmarks, efficient inference via FP8 quantization, and context extension up to 1M tokens.

Arcee Trinity Large: Technical Overview and Implications

Model Architecture and Innovations

Arcee Trinity Large is an open-weight, sparse Mixture-of-Experts (MoE) LLM, scaling to 400B total parameters with 13B active per token. Its design leverages architectural features for optimal training and inference efficiency:

• Attention: Implements interleaved local/global attention with a 3:1 local to global layer ratio, employing RoPE exclusively in local layers and NoPE in global layers. Grouped-Query Attention (GQA), QK-normalization, and gated attention are integrated for stability and context comprehension, with evidence of length extrapolation.
• Mixture-of-Experts: MoE layers employ fine-grained routed experts and an always-active shared expert, using sigmoid routing and auxiliary-loss-free load balancing for Trinity Mini/Nano. Trinity Large introduces Soft-Clamped Momentum Expert Bias Updates (SMEBU), a continuous, $\tanh$-clamped magnitude-aware update with momentum for bias stabilization, addressing observed oscillatory instability in expert routing under standard load balancing.
• Normalization and Initialization: Depth-scaled sandwich RMSNorm, initialized via width-scaled truncated normal; embedding activations are scaled by $\sqrt{d}$, matching leading implementations (e.g., Grok).
• Tokenizer: Custom 200k BPE tokenizer, with multi-stage pre-tokenization for digit isolation, script-aware multilingual processing, and byte-level fallback. Empirically, larger vocabulary yielded improved compression and downstream task performance.

The architecture effectively balances sparsity and throughput requirements, activating fewer, larger experts per token and maximizing model capacity (Figure 1).

Figure 1: Trinity family architecture illustrating interleaved attention and expert routing.

Training Pipeline and Stability Engineering

Pre-training uses a multi-phase, massive mixed corpus, with Trinity Large trained on 17T tokens from a 20T mix targeting code, STEM, reasoning, and multilingual domains. Synthetic data generation exceeded 8T tokens, utilizing scalable infrastructure (Ray, vLLM, Kubernetes). The Random Sequential Document Buffer (RSDB) was introduced for phase 3, reducing batch heterogeneity and improving gradient stability, quantifiable by BatchHet metric.

Stability measures in Trinity Large included a switch to SMEBU for MoE load balancing, increased dense layers, intra-doc masking, z-loss regularization, auxiliary sequence-wise balancing loss, and fallback to BF16 for linear layers. These interventions eliminated expert collapse, preserved utilization balance, and maintained smooth loss convergence, as shown in the uninterrupted loss curve (Figure 2).

Figure 2: Training loss for Trinity Large, highlighting batch size increases and data mix transitions.

Context Extension and Post-Training Procedures

Context extension is performed by directly training at extended sequence length (up to 256K, evaluated at 512K and beyond), leveraging data mixes biased towards long sequences and document concatenations (e.g., code repositories, OCR-derived PDFs). Extension methodology aligns with findings that larger models and sequence lengths facilitate ease of extension, with Trinity Large achieving MK-NIAH (@256K) 0.994, and extrapolating effectively to 1M context.

Post-training encompasses instruction-tuning on blended datasets, agentic coding supervision via OpenCode harnesses, supervised fine-tuning using greedy packing and Cut Cross-Entropy for memory efficiency, and a short RL stage with verifiable rewards (prime-rl, verifiers API).

Empirical Evaluation and Benchmarking

Trinity Large Base and Trinity Large Preview were evaluated across coding, math, commonsense, knowledge, and reasoning tasks. Notable metrics include:

• MBPP+ (88.62), Minerva MATH500 (65.20), HellaSwag (90.11), MMLU (82.58), BBH (65.70), GPQA Diamond (43.94)
• Trinity Large Preview: MMLU (87.21), GPQA Diamond (63.32), SimpleQA (23.92), AIME25 (24.36)

Performance is competitive against recent open-weight MoE and dense models, despite Trinity's higher sparsity and lower active parameter count (Figure 3).

Figure 3: Benchmark comparison of Trinity Large Base versus peer open-weight models.

For inference throughput, FP8 quantization on vLLM (8xH200) demonstrates strong performance due to extreme sparsity and efficient attention split (Figure 4).

Figure 4: Throughput comparison across models, leveraging Trinity Large’s sparsity and attention structure.

Theoretical and Practical Implications

The Trinity Large technical report showcases several implications:

• Sparsity Scalability: The SMEBU update mechanism suggests a pathway to stabilizing MoE training at extreme sparsity levels, enabling larger scaling without active parameter count increases or expert collapse.
• Batch Efficiency: RSDB and context extension strategies augment sample efficiency, reduce noise in training objectives, and facilitate context scaling for long-running agentic tasks.
• Open-Weight Adoption: Trinity’s design, data provenance, and licensing transparency target enterprise deployment, regulatory, and jurisdictional requirements, supporting full auditing and adaptation.
• Inference Optimization: Architectural choices (interleaved attention, GQA, gated attention, large experts) yield high inference throughput even under tight quantization constraints, supporting practical applications in long-context agentic workflows.

Future Directions

Key areas highlighted for further investigation:

• Further scaling sparsity in MoE (higher expert count, more aggressive routing) to maximize capacity and efficiency;
• Algorithmic advances in critical batch size and sample efficiency, enabling even faster and more stable training on larger hardware clusters;
• Controlled ablation studies on stabilization interventions (SMEBU, dense layers, masking);
• Exploration of 1M+ context generalization, sustained agentic code interaction, and synthetic dataset designs for reasoning and STEM.

Conclusion

Arcee Trinity Large represents a technically rigorous advance in sparse MoE language modeling, offering open-weight checkpoints alongside innovations in expert routing stability, batch efficiency, context extension, and inference throughput. Its empirical results demonstrate strong benchmark performance relative to active parameter count and practical inference scalability. Ongoing research into extreme sparsity and scalable training promises further gains in efficient LLM deployment and capability (2602.17004).