Trinity Large: Scalable Sparse MoE Model

• Trinity Large is a sparse Mixture-of-Experts language model that integrates a 66-layer architecture and advanced MoE routing, enabling high-capacity performance with only 13B active parameters per token.
• It employs innovative attention mechanisms such as QK-normalization and a 3:1 RoPE/NoPE interleaving pattern to facilitate long-context handling and improve training stability.
• The model introduces the SMEBU algorithm for smooth expert bias updates and uses RSDB tokenization to optimize expert load balance and reduce batch heterogeneity during training.

Trinity Large is a sparse Mixture-of-Experts (MoE) LLM developed in the Arcee Trinity series. Featuring 400 billion total parameters with 13 billion activated per token, Trinity Large introduces architectural, training, and stabilization innovations targeted at scaling Transformer-based models while maintaining stability and efficient inference. The series also includes Trinity Nano (6B total parameters, 1B active) and Trinity Mini (26B, 3B active), but Trinity Large represents the flagship high-capacity model (Singh et al., 19 Feb 2026).

1. Architectural Design

Trinity Large employs a 66-layer stack comprised of an initial 6 dense “warm-up” layers followed by 60 decoder-only Transformer layers. The model dimension $d$ is 3072 with an FFN inner dimension of 12,288. Its attention mechanism uses 48 query heads ($h_q$) with per-head dimension 128 and 8 key-value heads. Sequence length during pre-training is 8192, utilizing a local window for sliding-window attention (SWA) of 4096.

A defining feature is the Mixture-of-Experts architecture: each MoE layer consists of 1 shared and 256 routed experts, with 4 experts activated per token. Each expert has a hidden size of 3072, with a route scale factor of 2.448. The hierarchical MoE structure allows high total parameter count but keeps per-token computation tractable by sparsely routing inputs.

The tokenization process utilizes a custom 200,000-token BPE vocabulary, trained on approximately 10 billion tokens, and is implemented with the Random Sequential Document Buffer (RSDB), which is pivotal for reducing minibatch imbalance. Embeddings are scaled by $\sqrt{d}$.

2. Attention Mechanisms and Normalization

Trinity Large implements QK-normalized multi-query attention, in which queries and keys for each token are projected via $W^Q$ and $W^K$ then RMS-normalized. Values are likewise projected to the respective heads. The attention mechanism alternates between sliding-window local attention (three layers, rotary embeddings via RoPE) and global attention (one layer, full causal using NoPE), in a 3:1 interleaving pattern. This scheme enables long-context handling without prohibitive training cost.

Each query head is grouped and mapped to KV heads with scaled dot-product attention, and grouped-query routing further concentrates the model’s capacity. Attention output is gated: $g_t = \sigma(W^G x_t)$, split across heads, multiplies with the attended outputs to produce $\tilde{o}_{t,i}$, which is projected to $u_t$. This gated mechanism is designed to suppress "attention sinks," contributing to more stable and generalizable representations.

Layer and sublayer normalization is handled via a depth-scaled sandwich norm: $$y_\ell = x_\ell + \mathrm{RMSNorm}_\ell^{(2)}( \mathcal{M}_\ell(\mathrm{RMSNorm}_\ell^{(1)}(x_\ell)))$$, where the gain of the second RMSNorm is initialized to $1/\sqrt{L}$, reflecting the model’s depth.

3. MoE Routing and the SMEBU Algorithm

The Mixture-of-Experts architecture in Trinity Large features routed expert selection using sigmoid gating. Given input $h_q$0, the router computes selection logits $h_q$1 and expert activation is gated through a bias-adjusted TopK. Only experts with top scores receive nonzero activation; $h_q$2 values are normalized across selected experts. All expert MLPs use SwiGLU activation.

Expert bias updates in Trinity Large are stabilized by the Soft-clamped Momentum Expert Bias Updates (SMEBU) algorithm. SMEBU replaces sign-based, magnitude-insensitive updates with a soft-clamped, momentum-based rule:

• $h_q$3, where $h_q$4 is expert load and $h_q$5 the mean
• Update is bounded: $h_q$6
• Bias is updated with momentum: $h_q$7
• All bias updates are recentered across experts.

Compared to sign-based updates, SMEBU allows updates to decay smoothly as the load approaches balance, eliminates perpetual oscillations, and applies momentum to reduce update variance. Empirical ablations show that SMEBU was essential for stable expert-load curves and for avoiding “expert collapse” modes (Singh et al., 19 Feb 2026).

4. Training Regimen and Infrastructure

Training utilized the Muon optimizer for hidden layers and AdamW for embedding and LM heads, with weight decay only on non-normalization parameters. The warm-up schedule spanned 2000 steps, with peak learning rates at $h_q$8 (Muon) and $h_q$9 (AdamW). The global batch size was 12,288 (sequence length 8192) and increased to 16,384 after 4.9T tokens, with cosine decay down to 1/20 of peak LR during context extension.

Pre-training data comprised 17 trillion tokens sampled from a 20T mix, split across three phases:

• Phase 1: 13T tokens of general web, code, and STEM
• Phases 2 and 3: increased fractions of code, math, instruction, and multilingual content
• Roughly 8T synthetic tokens were created through large-scale rephrasing

The RSDB on-the-fly tokenization reduced batch heterogeneity (BatchHet) by 4.23× and variance by 2.4× in phase 3, eliminating loss spikes.

Training ran on 2048 B300 GPUs using HSDP + Expert Parallelism (group size 8) and context parallelism of 4. Continuous monitoring—disabling custom FP8 kernels if necessary—precluded interruptions. Reported training stability was high: no loss spikes occurred, and loss decayed monotonically even through batch-size and data-mixture transitions.

5. Model Family Comparison

Trinity Large is one model in the Arcee Trinity series. Its smaller counterparts, Trinity Nano and Trinity Mini, share core architectures but differ in scale and parameterization.

Property	Nano	Mini	Large
Total params	6B	26B	400B
Active params/token	1B	3B	13B
Transformer layers	56	32	60
Initial dense layers	2	2	6
d_model	1024	2048	3072
FFN dim	3072	6144	12,288
Query / KV heads	8/2	32/4	48/8
Local window	2048	2048	4096
Pre-train seq len	4096	4096	8192
Shared / routed experts	1/128	1/128	1/256
Experts activated/token	8	8	4
Route scale	2.826	2.826	2.448
Expert size	256	1024	3072
Init $\sqrt{d}$0 (trunc-norm)	0.016	0.011	0.009
Pre-train tokens	10T	10T	17T

This scaling strategy emphasizes that the sparse MoE routing allows for massive parameter counts (400B) but with a practical activation footprint (13B/token) to optimize throughput and memory utilization.

6. Evaluation, Performance, and Ablations

Trinity Large demonstrates high performance over a diverse selection of benchmarks. Zero- and few-shot evaluation indicates strong results:

• MBPP+ (code): 88.62
• Minerva MATH500: 65.20
• HellaSwag (5-shot): 90.11
• WinoGrande (5-shot): 80.82
• MMLU (5-shot): 82.58
• MMLU-Pro (5-shot): 66.02
• TriviaQA (5-shot): 83.30
• ARC Challenge (0-shot): 65.44
• BBH (few-shot): 65.70
• GPQA Diamond (5-shot): 43.94

After instruction tuning (“Preview”), improvement is observed: MMLU 87.21, MMLU-Pro 75.25, GPQA Diamond 63.32, SimpleQA 23.92, and AIME25 24.36.

Figure 1 of the report visualizes seamless loss decay throughout training phases and parameter changes. No loss-spike events occurred.

Inference throughput, tested using vLLM and FP8 precision on 8 H200 GPUs, shows Trinity Large outperforming comparable dense and sparse models on throughput, with results referenced in "trinity-large-throughput-comp.png."

Ablation studies underline that gated attention and QK-normalization are critical for eliminating "attention sink" instabilities. The 3:1 RoPE/NoPE pattern further supports long-context extrapolation efficiently. SMEBU was crucial for maintaining expert load balance and eliminating “expert collapse,” whereas RSDB dramatically improved training stability by reducing batch heterogeneity and gradient-norm kurtosis.

7. Availability and Significance

Trinity Large, along with Nano and Mini models, is available with checkpoints at https://huggingface.co/arcee-ai. The introduction of SMEBU, advanced attention schemes, and innovations in training stability position Trinity Large as a representative achievement in large-scale sparse Transformer LLMs. The architecture and stabilization findings contribute practical insights for MoE design, load balancing, and robust pretraining at scale (Singh et al., 19 Feb 2026).