ZAYA1-8B: Sparse MoE Reasoning Transformer

• ZAYA1-8B is a reasoning-focused, decoder-only transformer that uses a sparse mixture-of-experts architecture to activate only 700M–760M parameters per token.
• Architecturally, it comprises a 40-layer Transformer with compressed convolutional attention, expressive routing, and lightweight residual scaling optimized for AMD MI300X GPUs.
• It integrates hardware-software co-design with a multi-stage training pipeline, including reinforcement learning, to achieve efficient long-context reasoning and verifiable computation.

Searching arXiv for ZAYA1-8B and related system paper to ground the article with the relevant preprints. ZAYA1-8B is a reasoning-focused, decoder-only mixture-of-experts transformer built on Zyphra’s “MoE++” architecture and trained on a full-stack AMD compute, networking, and software platform. The model is presented most fully in the technical report “ZAYA1-8B Technical Report,” while an earlier systems paper previews “ZAYA1-base” as the base model used to demonstrate large-scale MoE pretraining on pure AMD hardware. The preview reports approximately 760 M active parameters and approximately 8.3 B total parameters, whereas the later technical report describes ZAYA1-8B as having 700M active and 8B total parameters; both sources agree on the model family’s central property that only a sub-billion active parameter set is invoked on each token, despite an 8B-scale total parameter count (Anthony et al., 21 Nov 2025, Washbourne et al., 6 May 2026).

1. Identity within the ZAYA1 model family

ZAYA1-8B belongs to a model family in which sparse activation is used to decouple total parameter capacity from per-token inference cost. In the later report, ZAYA1-8B is described as an 8 billion-parameter model with only 0.7 billion active parameters invoked on each token. In the earlier systems paper, the corresponding base model is introduced as ZAYA1, with approximately 760 M active parameters and approximately 8.3 B total parameters. The earlier paper calls this system “ZAYA1-base” and explicitly frames it as a preview to be improved in forthcoming work (Washbourne et al., 6 May 2026, Anthony et al., 21 Nov 2025).

A common misconception is to read “8B” as a dense per-token compute budget. The published descriptions reject that interpretation. Because the architecture uses top-1 routing in a 16-expert MoE stack, only one expert per token per layer is dispatched, so the active path is far smaller than the total parameter inventory. This suggests that the model’s design objective is not merely parameter count scaling, but the concentration of reasoning and specialization capacity into a low active-parameter operating point.

The two papers also divide labor conceptually. The systems paper emphasizes compute, networking, kernel behavior, checkpointing, and MI300X-aware sizing rules, while the technical report expands the account into reasoning-oriented pretraining, supervised fine-tuning, a four-stage reinforcement-learning cascade, and Markovian RSA as a test-time compute method. Taken together, they define ZAYA1-8B as both a model architecture and a hardware-software co-design exercise.

2. Architectural specification

ZAYA1-base is a 40-layer mixture-of-experts Transformer with hidden size $h=2048$, $16$ attention heads, $16$ local experts per layer, and top-1 routing. The architectural core is shared across the family descriptions: sparse MoE feed-forward blocks, Compressed Convolutional Attention, a three-layer MLP router with depth-aware mixing, and lightweight residual scaling. The systems paper gives the canonical hyperparameter table, while the technical report restates the same backbone as 40 transformer layers of width $2048$, each containing a sparsely activated feed-forward block with $16$ experts, each expert being a $4096 \rightarrow 2048$ two-layer MLP (Anthony et al., 21 Nov 2025, Washbourne et al., 6 May 2026).

Symbol	Definition	Value
$L$	# of layers	40
$h$	Hidden size	2048
$a$	# of heads	16
$g$	# of KV heads	2
$16$0	# of query heads (CCA)	8
$16$1	Experts per layer (local)	16
$16$2	top-k routing	1
$16$3	Expert FFN width (pre-act)	4096
$16$4	Expert post-act width (SwiGLU)	2048
$16$5	Router downproject dim	256

Compressed Convolutional Attention is the family’s principal attention modification. The technical report states that queries are downprojected $16$6 and keys/values $16$7 via a small grouped convolutional preconditioner. The systems paper encodes this in the head counts: $16$8, $16$9, $16$0, and per-head dimension $16$1, chosen to be a multiple of $16$2 for MI300X GEMM efficiency. The later report attributes a $16$3–$16$4 reduction in training and decoding compute to this compressed latent-space attention while retaining full-attention expressivity at long contexts.

The router is explicitly more expressive than a single linear gate. Given a residual stream $16$5, the model forms a router latent $16$6 with $16$7, applies Exponential Depth Averaging, and then passes the normalized latent through a three-layer MLP to produce expert scores. Load balancing is handled with learned bias vectors updated via a PID-inspired AdamW loop:

$16$8

The systems paper characterizes this design as yielding lower entropy and high expert specialization, thereby obviating residual experts and higher top-$16$9.

Residual Scaling introduces learned gates and biases on both the residual and layer-output branches:

$2048$0

The report describes these as negligible-cost parameters used to control residual-norm growth and improve deep-network stability. This suggests that ZAYA1-8B’s architectural distinctiveness lies less in exotic macro-topology than in a coordinated set of efficiency-oriented modifications to routing, attention compression, and residual dynamics.

3. Hardware platform and system co-design

The pretraining platform is described as an end-to-end AMD cluster centered on MI300X GPUs and Pollara networking. Each compute node contains eight MI300X GPUs with 192 GB HBM each, interconnected via $2048$1 GB/s xGMI links, and eight Pollara 400 Gbps NICs, one 400 Gb/s interface per GPU, in a rails-only, two-level leaf/spine fabric. The later report adds one 200 Gb/s Pensando NIC for data and checkpoints, 25.6 TB NVMe, 2 TB DDR5 memory, dual Intel Xeon Platinum CPUs, a dedicated 120 TB RAID 0 NVMe storage node, and login nodes under KVM for orchestration (Anthony et al., 21 Nov 2025, Washbourne et al., 6 May 2026).

The software stack spans HIP, Composable Kernel, TransformerEngine, rocBLAS, RCCL, a Megatron-LM fork, Primus kernels, Muon, ROCm-ROS-based I/O, AITER, and AMD Pensando fabric software. The systems paper emphasizes often-ignored operational utilities, including fault tolerance and checkpoint reshaping, and the technical report extends that emphasis into RL-scale MoE training with exact router replay between inference and trainer execution.

Hardware-aware model sizing is treated as a first-class design constraint. The systems paper states that MI300X GEMM performance peaks at approximately $2048$2 GFLOP problem sizes, motivating the choices $2048$3 and $2048$4. Macro-shapes $2048$5 and micro-shapes $2048$6 or $2048$7 are rounded to powers of two or multiples of $2048$8 to hit the performance “sweet spots.” For the interconnect, intra-node xGMI AllReduce peaks at approximately $2048$9 GB/s at large messages, with latency of approximately $16$0 for $16$1 MB; inter-node Pollara AllReduce saturates at approximately $16$2 Tb/s aggregate, with optimal fusion buffer approximately $16$3 KiB; and AllGather and ReduceScatter show similar bandwidth/latency crossover at approximately $16$4 KiB messages. The design lesson drawn explicitly in the paper is that per-message sizes should be at least approximately $16$5 KiB to saturate the 400 Gb/s links.

The same co-design logic extends to inference. The systems paper attributes low time-to-first-token to CCA’s compressed KV cache, and states that rampable context via CCA reduces KV cache footprint by $16$6, decomposed as $16$7 query compression and $16$8 KV compression in CCGQA. A plausible implication is that ZAYA1-8B’s test-time compute methods depend not only on algorithmic aggregation, but also on a memory hierarchy in which long-context prefill remains economically tractable.

4. Pretraining, midtraining, and supervised fine-tuning

The training recipe is reported in two forms. The systems paper describes three phases totaling approximately $16$9 T tokens: Phase 1 uses $4096 \rightarrow 2048$0 T tokens from a mix of web-crawl, code, and math with cosine decay learning rate $4096 \rightarrow 2048$1 and batch size $4096 \rightarrow 2048$2 M tokens; Phase 2 uses $4096 \rightarrow 2048$3 T tokens with upweighted code, math, and reasoning and learning rate $4096 \rightarrow 2048$4; Phase 3 uses $4096 \rightarrow 2048$5 T tokens to extend context from $4096 \rightarrow 2048$6 K to $4096 \rightarrow 2048$7 K with cosine $4096 \rightarrow 2048$8 and then linear decay to $4096 \rightarrow 2048$9, with batch size $L$0 M tokens. The later technical report reorganizes the lifecycle into base pretraining at $L$1 K context for $L$2 trillion tokens, continued base pretraining at $L$3 K for $L$4 trillion tokens, reasoning-focused midtraining at $L$5 K context for $L$6 trillion tokens, and supervised fine-tuning at $L$7 K context for $L$8 billion tokens (Anthony et al., 21 Nov 2025, Washbourne et al., 6 May 2026).

A key point is that reasoning data is not described as a purely post-training addition. The technical report states that reasoning data was included from pretraining onward using “answer-preserving trimming” for long traces that exceed the context budget. The procedure is defined stepwise: keep the sample intact if its full length is within budget $L$9; otherwise truncate the tail of the last > … block while always preserving the final answer; if still too long, drop earlier <think> blocks while keeping their answers and then re-trim the final block; if answers alone exceed $h$0, discard the sample. Formally,

$h$1

The report states that this avoids “train on dangling CoT” artifacts.

Long-context execution uses context parallelism for $h$2 K via Ring Attention and custom CCA sharding. In the $h$3 K SFT stage, best-fit decreasing bin packing is used to avoid hallucination at truncation boundaries. The SFT mixture includes chat templates, instruction prompts, code, math, reasoning, tool-calls, and Markovian RSA aggregation examples built from sampled reasoning tails.

Optimization and training-state management are likewise explicit. The systems paper uses Muon with five Newton–Schulz steps for 2D parameters and momentum-only dynamics, plus AdamW on 1D and other parameters, with shared learning rate and decoupled weight decay in the Muon update. It also employs ZeRO-1 to shard optimizer states and distributed checkpointing with per-rank shards plus root-only weights for $h$4 saves. The technical report adds BF16 training with selective FP32 upcasting of cross-entropy, CCA state, router softmax, and RMSNorm, and reports rollout-versus-trainer log-prob KL under $h$5. MoE router replay is implemented so the trainer reuses the exact per-token expert assignments produced by the inference engine, eliminating top-1 routing mismatches.

5. Reinforcement-learning cascade for verifiable reasoning

Post-training is organized as a four-stage RL cascade. The technical report names the stages, gives step counts, and specifies distinct objective structures for each. Stage 1, “Reasoning Warmup,” runs for $h$6 steps on $h$7 hard math, puzzle, and chain-of-thought prompts with pass rate at most $h$8. Its algorithmic spine is PipelineRL with DPPO Binary-TV trust region $h$9, Dr-GRPO SMTSN loss aggregation, and the MaxRL advantage estimator

$a$0

The objective is verifiable correctness, $a$1 (Washbourne et al., 6 May 2026).

Stage 2, the RLVE-Gym curriculum, runs for $a$2 steps on $a$3 adaptive, verifiable puzzle-like environments. It uses an online difficulty scheduler based on Thompson sampling over a logistic Item Response Theory model,

$a$4

with each generator maintained at approximately $a$5 solve rate. The reward is the environment verifier.

Stage 3 is split into “Math + Code + TTC RL” phases of $a$6 and $a$7 steps. Phase 1 mixes $a$8 rows of olympiad math, standard code, Markovian RSA prompts, PaCoRe variants, and Code-I/O/ARC/falsification environments. Phase 2 increases emphasis on competitive-programming code while retaining math and RSA. Rewards are binary correctness on math or code, or pass on a verifier, augmented with a difficulty-scaled length penalty:

$a$9

The report states that $g$0 is group solve rate, $g$1 gates only solved responses near the frontier, and $g$2 linearly penalizes length beyond the shortest correct solution.

Stage 4, “Behavioral RL,” runs for $g$3 steps on $g$4 K chat prompts and then two epochs of simple and hard instruction-following. The reward is a preference-model score gated by a binary instruction-following checker, and optimization uses standard GRPO with reward-standard-deviation normalization.

Across RL stages, optimization is done with Muon, described here as an AdamW-style Newton-Schulz orthogonalized optimizer, with zero momentum for actor weights in a $g$5 rolled-out asynchronous PipelineRL regime. This suggests that ZAYA1-8B’s reasoning specialization is not treated as a single RL phase with generic reward shaping, but as a staged curriculum that progressively broadens from hard-verifiable reasoning to adaptive environments, then to code and test-time-compute behaviors, and finally to chat and instruction following.

6. Markovian RSA and test-time compute

Markovian RSA is the model family’s principal test-time compute method. The technical report describes it as recursively aggregating parallel reasoning traces while carrying forward only bounded-length reasoning tails between rounds. The algorithm is parameterized by population size $g$6, sample-per-aggregate $g$7, rounds $g$8, per-rollout budget $g$9, and tail length $16$00: in round $16$01, the model generates $16$02 reasoning chains and stores tails $16$03; in later rounds, each candidate is generated from a prompt consisting of the problem and sampled tails from the previous round, after which a new bounded tail is stored (Washbourne et al., 6 May 2026).

The report makes the cost structure explicit. By carrying only $16$04 tokens per candidate, attention-prefill cost per aggregate is bounded by $16$05, independent of the total reasoning depth $16$06. It also enumerates special cases: $16$07 gives pure Markovian chunked generation, $16$08 gives full-chain RSA, and $16$09 gives parallel sampling or best-of-$16$10 with an external selector.

ZAYA1-8B is trained on this workflow, not merely evaluated with it. SFT includes aggregation examples built from sampled tails, and Stage 3 RL includes Markovian RSA prompts. The report therefore argues that the model has seen its own inference workflow. A plausible implication is that the gain from Markovian RSA is partly algorithmic and partly distributional, because the model’s training distribution includes aggregation-style prompts.

For deployment, the report recommends $16$11 as a default cost–accuracy balance, noting that this runs at approximately $16$12 the wall-clock of a single $16$13 K long-CoT baseline under the batched decoder. For capability ceiling, it uses $16$14 K and still only $16$15 K. Under the $16$16 K/$16$17 K regime, total newly generated tokens per problem are reported as approximately $16$18 K.

7. Performance profile and comparative position

The performance narrative has two layers: base-model evaluation from the systems paper and reasoning-focused evaluation from the technical report. For ZAYA1-base, the systems paper reports zero-shot and pass@$16$19 results against dense and MoE peers. On MMLU(0), ZAYA1-base scores $16$20, compared with Qwen3-4B at $16$21, Gemma3-12B at $16$22, Llama3-8B at $16$23, and OLMoE-1B-7B at $16$24. On MMLU-Pro(5), it scores $16$25 versus $16$26 for Qwen3-4B and $16$27 for Llama3-8B. On GPQA it scores $16$28 versus $16$29 for Qwen3-4B. On MATH-hard it scores $16$30, compared with $16$31 for Gemma3-12B. On MBPP+ it reaches $16$32 pass@1, compared with $16$33 for Qwen3-4B. In pass@$16$34 reasoning evaluation after mid-train, MathArena avg@64 is $16$35 and MCQA avg@16 is $16$36 (Anthony et al., 21 Nov 2025).

The later technical report shifts emphasis to reasoning and test-time compute. With single-rollout sampling $16$37, ZAYA1-8B scores $16$38 on AIME’25 and $16$39 on HMMT’25. With Markovian RSA $16$40, those results increase to $16$41 on AIME’25 and $16$42 on HMMT’25. The report compares these numbers with DeepSeek-R1-0528 at $16$43, Gemini-2.5 Pro at $16$44, DeepSeek-V3.2 at $16$45, and GPT-5-High at $16$46. On LiveCodeBench-v6, Markovian RSA raises ZAYA1-8B from $16$47 to $16$48 (Washbourne et al., 6 May 2026).

Training efficiency figures reinforce the system-level claims. The systems paper reports sustained approximately $16$49 TFLOP/s per MI300X at BF16 for $16$50 k by $16$51 k GEMM shapes, iteration time of approximately $16$52 ms at $16$53 k and $16$54 microbatch per GPU, approximately $16$55 tokens/s/GPU, and approximately $16$56 TB/s HBM bandwidth for fused kernels on contiguous reads and writes. It also states that custom kernels for Muon and fused RMSNorm reduce optimizer overhead to less than $16$57 of iteration time, and that Aegis fault tolerance automates NIC/CQE recovery, NCCL failure healing, and node replacements.

Taken together, the reported evidence places ZAYA1-8B at the intersection of sparse MoE design, long-context reasoning training, RL for verifiable tasks, and AMD-specific systems engineering. The recurring comparative claim across the two papers is consistent: a model with under $16$58B active parameters can be trained and deployed so as to match or exceed substantially larger baselines on several mathematics, coding, and reasoning benchmarks, while retaining low-latency inference characteristics associated with compressed attention and bounded-workspace aggregation.