Soofi S 30B-A3B Foundation Model

• The model’s main contribution is its novel Mixture-of-Experts design that activates only 3.2B parameters per token, enabling efficient bilingual processing.
• Its architecture combines Mamba-2, MoE, and GQA layers to maintain near-constant long-context serving with a minimal memory footprint.
• Pretrained on 27 trillion tokens with a focus on German content, it was built on German sovereign industrial AI infrastructure for transparency and reproducibility.

Soofi S 30B-A3B is a sovereign, open-source Mixture-of-Experts hybrid Mamba Transformer foundation model for German and English. It activates only 3B of 30B parameters per token, was pretrained on roughly 27 trillion tokens with deliberately up-weighted German, and was built end-to-end on the German Industrial AI Cloud operated by Deutsche Telekom in Munich. In the reported comparison set, it matches dense 14 to 27B models on aggregate English and German benchmarks, achieves the best code aggregates in both languages among 17 open base models, and attains the highest English and German evaluation scores among fully open models, ahead of Olmo 3 32B and Apertus 70B (Soofi-Team et al., 10 Jul 2026).

1. Definition and design position

Soofi S 30B-A3B is presented as a bilingual German–English foundation model whose defining constraints are sovereignty, openness, and serving efficiency. The model is explicitly framed as a sovereign system because training and release were organized around European infrastructure and auditable data accounting rather than around opaque training mixtures or closed deployment dependencies. The release plan includes weights, selected intermediate checkpoints, full per-source data accounting, hyperparameters, and training and evaluation code, with aggregate statistics and exact mixture accounting for commercially licensed sources (Soofi-Team et al., 10 Jul 2026).

Architecturally, Soofi S implements the Nemotron 3 Nano reference design, which places it in the line of hybrid Mamba-Transformer MoE models rather than purely dense Transformers. Nemotron 3 Nano is itself a Mixture-of-Experts hybrid Mamba-Transformer LLM with a 30B-A3B operating point, and Soofi S adopts that reference architecture for software compatibility, efficient serving, and controlled comparison (NVIDIA et al., 23 Dec 2025).

This positioning has two immediate implications. First, the model is intended to compete with dense 14B–27B systems while exposing only a small active-parameter footprint per token. Second, the bilingual emphasis is not an afterthought layered onto a generic multilingual checkpoint; the training curriculum deliberately up-weights German and evaluates German as a first-class target alongside English (Soofi-Team et al., 10 Jul 2026).

2. Architecture and computational profile

The model has approximately 31.6B total parameters and approximately 3.2B active parameters per token, or approximately 3.6B including embeddings. Its 52-layer stack comprises 23 Mamba-2 sequence-mixing layers, 23 granular MoE layers, and 6 Grouped-Query Attention layers. The MoE subsystem contains 128 routed experts, activates 6 routed experts per token, and includes 2 shared experts. Additional reported hyperparameters include model dimension 2688, 32 attention query heads, 2 attention KV heads, attention head dimension 128, Mamba-2 state dimension 128, Mamba-2 groups 8, Mamba-2 heads 64, Mamba-2 head dimension 64, and expert dimension 1856 (Soofi-Team et al., 10 Jul 2026).

Hyperparameter	Value
Total parameters	31.6B
Active parameters per token	3.2B (3.6B incl. embeddings)
Layers	52 (23 Mamba-2, 23 MoE, 6 GQA)
Routed experts	128
Activated experts per token	6
Shared experts	2

Several implementation choices define the model’s computational behavior. The MoE activation is squared ReLU, the router is a learned MLP with sigmoid gating, positional embeddings are absent, normalization is RMSNorm, and embedding/projection weights are untied. The hybrid structure matters operationally because only the 6 GQA layers maintain token-level attention state, while the Mamba-2 layers use a fixed-size recurrent state independent of context length. The paper reports an incremental attention-cache footprint of approximately 6 KB per token per sequence, or 11–53× smaller than comparable dense models (Soofi-Team et al., 10 Jul 2026).

This near-constant cache behavior is central to the model’s identity. In dense decoder-only systems, serving cost at long context is dominated by the growth of KV state. In Soofi S, the asymmetry between a small number of GQA layers and a larger number of state-space layers is used to decouple long-context serving from the usual cache explosion. A plausible implication is that the model is optimized not merely for benchmark efficiency but for high-concurrency inference regimes where memory pressure, not only FLOP count, is the main systems bottleneck.

3. Pretraining corpus and curriculum

The reported token budget is approximately 27 trillion tokens, with a precise consumed total of 26.68T tokens. Training is organized in three phases. Phase 1, described as diverse stable pretraining, uses approximately 20T tokens. Phase 2, described as high-quality annealing, uses approximately 6.58T tokens. Phase 3, a long-context extension stage, uses approximately 0.10T tokens and extends sequence training out to 1M tokens (Soofi-Team et al., 10 Jul 2026).

The corpus composition is notable for its explicit German up-sampling. In Phase 1, German accounts for 7.2% of tokens versus 5% in the reference. In Phase 2, German rises to 15.32%, described as more than 3× the reference. Phase 1 includes quality-tiered web data, English and German Wikipedia/PDFs, code, math, SFT/data-mix, and regional German sources such as FinePDFs, FineWiki, and HPLT v3. Phase 2 drops lower-tier web data and concentrates on code, math, STEM, SFT, reasoning, academic data, increased German content, German translation of ClimbMix via the KletterMix pipeline, and HPLT v4. Phase 3 blends high-quality document sources with SFT-formatted content, and all long-context tokens are tokenized with the model’s own tokenizer (Soofi-Team et al., 10 Jul 2026).

The curriculum is relevant for interpreting the evaluation profile. The model’s gains over the direct backbone reference are attributed in the paper to “its data and curriculum,” with improvements of +4.2 on the German aggregate, +15.1 on GLP-DE, and +6.7 on held-out English relative to Nemotron 3 Nano. Since Nemotron 3 Nano provides the reference design and a large open pretraining baseline, these deltas isolate the effect of Soofi S’s bilingual mixture design more cleanly than comparisons against unrelated dense baselines (NVIDIA et al., 23 Dec 2025).

The paper also emphasizes transparency at the data-accounting level. It states that all per-source counts, epochs, rationale, and a list of evaluated-but-excluded datasets are released. This is unusual within frontier-scale model reporting and situates Soofi S as a reproducibility-oriented training run rather than only a checkpoint release.

4. Evaluation profile and benchmark standing

The evaluation spans English and German code, mathematics, knowledge, reasoning, and question answering. English benchmarks include HumanEval, MBPP, LBPP, GSM8K, Minerva, MMLU, BBH, AGIEval, and NaturalQuestions. German benchmarks include HumanEval-DE, MBPP-DE, GSM8K-Platinum-DE, INCLUDE-DE, GPQA-Diamond-DE, GLP-DE, and ARC-Challenge-DE (Soofi-Team et al., 10 Jul 2026).

The headline result is not dominance over every larger model on every benchmark, but a specific Pareto position: Soofi S matches dense 14 to 27B models on aggregate English and German benchmarks while outperforming every European sovereign baseline in the comparison, including models far larger in active parameters. The abstract further states that among fully open models it obtains the highest English and German evaluation scores, ahead of Olmo 3 32B and Apertus 70B, and that it achieves the best code aggregates in both languages among 17 open base models (Soofi-Team et al., 10 Jul 2026).

Benchmark	Soofi S	Nemotron 3 N	Qwen3.5 35B	Ministral 3 14B	Gemma 3 27B
English agg.	70.1	68.3	74.6	70.3	70.3
German agg.	79.1	74.9	81.6	78.3	78.4
GLP-DE	88.8	73.7	94.0	89.5	88.3
HumanEval	73.8	72.2	67.1	60.9	60.2
HumanEval-DE	65.5	68.8	59.5	55.5	57.7

The code results are especially central to the model’s profile. The paper states that Soofi S achieves the best pass@1 performance among all open base models in both English and German code evaluation. This result is significant because it occurs in a bilingual base model rather than in a code-specialized checkpoint, suggesting that the curriculum did not trade off German competence against programming performance (Soofi-Team et al., 10 Jul 2026).

The reported limitations are also specific. Soofi S slightly trails the largest dense models in open-domain factual recall on NaturalQuestions and in German competition mathematics at the frontier. The paper also reports some decrease in long-context performance on rare aggregation-style tasks, specifically RULER common-word extraction beyond 32K. These caveats delimit the model’s claims: the strongest evidence is on aggregate bilingual evaluation and code, not on every specialized retrieval or competition-math regime.

5. Serving behavior and long-context efficiency

The model is explicitly optimized for long-context, high-concurrency deployment. Because only the 6 GQA layers keep token-level context and the Mamba-2 layers maintain fixed-size recurrent state, the paper reports that throughput stays “essentially flat” from 4K to 256K context, whereas dense models degrade rapidly as KV caches grow. At 40K context length and batch 32, Soofi S achieves 4.82k aggregate decode tokens/sec per GPU, reported as 9.2× higher than Ministral 3 14B and up to 9× higher than other dense 14–24B models. For prompt prefill, it reports 22.7s at 40K context, compared with 71.9s for Mistral 3 14B and 92.9s for Gemma 3 27B, while remaining the fastest system at 256K context in the comparison (Soofi-Team et al., 10 Jul 2026).

The serving analysis uses the following throughput definition:

$$\mathrm{TPS}_{\mathrm{decode}}^{\mathrm{agg/GPU}} = \frac{ B \left( O_{\mathrm{long}} - O_{\mathrm{short}} \right) } { N_{\mathrm{GPU}} \left[ t\left(O_{\mathrm{long}}\right) - t\left(O_{\mathrm{short}}\right) \right] }.$$

Here, $B = 32$, $O_{\mathrm{long}} = 1024$, $O_{\mathrm{short}} = 1$, $N_{\mathrm{GPU}} = 1$, and $t(\cdot)$ is measured batch latency. The paper states that this isolates per-GPU decode throughput and removes prefill cost (Soofi-Team et al., 10 Jul 2026).

This systems profile aligns with broader evidence from long-context serving work showing that large 30B-A3B models can become I/O- and cache-bound at extended context lengths. Unified KV pooling was proposed for Qwen3-30B-A3B precisely because long-context TTFT can otherwise rise to approximately 30.7 s at 128K tokens, far beyond typical serving targets (Kang et al., 10 Jun 2026). Soofi S addresses the same deployment regime through architecture rather than through KV offload orchestration. This suggests a complementary design philosophy: instead of optimizing the memory hierarchy around dense attention, Soofi S reduces the amount of attention state that must exist in the first place.

6. Openness, sovereignty, and release conditions

The model’s release policy is one of its defining research contributions. The planned release includes model weights, selected intermediate checkpoints on Hugging Face, full per-source data accounting with scripts, all hyperparameters, optimizer schedule, batch configuration, per-phase token budgets, and training and evaluation code under permissive OSI-compliant open-source licenses. For non-redistributable commercial sources such as the Genios corpus, the release provides aggregate statistics and mixture accounting rather than raw data (Soofi-Team et al., 10 Jul 2026).

The paper states that the release satisfies OSI’s Open Source AI Definition 1.0, with fewer than 1.3% of tokens from non-redistributable but documented commercial sources. It also states that the European open-data standard is not fully satisfied because of Genios, although 99% of tokens are reconstructible. This distinction is important: the model is presented not as an unrestricted data dump, but as a legally constrained yet auditable training artifact.

The sovereignty claim is therefore technical as well as institutional. Soofi S was built end-to-end on the German Industrial AI Cloud, a sovereign HPC-scale AI infrastructure in Munich. In practical terms, the paper treats sovereignty as a combination of infrastructure locality, permissive model release, exact data-mixture accounting, and the ability to reconstruct nearly the entire training corpus where source licenses permit (Soofi-Team et al., 10 Jul 2026).

Taken together, these characteristics position Soofi S 30B-A3B as a bilingual, efficient, and unusually transparent foundation model. Its main contributions are not only aggregate bilingual performance and strong code results, but also a concrete demonstration that a hybrid Mamba-Transformer MoE design can be paired with sovereign infrastructure and auditable data practices without relinquishing competitiveness against larger or denser open baselines.