A Sovereign, Open-Source Foundation Model for German and English

Abstract: We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English. Its hybrid design activates only 3B of 30B parameters per token and keeps the inference cache near-constant as context grows, giving it a decisive throughput advantage over dense models for long-context, high-concurrency deployment. Pretrained on roughly 27 trillion tokens with deliberately up-weighted German, Soofi S matches dense 14 to 27B models on aggregate English and German benchmarks while achieving the best code aggregates in both languages among 17 open base models, and outperforms every European sovereign baseline in our comparison, including ones far larger in active parameters. Among fully open models, Soofi S obtains the highest English and German evaluation scores, ahead of Olmo 3 32B and Apertus 70B. Soofi S was built end-to-end on the German Industrial AI Cloud, a sovereign HPC scale AI infrastructure operated by Deutsche Telekom in Munich. Soofi S will be released under highly permissive, open-access terms: weights, selected intermediate checkpoints, full per-source data accounting, hyperparameters, and training and evaluation code. Where source licenses permit, data-construction artifacts are released under permissive licenses; commercially licensed sources are documented with aggregate statistics and exact mixture accounting.

• The paper presents Soofi S, a transparent, data-auditable bilingual LLM that combines a high-capacity MoE architecture with a rigorous three-phase curriculum.
• It leverages a hybrid Mamba–Transformer design with efficient KV caching to maintain flat throughput across context sizes up to 256K tokens.
• Evaluation demonstrates robust improvements in coding, math, and language tasks, particularly strong in German benchmarks compared to other open models.

Detailed Summary and Analysis of "A Sovereign, Open-Source Foundation Model for German and English" (Soofi S) (2607.09424)

Introduction and Motivation

The Soofi S 30B-A3B model represents a significant open-science initiative to produce a high-capacity, fully auditable Mixture-of-Experts (MoE), hybrid Mamba–Transformer LLM optimized for both German and English. Motivated by notable gaps in existing LLM releases—namely, insufficient transparency in data/recipe artifacts, persistent English-centrism in multilingual training, and throughput inefficiencies for long-context/high-concurrency deployments—the authors build and release a model and artifact suite that address these issues head-on. This work is situated within the broader context of sovereign AI efforts in Europe, aiming to set open-access standards for reproducibility, auditability, and deployment cost.

Model Architecture and Training

Soofi S essentially inherits the Nemotron~3~Nano (A3B) reference as its backbone but is distinguished by its data and deployment recipe. This 52-layer network interleaves Mamba-2 sequence-mixing layers, granular MoE layers with shared experts, and sparsely distributed Grouped-Query Attention (GQA) layers. Critical architectural parameters (as detailed in the paper) yield a total of approximately 31.6B parameters, but only 3.2B are active per forward pass; embeddings included, this totals 3.6B per input. Only the GQA layers maintain a KV cache, enabling near-constant per-sequence state, which is vital for inference at long context lengths and with high request concurrency.

The model is optimized under an extensive three-phase curriculum: stable broad pretraining, high-quality annealing, and a long-context extension to 1M tokens. Training is conducted exclusively on the German Industrial AI Cloud, achieving 253,000 GPU hours using B200 hardware and benefitting from high-throughput, energy-efficient infrastructure.

Figure 1: Training dynamics over the full $\sim27$T-token run, showing loss, gradient norm, and throughput across curriculum phases.

Data and Pretraining Regime

Perhaps the most distinguishing aspect is the rigor and transparency applied to the data curation and reporting. Full per-source accounting is provided, including sources evaluated but excluded, epoch multipliers, and effective token counts. Soofi S is explicitly bilingual, with deliberate up-weighting of German: $7.2\%$ of Phase~1, rising to $15.3\%$ in annealing. Data is sourced through extensive open datasets (Nemotron CC, Dolma, HPLT, FinePDFs etc.), complemented with curated code/math/synthetic corpora and a minority component of machine-translated and synthetic German to mitigate the imbalance in high-quality native German data.

Figure 2: Flow diagram visualizing token mixture evolution across three training phases; note the increasing density and skill-orientation in later stages.

The careful balance, reuse, and severe documentation of data blending, including for long-context (e.g., length-bucketed upsampling), is a strong claim to reproducibility and scientific validity, with open-source status nearly complete but for a small, documented non-redistributable (Genios) component.

Evaluation: Capability and Efficiency

Soofi S is benchmarked against 16 major open and open-weight models under a harmonized harness, with a particular focus on both capability (code, math, knowledge, reasoning, language proficiency) and real-world deployment metrics (tokens per second under high context/concurrency).

Capability

Strong claims are made regarding Soofi S's performance:

• English and German Aggregates: Outperforms all other fully open models on both aggregates, e.g., +2.8 English aggregate points over Olmo~3~32B, +6.3 German aggregate over Apertus~70B.
• Code and Math: Surpasses competitors on HumanEval/MBPP metrics in both languages, with leading or tightly competitive math scores (e.g., +10.8 over the next best open source on HumanEval).
• Knowledge/Reasoning: Sits at the frontier in MMLU-STEM, GPQA-Diamond, AGIEval, BBH etc., matching or exceeding large, dense international models, thanks to the high-density reasoning and SFT data in Phase~2.
• German Regional/Technical Benchmarks: Marks the largest margins for Soofi S, e.g., INCLUDE-DE, GLP-DE, and a comprehensive German aggregate.

  Figure 3: Evaluation overview situating Soofi S against major open-source competitors across aggregates and code/math subdomains.

  

  Figure 4: Code generation (pass@1) statistics highlight Soofi S’s leadership on HumanEval and MBPP in both English and German.

  

  Figure 5: Mathematics benchmark results underline top-tier GSM8K and Minerva performance, especially strong in German grade-school math.

  

  Figure 6: Knowledge and reasoning aggregates with leading or tied-best performance in key benchmark families.

  

  Figure 7: Soofi S achieves unmatched scores on every major German evaluation dimension vs. open-source models.

The open-weight comparison (e.g., Qwen3.5~35B-A3B, Ministral~3~14B) reveals that Soofi S trades within a narrow performance band of much larger models but at dramatically reduced active parameter and inference cost, with especially strong effects isolated via direct comparison with its architectural reference, Nemotron~3~Nano:

Figure 8: Score deltas vs. architecture-identical Nemotron baselines, isolating the effect of the German–English recipe; major improvements on German metrics with no loss on English aggregate.

Efficiency

The architectural design confers a highly advantageous inference profile. By minimizing the KV cache (only 6 layers, low-per-token cache) and leveraging Mamba-2’s recurrent state, Soofi S achieves:

• Up to $9.2\times$ higher aggregate decode TPS/GPU compared with dense 14–24B models at 40K context (batch size 32).
• Throughput remains essentially flat across context sizes up to 256K, unlike dense baselines.
• Sustained single-GPU prompt prefill and first-token latency (TTFT) are consistently lower than major dense models.

Figure 9: Soofi S maintains high measured aggregate decode TPS (tokens per second) as context increases, distinctly outperforming dense baselines.

Long-Context Generalization

While the model achieves near parity with Nemotron~3~Nano on most long-context retrieval tasks, it exhibits regression in common-word extraction beyond 32K tokens, attributed to data blend rather than backbone limitations.

Figure 10: RULER long-context accuracy; Soofi S is strongly competitive except for the CWE subtask beyond 32K, which is flagged as a target for data-side improvement.

Implications and Future Directions

Practical Impact: The Soofi S release offers immediate, open-weight and data-auditable deployment for organizations requiring sovereignty, data reproducibility, and high-concurrency, long-context inference—particularly critical for regulatory, industrial, and public-sector use in Germany and Europe.

Theoretical Advancement: The separation of architecture and data/curriculum enables clear attribution of gains, providing a template for future controlled studies in data-centric LLM pretraining. It demonstrates that architecture sharing (with Nemotron~3~Nano) coupled with a high-quality, highly transparent bilingual recipe can yield substantial capability gains in the target low-resource language without trade-offs in majority language performance.

Standard for Openness: By publicizing not only weights but also intermediate checkpoints, full data accounting, recipes, and rejected/excluded datasets, Soofi S raises the standard for open-source LLMs and provides a blueprint for reproducible base-model research, addressing calls for robust open standards in European AI policy.

Limitations and Open Problems: The model’s performance plateau in German competition-level math and minor regression in open-domain factual recall at scale are acknowledged. Both are attributed to capacity/data trade-offs and suggest targets for post-training/interventional work or alternative retrieval-augmented deployments.

Future Work: Directions include modular reasoning variants, enhanced post-training (SFT/RL), further scaling of high-quality German resources, and expansion of evaluation suites.

Conclusion

Soofi S 30B-A3B sets a new benchmark in sovereign, bilingual, fully auditable LLM pretraining—balancing per-token parameter efficiency, strong numerical performance on German/English technical and general tasks, and unprecedented data and training process transparency. It demonstrates that a focused, targeted data curriculum and open infrastructure can bring sovereign European models to parity with the leading global LLMs in their parameter and cost class. This work provides a replicable template for the creation of high-capability, fully open foundation models for other languages and contexts.