Llama-GENBA-10B: German-English-Bavarian Model

• Llama-GENBA-10B is a trilingual foundation model for German, English, and Bavarian, combating English-centric bias by integrating low-resource Bavarian data into a balanced multilingual framework.
• The model scales Llama 3.1-8B to 10B parameters via block-expansion, adding eight zero-initialized Transformer layers for stability and efficient capacity growth.
• A novel trilingual evaluation protocol with translated benchmarks and controlled language ratios demonstrates superior Bavarian performance and competitive accuracy in English and German.

Llama-GENBA-10B is a trilingual foundation model for German, English, and Bavarian that was introduced to address English-centric bias in LLMs while targeting the German NLP community and promoting Bavarian as a low-resource language (Hoffmann et al., 6 Sep 2025). It is built on Llama 3.1-8B, scaled to approximately 10 billion parameters, and continuously pretrained on a corpus comprising 82 billion English tokens, 82 billion German tokens, and 80 million Bavarian tokens. The model combines corpus balancing, tokenizer extension, block-expansion-based scaling, and a trilingual evaluation protocol that includes Bavarian benchmark translations. In the reported evaluation, the fine-tuned variant surpasses Apertus-8B-2509 and gemma-2-9b in Bavarian, outperforms EuroLLM in English, and matches EuroLLM in German, while training on a single Cerebras CS-2 with documented energy use (Hoffmann et al., 6 Sep 2025).

1. Scope and research problem

Llama-GENBA-10B is framed as a response to English-centric bias in foundation models and as an effort to construct a multilingual system in which German and English are balanced while Bavarian is explicitly represented despite its low-resource status (Hoffmann et al., 6 Sep 2025). The model’s stated audience is the German NLP community, but its design also elevates Bavarian from a marginal dialectal presence to a directly modeled language variety within a single foundation model.

The development process is organized around four challenges: curating a multilingual corpus despite Bavarian scarcity, creating a unified tokenizer for English, German, and Bavarian, optimizing architecture and language-ratio hyperparameters for cross-lingual transfer, and establishing the first standardized trilingual evaluation suite by translating German benchmarks into Bavarian (Hoffmann et al., 6 Sep 2025). These choices place the model at the intersection of multilingual pretraining, low-resource language modeling, and benchmark construction.

A plausible implication is that the project is not limited to model release alone. It also proposes a reproducible workflow for integrating a low-resource variety into a larger multilingual system without allowing a high-resource language to dominate the optimization trajectory.

2. Architecture and parameter expansion

The base checkpoint is Llama 3.1-8B, which is scaled to 10 billion parameters via block-expansion by adding eight new Transformer blocks (Hoffmann et al., 6 Sep 2025). After expansion, the model has $L=40$ layers, consisting of the original 32 plus 8 added layers, with $H=32$ attention heads and the hidden dimension $d$ unchanged from Llama 3.1-8B. Total parameters are reported as $P\approx10\times10^9$, corresponding to 20% growth.

The added blocks are zero-initialized in a manner intended to preserve backbone stability. Specifically, the newly inserted blocks’ FFN output and attention-projection weights are initialized to zero so that they can learn without disturbing the pretrained layers (Hoffmann et al., 6 Sep 2025). The paper summarizes the scaling relation as

$$P_{\rm new}=P_{\rm base}\times1.2,\quad L_{\rm new}=L_{\rm base}+8.$$

This scaling strategy is presented as a way to increase capacity starting from an existing checkpoint rather than training an entirely new model from scratch. In the authors’ broader recommendations, architecture scaling by block-expansion is described as a method for increasing capacity without full retraining (Hoffmann et al., 6 Sep 2025).

3. Corpus construction and language balancing

The final corpus composition is explicitly balanced between English and German, with Bavarian included at much smaller absolute scale but with targeted upsampling (Hoffmann et al., 6 Sep 2025).

Language	Dataset	Tokens
English	EnglishDataset	$82\times10^9$
German	GermanDataset	$82\times10^9$
Bavarian	BavarianDataset	$80\times10^6$

English data are drawn from Knowledge Pile, Cosmopedia, UltraTextbooks-2.0, and Proof-Pile 2. German data come from occiglot-fineweb-v1.0, with 110.96 billion raw tokens truncated to 82 billion. Bavarian raw data amount to approximately 20 million tokens and are up-sampled to 80 million (Hoffmann et al., 6 Sep 2025).

The Bavarian sub-corpus is assembled from multiple small sources: Uni Leipzig Wikipedia at 100k tokens, OPUS Wikimatrix at 579k, OPUS XLEnt at 19k, OPUS Wikimedia at 55k, OPUS Tatoeba at 0.6k, and Filtered FineWeb at 262k sentences (Hoffmann et al., 6 Sep 2025). The scarcity of Bavarian material is therefore an explicit design constraint rather than an incidental limitation.

The balancing strategy truncates English and German to equal shares of 82 billion tokens each, upsamples Bavarian by a factor of 4, and introduces Bavarian only in the last 10% of training in order to avoid overwhelming low-resource data (Hoffmann et al., 6 Sep 2025). This is a central methodological choice: corpus balance is treated not merely as static sampling, but as schedule design over training time. A common misconception would be to read the model as simply “trilingual from the start”; the reported procedure instead delays Bavarian exposure until late-stage training.

4. Tokenization, language ratios, and training schedule

Tokenizer design is treated as a distinct engineering problem. The base tokenizer is the Llama-3.1-8B vocabulary with 128,256 tokens, and the trilingual system extends it via BPE in three variants corresponding to 10%, 20%, and 30% vocabulary growth (Hoffmann et al., 6 Sep 2025). The reported fertility scores are 1.9026 for the 10% expansion, 1.8372 for the 20% expansion, and 1.8214 for the 30% expansion, where fertility is defined as the average number of tokens per word and lower values are better.

On that basis, the selected tokenizer is the 20% expansion, corresponding to approximately 153,856 tokens (Hoffmann et al., 6 Sep 2025). The paper does not report explicit OOV rates, but it states that fertility improvement indicates good coverage of German umlauts and Bavarian subwords. Tokenizer choice is thus tied directly to measurable segmentation behavior rather than to vocabulary size alone.

Cross-lingual optimization is similarly empirical. Language-mix experiments with a 16 billion token budget compare a balanced 1:1 English/German split, corresponding to 8 billion and 8 billion tokens, against a 9:1 split, corresponding to 1.6 billion and 14.4 billion tokens. The balanced 1:1 mix is reported to outperform the 9:1 setting for both English and German accuracy (Hoffmann et al., 6 Sep 2025). For the first 90% of training steps, the language mixture is summarized as

$$\alpha_{\rm EN}=\alpha_{\rm DE}=0.5,\quad \alpha_{\rm BA}=0,$$

and for the final 10% as

$$\alpha_{\rm EN}=\frac{82}{82+82+0.08},\;\alpha_{\rm DE}=\frac{82}{82+82+0.08},\; \alpha_{\rm BA}=\frac{0.08}{82+82+0.08}.$$

The pretraining schedule uses 41,707 steps, a batch size of $H=32$0 tokens, a warmup of 417 steps to $H=32$1, and cosine decay to $H=32$2 (Hoffmann et al., 6 Sep 2025). Taken together, these choices indicate that multilingual performance is not attributed to scale alone; it depends on vocabulary adaptation, mixture tuning, and staged data introduction.

5. Trilingual evaluation methodology

The evaluation framework is a trilingual suite covering English, German, and Bavarian, with the Bavarian component created by translating German benchmarks using Gemini-flash (Hoffmann et al., 6 Sep 2025). The task set is the same across all three languages: ARC-Challenge, HellaSwag, MMLU, TruthfulQA, and WinoGrande, each measured by accuracy.

Standardization is achieved using the EleutherAI harness for English, its German adaptation for German, and Bavarian translations that follow the same format (Hoffmann et al., 6 Sep 2025). The paper presents this as the first standardized trilingual evaluation suite for the three-language setting under consideration.

Methodologically, the Bavarian benchmark layer is important because it converts existing German tasks into a comparable trilingual protocol rather than requiring a wholly new benchmark ecosystem. At the same time, the evaluation should be interpreted in light of its construction: Bavarian results are reported on translated benchmarks rather than on independently collected native Bavarian benchmarks. That point does not negate the evaluation, but it clarifies what kind of evidence the reported scores provide.

6. Reported performance, efficiency, and broader methodological significance

For the base model, the paper reports strong middle-tier English performance close to top 7–9 billion parameter models, German performance competitive with mid-tier European models and close to EuroLLM-9B, and Bavarian performance in which the base model ranks fourth among baselines while outperforming EuroLLM-9B and Qwen2.5-7B (Hoffmann et al., 6 Sep 2025). In the Table 5 excerpt, Llama-GENBA-10B-base records accuracies of 0.3776 on ARC, 0.4792 on HellaSwag, 0.4631 on MMLU, 0.3729 on TruthfulQA, and 0.7364 on WinoGrande.

For the instruction-tuned model, the reported comparison is more favorable in Bavarian. Llama-GENBA-10B-instruct outperforms Apertus-8B-instruct-2509 and gemma-2-9B-it on Bavarian, outperforms EuroLLM-9B-instruct in English, and matches it in German (Hoffmann et al., 6 Sep 2025). The abstract further states that the fine-tuned variant establishes itself as the best model in its class for Bavarian.

The efficiency profile is explicitly documented. Pretraining is performed on a single Cerebras CS-2 with 850k cores and 40 GB SRAM, with an average power draw of 22.3 kW, a training duration of 66 days, and total energy consumption of 35.23 MWh (Hoffmann et al., 6 Sep 2025). The energy estimate is given by

$H=32$3

The paper concludes with a blueprint for inclusive foundation models consisting of six recommendations: corpus balance by limiting high-resource languages to equal shares and upsampling low-resource data late; dialect handling through a unified tokenizer and fertility-based BPE extension; architecture scaling through block-expansion; language ratio tuning through explicit mixing-ratio experiments; benchmark creation by translating existing tasks into the target low-resource variant; and efficiency tracking through single-chip acceleration with recorded energy and power use (Hoffmann et al., 6 Sep 2025). This suggests that Llama-GENBA-10B is intended not only as a trilingual model release, but also as a documented recipe for resource-efficient multilingual pretraining in settings where one of the target languages is severely under-resourced.