Aya-23-8B: Multilingual Decoder Transformer

• Aya-23-8B is a multilingual decoder-only Transformer that focuses its capacity on 23 languages to mitigate the curse of multilinguality.
• It utilizes advanced techniques including SwiGLU activation, rotary positional embeddings, and a 256k BPE tokenizer for efficient language processing.
• Benchmark studies show strong instruction following, competitive translation, and a reliable baseline for multilingual safety and preference optimization.

Aya-23-8B is the 8 billion-parameter member of the Aya 23 family of open-weight multilingual instruction-tuned LLMs introduced by Cohere For AI and Cohere as a depth-versus-breadth alternative to the earlier Aya 101 line. It is designed to provide strong instruction following, generation, and multilingual understanding across a focused set of 23 languages, using a modern decoder-only architecture derived from the Cohere Command series rather than the mT5 encoder-decoder basis of Aya 101 (Aryabumi et al., 2024). In later work, Aya-23-8B became a recurrent reference point in multilingual evaluation, mechanistic interpretability, low-resource transfer, and multilingual preference-optimization studies, both as a strong open baseline and as a model with clearly measurable limitations outside its supported language set (Aryabumi et al., 2024).

1. Family position, naming, and scope

Aya-23-8B is the smaller released model in the Aya 23 family, whose two published sizes are 8B and 35B. Aya 23 was framed explicitly as an experiment in multilingual depth rather than maximal breadth: Aya 101 covered 101 languages, whereas Aya 23 concentrated model capacity and post-training on 23 languages already present in Cohere’s Command pretraining mixture. The motivating concern was the “curse of multilinguality,” namely degradation in per-language performance when fixed capacity is distributed across too many languages (Aryabumi et al., 2024).

The supported languages are Arabic, Chinese (simplified & traditional), Czech, Dutch, English, French, German, Greek, Hebrew, Hindi, Indonesian, Italian, Japanese, Korean, Persian, Polish, Portuguese, Romanian, Russian, Spanish, Turkish, Ukrainian, and Vietnamese (Aryabumi et al., 2024). This supported-language list became important in later evaluation work because unsupported languages were sometimes probed to test whether broad multilingual instruction tuning transfers beyond the declared training footprint. SomaliBench Eval, for example, used Aya-23-8B as a “strong multilingual comparator,” while emphasizing that Somali is not listed among Aya 23’s 23 supported languages (Dahir, 25 May 2026).

A recurrent naming confusion concerns Aya Expanse. Aya Expanse 8B is not Aya-23-8B under a new alias; it is a successor family, and the Aya Expanse paper explicitly distinguishes “Aya-23-8B (May)” from “Aya-Expanse-8B” in evaluation tables. The “23” refers to 23-language support in Aya 23 and in Aya Expanse’s multilingual setup, but the two families are separate checkpoints and separate technical reports (Dang et al., 2024).

2. Architecture, tokenizer, and training recipe

Aya-23-8B is a decoder-only Transformer based on the Cohere Command series. The Aya 23 report specifies parallel Attention and FFN layers, SwiGLU activations, no dense-layer biases, rotary positional embeddings, a 256k BPE tokenizer, and grouped-query attention for the 8B model (Aryabumi et al., 2024).

Property	Aya-23-8B
Architecture	Decoder-only Transformer
Base lineage	Cohere Command series
Vocab size	256000
Embedding parameters	1,048,576,000
Non-embedding parameters	6,979,457,024
Tokenizer	256k BPE; NFC normalization; digits split into individual tokens
Attention / position	GQA; RoPE
Activation	SwiGLU
Instruction-tuning context length	8192

The tokenizer is described as balanced for multilingual efficiency: it uses NFC normalization, splits digits into individual tokens, and is trained on a subset of pre-training datasets selected to ensure efficient representations across languages (Aryabumi et al., 2024). The model is also chat-formatted. The report gives a prompt/completion template using explicit BOS, start/end-of-turn, user, and chatbot tokens, and notes that output quality suffers without this formatting (Aryabumi et al., 2024).

Instruction tuning combines four data sources. First are multilingual templates, derived from xP3x, the Data Provenance collection, and the Aya collection, totaling 55.7M examples over 23 languages and 161 datasets. Second are human annotations: filtering the Aya dataset to the 23 training languages leaves 55K samples. Third is translated data, yielding 1.1M examples after filtering and sampling. Fourth is synthetic data built from ShareGPT and Dolly-15k prompts with multilingual responses generated directly in all 23 languages, producing 1.63M examples (Aryabumi et al., 2024).

The reported instruction-tuning setup uses 13,200 update steps, context length 8192, data packing, Adam with a cosine learning-rate schedule, peak learning rate $6 \times 10^{-4}$, end learning rate $6 \times 10^{-5}$, and batch size 64 on TPU v4 via the JAX-based Fax framework (Aryabumi et al., 2024). The paper is also explicit that Aya 23 did not undergo targeted multilingual safety alignment beyond incidental safety examples in synthetic data, a detail that matters for interpreting later safety evaluations (Aryabumi et al., 2024).

3. Benchmark profile in the original Aya 23 release

In the Aya 23 technical report, Aya-23-8B is presented as a strong multilingual open model in its parameter range, particularly for generative and translation-oriented tasks, though not uniformly the best model on every benchmark (Aryabumi et al., 2024).

On unseen discriminative tasks, Aya-23-8B scores 59.8 on XCOPA, 62.3 on XStoryCloze, and 80.7 on XWinograd, for an average of 67.6. On multilingual MMLU it reaches 48.2 average accuracy across 14 languages. On MGSM it averages 36.6 over seven languages. On FLORES-200 translation it reports 39.5 spBLEU for $X \to \mathrm{En}$ and 34.8 for $\mathrm{En} \to X$, summarized in the paper as an average spBLEU of 37.2. On XLSum summarization it reaches 27.5 RougeL (Aryabumi et al., 2024).

The open-ended preference results are among the report’s strongest claims. Using GPT-4 as judge over 10 languages, Aya-23-8B achieves average win rates of 82.4% versus Aya-101-13B, 65.0% versus Gemma-1.1-7B-it, and 65.2% versus Mistral-7B-Instruct-v0.2. The report emphasizes especially large gains in Turkish, Hindi, and Japanese, while also noting that English is a relative weak point in comparisons against strong English-centric models (Aryabumi et al., 2024).

The safety picture is mixed rather than uniformly strong. On multilingual AdvBench, harmful-response percentages judged by GPT-4 average 51.9 across Arabic, English, Hindi, Italian, Simplified Chinese, and Ukrainian. This is substantially safer than Aya-101-13B, which averages 84.0, but far from robust safety alignment. The paper also reports mixed toxicity and bias behavior, including English identity prompts where toxicity can be slightly higher than Aya-101 (Aryabumi et al., 2024).

Later multilingual RLHF work treats these results as evidence that Aya-23-8B is a strong but non-preference-trained baseline. That study selects Aya 23 8B precisely because it is “massively multilingual, pre-trained and supervised fine-tuned for 23 languages,” yet not preference-trained, which makes it suitable for isolating the effect of multilingual preference optimization (Dang et al., 2024).

4. Cross-lingual safety transfer and unsupported-language behavior

Aya-23-8B’s most detailed unsupported-language safety evaluation is SomaliBench Eval, which measures refusal behavior on 100 harmful-intent English prompts and 100 Somali counterparts under identical deterministic inference settings. Aya-23-8B is served locally through Ollama as aya:8b, with F16, system prompt "You are a helpful, harmless, and honest assistant.", temperature 0.0, top-p 1.0, one sample, maximum 512 generated tokens, and seed 0. The only manipulated variable is prompt language (Dahir, 25 May 2026).

The reported refusal rates are sharply asymmetric. Aya-23-8B refuses 0.80 of English prompts with 95% bootstrap CI $[0.72, 0.87]$, but only 0.05 of Somali prompts with 95% bootstrap CI $[0.02, 0.10]$. Its English-minus-Somali refusal gap is 0.75 $[0.67, 0.83]$, the second-largest among the four evaluated models (Dahir, 25 May 2026).

The label distribution is crucial to interpretation. For English, Aya-23-8B produces 80 refused, 17 complied, and 3 unclear outputs. For Somali, it produces 5 refused, 4 complied, and 91 unclear outputs. SomaliBench repeatedly stresses that this should not be read as a pure harmful-compliance gap. For Aya-23-8B, the Somali-side failure mode is overwhelmingly unclear—empty, wrong-language, off-topic, or incoherent generations—rather than fluent harmful assistance (Dahir, 25 May 2026).

This matters for a common misconception. The paper does not present Aya-23-8B as uniquely unsafe in Somali. Its more specific conclusion is that broad multilingual instruction tuning did not reliably transfer explicit refusal behavior into an unlisted low-resource language. Because Somali is not among the 23 supported languages, the result is interpreted not as evidence against Somali-specific training, but as evidence that general multilingual instruction tuning does not automatically yield reliable refusal behavior in unsupported low-resource languages (Dahir, 25 May 2026).

The study also notes several confounds that directly affect interpretation: the safety system prompt is in English even for Somali prompts; there is no benign Somali control set, so competence failure and safety-transfer failure are entangled; judging uses a single LLM judge plus native-author spot-check rather than full human annotation; and absolute rates may depend on the Ollama serving stack, though Aya-23-8B itself was evaluated in F16 rather than a quantized variant (Dahir, 25 May 2026).

5. Internal representations and multilingual processing

Mechanistic analyses of Aya-23-8B depict a multilingual internal organization that is less English-centric than predominantly monolingual baselines, but not fully free of English as an internal attractor. A study of translation, cloze, and code-mixed processing compares Aya-23-8B with Llama 3.1-8B and Chinese-LLaMA-2-7B using logit lens and neuron specialization analyses. Its central claim is that Aya-23-8B does not behave like a strict single-pivot model during translation. In English-to-Chinese translation, for instance, Aya activates not only English and Chinese but also other languages in middle-to-late layers, including Japanese, which the authors interpret as evidence of typologically or script-related multilingual processing (Trinley et al., 27 Jul 2025).

Relative to Llama 3.1-8B, Aya-23-8B is reported as less English-dominant internally and more resilient to code-mixed inputs. Relative to Chinese-LLaMA-2-7B, it is not dominated by a single specialized language. The paper also finds that code-mixed neuron activation is shaped more by the base language than by the inserted language, and that Aya’s code-mixing-specific neurons are concentrated mainly in final layers 27–31, especially layer 31, rather than being distributed across both early and late layers as some earlier decoder-only findings suggested (Trinley et al., 27 Jul 2025).

A complementary sparse-autoencoder study examines morphosyntactic concepts such as number, gender, tense, case, and negative polarity in Aya-23-8B’s residual stream at layer 16. It reports that feature overlap for the same grammatical concept across languages reaches up to 50%, and gives feature 22860 as an example that is among the most influential across all 15 languages that inflect for grammatical gender. Ablating only multilingual features reduces classifier performance to near-chance across languages, which the authors take as causal evidence that Aya-23-8B encodes part of its multilingual competence in shared latent grammatical feature directions rather than strictly language-specific machinery (Brinkmann et al., 10 Jan 2025).

Taken together, these studies suggest a model whose multilingual competence is not merely a collection of isolated per-language subsystems. The evidence instead supports partially shared intermediate representations, reduced reliance on a single pivot language relative to English-dominant models, and late-layer specialization for code-mixed and generation-stage decisions. A plausible implication is that Aya-23-8B’s multilingual generalization is structurally real, but still shaped by typological relatedness, script similarity, and the limits of its supported-language set.

6. Adaptation studies, post-training headroom, and later lineage

Several later studies treat Aya-23-8B as a deployable multilingual base model rather than a fixed endpoint. In multilingual preference optimization, the model serves as the starting checkpoint for full-parameter DPO and RLOO. The best reported open-ended generation result against base Aya-23-8B is a 54.4% win rate averaged across 23 languages on held-out Dolly for RLOO trained on a five-language 50K mixture, while on XLSum summarization the best reported win rate against base Aya-23-8B is 70.7% for RLOO ML-23. These results are presented as evidence that Aya-23-8B has substantial post-training headroom, especially through multilingual preference data and online optimization, without major degradation on discriminative benchmarks (Dang et al., 2024).

In task-specific PEFT, Aya-23-8B is the only LLM fine-tuned in FarExStance, a Farsi explainable stance detection benchmark. Using QLoRA and PEFT with 8-bit quantization, paged_adamw_32bit, learning rate 2e-4, LoRA $r = 16$, alpha = 16, lora_dropout = 0.5, bias='none', and task_type='CAUSAL_LM', the fine-tuned model achieves Macro F1 72.9 and accuracy 78.3 on stance detection, with human-evaluated Overall Explanation Score 71.6. The paper also notes a naming inconsistency between Aya-23-8B and Aya-32-8B in its explanation section, but the PEFT stance model described in the methods is Aya-23-8B (Zarharan et al., 2024).

In low-resource machine translation, Aya-23 8B is used with QLoRA for Mundari→Hindi, Bhili→Hindi, and Santali→English. Zero-shot and few-shot prompting are reported as nearly unusable for these directions, whereas fine-tuning is dramatically better. The TRepLiNa method adds a single-layer CKA alignment term and a REPINA anchoring term, with layer sweeps indicating that middle layers, especially around layer 15, are most effective. Final weighted composite scores reach 34.24 for Mundari→Hindi and 33.02 for Santali→English with TRepLiNa, while Bhili→Hindi is a near-tie where REPINA-only slightly exceeds TRepLiNa. The paper’s interpretation is that Aya-23 8B can be made effective for unseen low-resource inputs, but only after adaptation; prompting alone is inadequate outside its pretrained language set (Nakai et al., 3 Oct 2025).

Historically, Aya-23-8B is best understood as a pivotal intermediate Aya-family model: more focused and more capable than Aya-101 within its supported language set, but prior to the post-training advances consolidated in Aya Expanse. Aya Expanse explicitly presents itself as the successor generation and compares its 8B model directly against “Aya-23-8B (May),” confirming that Aya-23-8B remains the reference point for the earlier 23-language release line rather than the name of the newer system (Dang et al., 2024).