Aya Expanse: Multilingual Transformer Models

• Aya Expanse is a family of large-scale multilingual language models that use transformer architectures with advanced mechanisms like SwiGLU and grouped-query attention.
• It employs innovative training methods including data arbitrage, iterative multilingual preference training, and model merging to enhance performance and equitable language support.
• The framework enables neuron-level language control and fairness adjustments, demonstrating state-of-the-art performance across diverse benchmarks and applications.

Aya Expanse refers to a family of large-scale open-weight multilingual LLMs, along with a lineage of associated research breakthroughs, applications, and analyses. Developed by Cohere For AI and Cohere, the Aya Expanse models target robust multilingual capabilities at both the modeling and interpretability levels, with demonstrated state-of-the-art performance across public benchmarks and recent advancements in model control, fairness, and fine-tuning strategies. Aya Expanse also designates the broader research ecosystem involving technical recipes for multilingual model training, neuron-level analysis and manipulation, and downstream task adaptations.

1. Model Architecture and Technical Foundations

Aya Expanse comprises 8B and 32B parameter transformer models derived from the Cohere Command series. Both models utilize modern transformer components such as SwiGLU activation, rotary positional embeddings (RoPE), and grouped-query attention. The 8B model supports a context window of 8K tokens, while the 32B model is designed for 128K-token contexts, facilitating extended document processing. Grouped-query attention improves global context modeling, crucial for multilingual settings.

The architecture’s configuration supports efficient scaling—at 32B parameters, model merging (see Section 2) provides up to 3× the improvement compared to the 8B variant. Key architectural highlights are summarized below:

Model Variant	Parameters	Max Context Length	Core Innovations
Aya Expanse 8B	8B	8K tokens	SwiGLU, RoPE, grouped-query attention
Aya Expanse32B	32B	128K tokens	Same as above, enhanced model merging gains

This architecture is inherently designed for multilingual training spanning 23 languages, prioritizing parity and performance across typologically diverse linguistic groups (Dang et al., 5 Dec 2024).

2. Training Innovations: Data Arbitrage, Preference Alignment, and Model Merging

Aya Expanse models synthesize key technical breakthroughs for multilingual AI:

• Data Arbitrage: Instead of single-model supervision, completions are sampled from multiple teacher models per prompt. These are ranked by an internal "Arbiter" (a reward model), selecting the best. This method enhances data quality particularly across low-resource languages by diversifying and optimizing synthetic supervision.
• Iterative Multilingual Preference Training: Post-supervised tuning, models undergo two-stage preference training—an offline stage using pre-scored completions and an online stage via Direct Preference Optimization (DPO), which iteratively samples and ranks outputs in many languages. This sequence yielded, for example, an additional 7.1% win-rate boost for the 8B variant over the baseline.
• Model Merging: Separate checkpoints are trained per language family and later merged using weighted linear averaging (which outperformed alternatives like SLERP and TIES). This allows leveraging cross-lingual transfer and diversity. Gains scale with model size, reaching up to 3× improvement at 32B parameters.

Formally, merged model weights can be denoted:

$$W_\mathrm{merged} = \alpha W_\mathrm{model1} + (1 - \alpha) W_\mathrm{model2}$$

where $\alpha$ is optimized based on cross-lingual metrics.

3. Performance Evaluation and Benchmarking

Aya Expanse models have been extensively benchmarked on multilingual datasets:

• m-ArenaHard (translations of Arena-Hard-Auto into 23 languages): Aya Expanse 8B achieves win rates up to 70.6%. At 32B scale, win rates reach 76.6%, with performance exceeding that of Llama 3.1 70B (54.0% win rate vs. a model with double the parameters).
• Dolly Evaluation Set: 8B: 83.9% win rate vs. Llama-3.1 8B; 32B: 89.9% vs. Mixtral 8x22B, with training/evaluation distribution similarities contributing to higher scores.
• Academic benchmarks: Significant improvements over predecessor Aya 23 (16% higher language understanding, 1.8× mathematical reasoning accuracy).

Pairwise win rates across the 23 languages serve as primary metrics, demonstrating not only overall improvement but also equitable per-language advances. Aya Expanse substantially reduces historic performance disparities between high- and low-resource languages (Dang et al., 5 Dec 2024).

4. Language Representation and Control: Neuronal and Ideological Steering

Recent research identifies and exploits internal mechanisms of multilingual representation in Aya Expanse:

• Language-Specific Neurons: Employing the Language Activation Probability Entropy (LAPE) method, specific neurons—predominantly in deeper layers—are shown to be highly selective for individual languages. Non-Latin scripts recruit a greater share of such specialized neurons.
• Language Arithmetics: Additive and multiplicative interventions on these neurons can steer the model toward or away from producing text in a specific language. For instance, setting activations for the target language and suppressing others yields measurable gains across tasks such as translation (up to 10% BLEU improvement), question answering, and NLI.
• Cross-Lingual Neuron Steering: Overlapping neurons among related languages allow cross-lingual interventions to benefit low-resource or typologically similar languages.
• Fallback Mechanisms: Deactivating dominant language neurons reveals a robust fallback hierarchy: Aya Expanse automatically shifts to secondary languages, showcasing internal redundancy.

The public release of code for neuron identification and steering supports reproduction and further paper (Gurgurov et al., 30 Jul 2025).

5. Fairness, Bias, and Evaluability

Aya Expanse incorporates explicit design and evaluation for demographic and ideological neutrality:

• Demographic Neutrality: Studies confirm that, despite statistically significant behavioral differences in emotional valence across age and gender, individual adaptive learning within the model ensures no correlation of prediction performance (F1 scores) with these demographics, supporting broad applicability and fairness (Henriques et al., 2020).
• Political Orientation: Large-scale assessments using the Political Compass Test reveal that as model size increases (from 8B to 32B), Aya Expanse’s responses increasingly cluster in the libertarian–left quadrant, a scaling effect also seen in other models. However, language-specific deviations persist (e.g., Turkish, Persian), and response compliance varies across languages.
• Ideological Steering: The center-of-mass activation intervention (modifying output of attention heads using class-averaged activation differences) enables shifting model political stances at inference time—demonstrated for LLaMA-3.1-8B and, by implication, applicable to Aya Expanse architectures (Gurgurov et al., 30 Jul 2025).

6. Practical Applications and Open Resources

Aya Expanse models and datasets are released as open-weights, enabling further research and deployment:

• Content Generation & Translation: The model excels at equitable content generation, translation, educational, and technical domains, with consistent high performance across languages.
• Grammatical Error Correction (GEC): The 8B model, fine-tuned with MultiGEC, WikiEdits, and Reddit-MultiGEC data (using LoRA), attains state-of-the-art results for paragraph-level GEC in 11 languages. Fine-tuning strategies include incremental ablation, low-rank adaptation, and ablation studies. Gains are substantial for underrepresented languages, with standardized evaluation using GLEU and ERRANT F₀.5 metrics (Kovalchuk et al., 18 Sep 2025).
• Specialized Task Adaptation: For EvaCun 2025, Aya Expanse was fine-tuned using QLoRA under several masked token prediction paradigms, demonstrating adaptability to specialized and low-resource tasks, albeit with clear trade-offs among prompt formulations and decoding strategies (Jon et al., 17 Oct 2025).

Both the models and critical evaluation datasets (e.g., m-ArenaHard) are available via Hugging Face; task-specific tools and neuron control scripts are provided in associated repositories.

7. Future Prospects and Research Directions

Ongoing and plausible future advancements include:

• Refinement of Preference Training: Sustained optimization of alignment strategies, especially for low-resource and non-Latin languages, could further homogenize model performance across the linguistic spectrum.
• Expanded Model Merging Techniques: Continued experimentation with model merging and data arbitrage may improve representations while reducing compute.
• Expanded Control and Interpretability: Enhancement of neuron-level and attention-head interventions may yield greater reliability in language and ideological steering, informing both safe deployment and interpretability.
• Broader Applications: The robust, demographically neutral modeling approach supports broader adoption in research, education, and cross-cultural communication, and provides a foundation for investigating translational tasks extending to under-documented scripts and modalities.

Aya Expanse thus represents a comprehensive framework for multilingual modeling, characterized by architectural innovation, a principled approach to data and alignment, commitment to openness and fairness, and a vibrant research ecosystem spanning performance, control, and downstream adaptation.