Tiny Aya: Compact Multilingual Models

• Tiny Aya is a family of compact multilingual models designed to balance performance across 70+ languages through targeted data design and regional specialization.
• It employs a dense decoder-only Transformer with interleaved sliding-window and full-attention layers, augmented by Grouped-Query Attention for efficient inference.
• Region-aware posttraining and merging techniques enhance translation quality and safety, enabling practical deployment on resource-constrained devices.

Tiny Aya is a family of compact, open-weight multilingual LLMs designed to “bridge scale and multilingual depth” by prioritizing balanced performance across 70 languages in a small footprint (Salamanca et al., 12 Mar 2026). With 3.35B parameters, the family includes a pretrained foundation model, a globally balanced instruction-tuned variant, and three region-specialized models produced through region-aware posttraining and merging. The reported objectives are state-of-the-art translation quality, competitive multilingual understanding and reasoning, and high-quality target-language generation, alongside an “alternative scaling path” centered on efficiency, balanced performance across languages, and practical deployment rather than parameter count alone (Salamanca et al., 12 Mar 2026).

1. Definition, scope, and model family

Tiny Aya is presented as a family of compact multilingual models rather than a single checkpoint. The release includes Tiny Aya Base, described as a pretrained foundation model for 70+ languages; Tiny Aya Global, described as instruction-tuned and optimized for balanced multilingual performance; and three region-specialized variants: Tiny Aya Earth, strongest for Africa and West Asia; Tiny Aya Fire, strongest for South Asia; and Tiny Aya Water, strongest for Asia-Pacific and Europe (Salamanca et al., 12 Mar 2026).

The central design goal is balanced multilingual performance at small scale. The paper frames this as a data-centric strategy that reduces disparities between high-resource and low-resource languages without brute-force scaling. It also emphasizes region-aware posttraining and merging, comprehensive multilingual evaluation, and an efficient deployment path via architecture choices and quantization (Salamanca et al., 12 Mar 2026).

A concise view of the released variants is as follows.

Variant	Role	Reported strength
Tiny Aya Base	Pretrained foundation model	70+ languages
Tiny Aya Global	Instruction-tuned	Balanced multilingual performance
Tiny Aya Earth	Region-specialized	Africa and West Asia
Tiny Aya Fire	Region-specialized	South Asia
Tiny Aya Water	Region-specialized	Asia-Pacific and Europe

This model family is explicitly positioned against the common small-model failure mode sometimes termed the “curse of multilinguality,” where limited capacity exacerbates interference across languages. The reported response is not aggressive scaling, but targeted balancing, regional specialization, and merge-based preservation of global instruction-following and safety (Salamanca et al., 12 Mar 2026).

2. Architecture and tokenizer design

Tiny Aya uses a dense decoder-only Transformer with parallel Transformer blocks for training efficiency (Salamanca et al., 12 Mar 2026). Its attention stack is interleaved: sliding-window and full-attention layers are used at a 3:1 ratio. The positional encoding scheme is correspondingly mixed, with RoPE in sliding-window layers and NoPE in full-attention layers. The model also uses Grouped-Query Attention with 16 query heads and 4 KV heads to reduce inference memory footprint (Salamanca et al., 12 Mar 2026).

The MLP uses SwiGLU, and all dense-layer biases are removed for stability and efficiency. The reported sequence geometry is a sliding window of 4096 tokens, with input context length 8192 tokens and output context length 8192 tokens. From the architecture table, the model has 36 layers, hidden size 2048, FFN hidden dimensions 11008, vocabulary size 262k, and total parameters 3.35B, split into 0.5B embedding and 2.8B non-embedding parameters (Salamanca et al., 12 Mar 2026).

The tokenizer is a single massively multilingual tokenizer shared across all Tiny Aya models. It is trained with balanced weighting over languages and language-bucket groupings, with vocabulary size 262k and GPT-4o regex used for pre-tokenization with no normalization. The weighting for language $i$ is reported as

$$w_i = \frac{w^d_i \cdot w^b_i}{\sum_n (w^d_n \cdot w^b_n)}.$$

The exact tokenizer algorithm type is not explicitly reported. The paper nevertheless reports lower tokens/character than contemporaries across most scripts, with especially strong tokenization efficiency on under-served scripts such as Khmer, Telugu, Gujarati, Lao, and Ge’ez (Salamanca et al., 12 Mar 2026).

These architectural choices are tied directly to the deployment narrative. Interleaved attention, GQA, parallel blocks, and mixed-precision training are presented not as isolated optimizations but as the systems substrate for small-scale multilingual performance under realistic memory and throughput constraints (Salamanca et al., 12 Mar 2026).

3. Pretraining strategy and multilingual balancing

The foundation model is trained on 70 languages plus programming languages, with code included for broader reasoning and transfer benefits (Salamanca et al., 12 Mar 2026). The pretraining corpus is drawn from a large set of public and proprietary sources, and the tokenizer is trained on a 50 GB FineWeb-2 sample using the multilingual weighting described above. The pretraining mixture is described as extensively filtered by language ID and stopword filtering, heuristic cleaning, deduplication, domain classification, and quality filtering (Salamanca et al., 12 Mar 2026).

Balancing is a central part of the recipe. The paper reports deliberate upweighting of low-resource languages through language buckets defined by family or script, with the stated purpose of equilibrating coverage. A cooldown phase in the middle of training upsamples the highest-quality datasets and includes instruction-style data across all 70 languages to improve generalization. The training objective is the standard causal language modeling objective implied by the decoder-only setup, although the explicit loss expression and perplexity are not reported (Salamanca et al., 12 Mar 2026).

The reported pretraining budget is 6T tokens, with a Warmup-Stable-Decay learning-rate schedule. Training uses 256 NVIDIA H100 GPUs in Fax, a JAX-based framework, and mixed FP8/BF16/FP32 precision. The optimizer type, batch size, gradient clipping, weight decay, steps or epochs, and exact FLOPs or GPU-days are not reported (Salamanca et al., 12 Mar 2026).

A plausible implication is that Tiny Aya’s multilingual balance is intended to emerge from data mixture design at least as much as from architecture. The paper’s emphasis on balanced weighting, family/script buckets, and cooldown mixture suggests that cross-lingual parity is treated as a training distribution problem rather than only a capacity problem (Salamanca et al., 12 Mar 2026).

4. Region-aware posttraining, synthetic data, and merging

Tiny Aya’s posttraining pipeline is built around region-aware specialization. The stated motivation is that small models face the “curse of multilinguality,” and that specialization over regional clusters can mitigate interference while preserving a shared multilingual base (Salamanca et al., 12 Mar 2026). The targeted regions are Europe, West Asia, South Asia, Asia-Pacific, and Africa, each associated with specific language groups listed in the report.

The posttraining data mixture combines internal and public datasets and is extended to missing languages through translation, prompt transformations, and synthetic completions. Translation systems include command-a-translate and deepseek-v3, with system selection performed per language using reference-free quality estimators xCOMET-XL and AfriCOMET. Prompt-level transformations are used to counter translationese and English-centric framings; the named transformations are Naturalness, Cultural Adaptation, and Difficulty Enhancement, produced with Command A or DeepSeek-V3 depending on language (Salamanca et al., 12 Mar 2026).

Synthetic completions are generated via Fusion-of-N, abbreviated FusioN. The teachers are Gemma3-27B-It, Command A, and DeepSeek-V3, and the fusor or judge is Command A. The report states that the fusor aggregates the strongest candidate components, improving quality and cross-lingual robustness. In addition, the machine translation finetuning set is formed from public parallel corpora filtered by rules, FastText LID, QE filtering, and difficulty filtering with Sentinel-25-src, followed by backtranslation to 23 languages; the final MT finetuning set contains 312k parallel documents across 98 languages (Salamanca et al., 12 Mar 2026).

Each regional SFT run uses 16 H100 GPUs and completes within 24 hours, with 3 epochs, cosine decay learning rate, peak $2.5 \times 10^{-5}$, final $1.2 \times 10^{-6}$, and global batch size 32. Tiny Aya Global additionally receives a minimal preference tuning phase to establish identity and supported language list and to maintain multilingual safety, though the method or objective is not specified (Salamanca et al., 12 Mar 2026).

To preserve global instruction-following and safety while retaining regional gains, the paper applies SimMerge to merge each region-specialized SFT checkpoint with the global SFT checkpoint. Linear interpolation, SLERP, and TIES are tried with varying mixing strengths. Selection is based on task-agnostic similarity features computed on a held-out unlabeled multilingual probe of approximately 10k tokens per language, with final choice determined by regional development performance and safety checks (Salamanca et al., 12 Mar 2026).

A common concern with specialization is that it degrades global behavior or safety. The reported result is the opposite: region-specialized models improve translation in their target regions, and merging raises multilingual safe-response behavior across variants (Salamanca et al., 12 Mar 2026).

5. Evaluation framework and reported performance

Tiny Aya is evaluated on a broad multilingual suite covering translation, open-ended generation, mathematical reasoning, discriminative knowledge and commonsense tasks, safety, and cultural awareness (Salamanca et al., 12 Mar 2026). Translation uses Flores on 66 focus languages and WMT24++ on 55 languages, both reported with ChrF. Open-ended generation uses rubric-based absolute ratings from an LLM judge on mDolly and m-ArenaHard-v2.1. Mathematical reasoning uses GlobalMGSM. Discriminative tasks include Global MMLU, INCLUDE, and Global PIQA. Safety uses MultiJail and XSTest, while cultural awareness uses NormAd and BLEnD SQA (Salamanca et al., 12 Mar 2026).

The headline metrics reported for Tiny Aya Global are:

Evaluation area	Metric	Reported result
Flores	ChrF	43.5 ± 14.0
WMT24++	ChrF	46.0 ± 10.4
mDolly	Judge score	86.9 (6.2)
m-ArenaHard	Judge score	67.4 (6.1)
GlobalMGSM	Accuracy	52.8
Global MMLU	Accuracy	44.9 ± 7.3
INCLUDE	Accuracy	45.1 ± 11.1
Global PIQA	Accuracy	68.3 ± 10.6
MultiJail	Safe response rate	min 87.0%, mean 91.1%

On translation, Tiny Aya Global is reported to score highest on average relative to same-scale multilingual baselines and to outperform Gemma3-4B on 46 of 55 WMT24++ languages. Against TranslateGemma-4B, it is described as on par on average, winning 27 of 66 focus languages on Flores and 12 of 55 on WMT24++ (Salamanca et al., 12 Mar 2026).

On open-ended generation, the reported mDolly score is 86.9 with standard deviation 6.2, and the paper highlights the low variance across languages. Language fidelity, measured with FastText LPR, is reported as 97.4 on mDolly, 91.4 on m-ArenaHard, and 96.9 on GlobalMGSM, with the stated conclusion that Tiny Aya tends to produce the requested language more reliably than competitors by up to 3–6% on average (Salamanca et al., 12 Mar 2026).

On mathematical reasoning, Tiny Aya Global averages 52.8 on GlobalMGSM and is reported as strongest on African languages with an average of 39.2%, compared with 17.6% for Gemma3-4B, 6.25% for Qwen3-4B, and 20.9% for Qwen3.5-4B. On discriminative knowledge and understanding, the paper places Tiny Aya within the range of 3–4B competitors (Salamanca et al., 12 Mar 2026).

On safety, Tiny Aya Global reports the highest MultiJail safe-response mean at 91.1% with low disparities across languages. The report contrasts this with high invalid rates in some competitors on Swahili and states that Tiny Aya maintains 94% safe on Swahili. On XSTest in English, Tiny Aya Global reports 10.4% over-refusal and 15.5% under-refusal (Salamanca et al., 12 Mar 2026).

For cultural awareness, a Tiny Aya variant is reported as best in 39 of 75 countries on NormAd, with gains strongest for Fire and Global in West Asia, Asia-Pacific, and the Americas. BLEnD SQA reports gains in 8 of 16 regions over Gemma3-4B, notably in Nigeria, West Java, Greece, Ethiopia, and Spain. The paper also notes that prompt-language sensitivity is larger in Africa and South Asia than in Europe or the Americas (Salamanca et al., 12 Mar 2026).

6. Balanced multilinguality, efficiency, and deployment

A defining property of Tiny Aya is low variance across languages. The paper repeatedly emphasizes balanced multilingual behavior rather than simply average performance. On generative tasks, Tiny Aya is reported to have much smaller standard deviations than comparable models, and its quality remains relatively stable even in languages with low web presence, whereas competitors are described as exhibiting steeper drops (Salamanca et al., 12 Mar 2026).

The deployment argument is equally central. Tiny Aya supports llama.cpp quantization formats q4_0, q4_k_m, and q8_0, and is reported to work with MLX and llama.cpp. The Q4_K_M format yields an approximate footprint of 2.14 GB. Reported throughput is about 10 tokens/s decoding on an iPhone 13 and about 32 tokens/s on an iPhone 17 Pro for a 100-input/100-output workload. The unquantized model is reported to run out of memory on many devices (Salamanca et al., 12 Mar 2026).

The quality cost of quantization is characterized as modest. On mDolly, Q8_0 is reported as negligible, Q4_K_M as a decrease of 1.4 points on average, and Q4_0 as a decrease of 2.1 points on average. The degradation is described as modest even for low-web-presence languages (Salamanca et al., 12 Mar 2026).

These results underpin the paper’s “alternative scaling path.” Rather than expanding parameter count, Tiny Aya combines efficient architecture, balanced multilingual data design, region-aware specialization, and merge-based retention of global safety and instruction-following. This suggests a systems view in which quality, parity, and deployability are optimized jointly rather than sequentially (Salamanca et al., 12 Mar 2026).

7. Limitations, caveats, and relation to adjacent work

The report identifies several limitations. Some tasks still perform better when prompted in English, and cultural and commonsense performance shows higher sensitivity to prompt language in Africa and South Asia. The Americas cluster is not defined beyond major languages, and no indigenous languages from the Americas are covered (Salamanca et al., 12 Mar 2026).

Evaluation caveats are also explicit. Machine-translated benchmarks may introduce noise, although the pipeline attempts mitigation through transformation and quality checks. Open-ended evaluations use LLM judges, and the paper acknowledges known judge biases even though rubric-based scoring is used instead of win-rates (Salamanca et al., 12 Mar 2026).

Safety is described as dependent on exposure to safe training pairs in each language. The reported mitigation is the combination of a global model merge and FusioN with a safety-aware judge. Several implementation details remain unreported, including the exact tokenizer algorithm, the explicit causal LM loss expression and perplexity, optimizer type, weight decay, gradient clipping, pretraining batch size and steps, and the exact preference-learning objective (Salamanca et al., 12 Mar 2026).

A separate line of work connects Tiny Aya to function-calling adaptation. “STAR: Similarity-guided Teacher-Assisted Refinement for Super-Tiny Function Calling Models” describes a practical recipe to adapt STAR to a “Tiny Aya sub-1B function-calling model,” recommending that a Tiny Aya-style student keep Aya’s tokenizer and vocabulary and use a decoder-only Transformer with RoPE, RMSNorm, and gated activation, while pairing Constrained Knowledge Distillation with Similarity-guided RL (Ni et al., 3 Feb 2026). This does not redefine Tiny Aya itself, whose released family is 3.35B, but it indicates that Tiny Aya’s multilingual design is already being treated as a viable substrate for downstream specialization in adjacent research (Ni et al., 3 Feb 2026).

Taken together, Tiny Aya is best understood as a compact multilingual modeling program built around three linked theses: that balanced multilinguality can be pursued through data design rather than brute-force scale, that region-aware specialization can improve local performance without sacrificing global behavior when combined with merging, and that practical deployment constraints should shape model design from the outset (Salamanca et al., 12 Mar 2026).