- The paper combines dialect-specific LoRA fine-tuning, TIES-Merging, and ADI2-guided MBR decoding to balance dialect authenticity with translation accuracy.
- Experiments show that monolingual and translation adapters provide complementary benefits, while ADI2-based MBR raises Moroccan monolingual ADI2 from 0.38 to 0.51 but modestly reduces chrF++.
- The system achieves top shared-task ADI2 results for Syrian and Saudi Arabic and strong Syrian translation scores, although limited data, proxy-based evaluation, and costly decoding remain important constraints.
Overview
This paper describes the Maastricht University system for the AMIYA Shared Task on Dialectal Arabic (DA) generation and translation. The authors address the underrepresentation of Arabic dialects relative to Modern Standard Arabic (MSA) by adapting a pre-trained LLM through a three-part framework: parameter-efficient fine-tuning with LoRA on complementary supervision sources, TIES-Merging of task-specific adapters, and Minimum Bayes Risk (MBR) decoding with a dialect-aware scoring objective. The system was evaluated on Syrian, Moroccan, and Saudi Arabic in the closed data track, achieving the highest ADI2 scores among submissions for Syrian and Saudi Arabic and the highest chrF++ for English→DA and MSA→DA translation into Syrian.
Task setup and data
The AMIYA shared task evaluates monolingual dialect generation and cross-lingual translation using the AL-QASIDA benchmark. Automatic evaluation relies primarily on ADI2 — defined as the product of an ALDi-based dialectness score and a NADI 2024 dialect-identification confidence for the target class C — together with chrF++ for translation quality, supplemented by human judgments of fluency and instruction adherence.
The authors participate in the closed track on three of five dialects, combining two supervision types per dialect:
| Dialect |
Type |
Dataset |
Samples |
| Syrian |
Monolingual |
Shami Corpus |
25,136 |
| Syrian |
MT |
UFAL |
120,600 |
| Moroccan |
Monolingual |
DoDa |
10,000 |
| Moroccan |
MT |
DoDa |
10,000 |
| Saudi |
Monolingual |
SDC |
14,891 |
| Saudi |
MT |
SauDial |
1,000 |
The datasets are deliberately sub-sampled for computational efficiency; the authors note that more extensive data exists for most categories. This is a material constraint, particularly for Saudi Arabic, where only 1,000 parallel samples are used.
System description
LoRA fine-tuning. Two separate LoRA adapters are trained per dialect on JAIS-2 (8B). Monolingual adaptation uses raw dialectal sentences under a standard causal LM objective, targeting attention layers only, with gradient accumulation and checkpointing to fit memory constraints (max sequence length 512, 5 epochs, learning rate 3e-5, effective batch size 32, BF16 precision). Translation adaptation frames English↔Dialect conversion as instruction following in both directions, masking loss on instruction tokens so the model learns to follow rather than reproduce instructions.
Adapter merging. Because monolingual fine-tuning emphasizes surface-form authenticity while translation fine-tuning reinforces semantic grounding, the adapters are combined via TIES-Merging at the parameter level, preserving both supervision signals without joint training.
MBR decoding. At inference, N=20 candidates are sampled stochastically and reranked by ADI2, explicitly optimizing for dialectness during decoding.
Results
Backbone selection. A comparison between LLaMA 3.2 (3B) and JAIS-2 under identical training reveals a sharp trade-off on Syrian DA: LLaMA 3.2 attains a much higher ADI2 (0.78 vs. 0.33) but collapses on translation (chrF++ 0.14 vs. 0.43). Given the goal of reliability across both settings, JAIS-2 is selected despite its weaker monolingual dialect fidelity.
Fine-tuning and merging. On Moroccan DA, monolingual fine-tuning raises ADI2 from 0.18 to 0.44 while barely affecting chrF++; MT fine-tuning raises chrF++ to 0.42 but yields weaker ADI2 (0.26). TIES-Merging achieves the best chrF++ (0.44) and second-best ADI2 (0.38), confirming that the two supervision types are complementary and can be integrated without interference.
MBR objectives. The choice of MBR utility function is decisive. ADI2-scored MBR improves monolingual ADI2 from 0.38 to 0.51 and MT ADI2 from 0.27 to 0.36, at a modest chrF++ cost (0.44 → 0.40). In contrast, chrF++-optimized MBR drops monolingual ADI2 to 0.24 — below even standard decoding — indicating that character-overlap objectives bias generation toward standardized Arabic forms and erode dialectal distinctiveness. This is one of the paper's clearest findings: the reranking objective, not merely the use of MBR, determines whether dialect fidelity is gained or lost.
Official shared-task results
The final configuration — per-dialect LoRA fine-tuning, TIES-Merging, and ADI2-MBR decoding — produces the following automatic results:
| Dialect |
ADI2 |
DA→EN |
EN→DA |
DA→MSA |
MSA→DA |
| Moroccan |
0.679 |
49.93 |
30.02 |
39.53 |
33.77 |
| Syrian |
0.389 |
51.89 |
34.44 |
43.42 |
40.33 |
| Saudi |
0.464 |
0.03 |
19.82 |
37.21 |
24.23 |
Human evaluation favors Moroccan (adequacy 1.97, fluency 3.37), consistent with its higher ADI2. Among all teams, the system ranks first on ADI2 for Syrian and Saudi, first on EN→DA and MSA→DA chrF++ for Syrian, and first on human-evaluated fluency for Moroccan. The near-zero Saudi DA→EN chrF++ (0.03) is an anomalous result the paper does not explain, and it stands in tension with the claim of "strong generalization across dialects and task settings."
Limitations
The authors identify several constraints. First, ADI2 depends on automatic dialect identification, which may miss subtle or informal dialectal usage — meaning the MBR objective optimizes a proxy that may diverge from genuine authenticity. Second, training data are small and may not cover intra-dialect variation such as code-switching and colloquial registers. Third, inference cost is substantial: ADI2 scoring requires additional forward passes, and MBR over 20 candidates incurs a several-fold slowdown relative to standard decoding, limiting applicability in low-latency settings. Additionally, the paper leaves open whether joint multi-dialect training would outperform the per-dialect models actually submitted.
Conclusion
The paper demonstrates that separating supervision into monolingual and translation LoRA adapters, merging them with TIES, and applying ADI2-guided MBR decoding yields a better balance of dialectal fidelity and semantic accuracy than any single technique alone. The strongest empirical contributions are the quantified complementarity of the two supervision signals and the demonstration that MBR's utility function choice is decisive for dialect preservation. Open questions include scaling beyond sub-sampled data, handling code-switching, reducing MBR inference cost, and validating multi-dialect joint training.