AceGPT: Culturally Aligned Arabic LLM

• AceGPT is an open-source LLM built on a decoder-only Transformer architecture, featuring dual-language pretraining for Arabic and English.
• It employs supervised fine-tuning and reinforcement learning with AI feedback to achieve nuanced cultural alignment and robust instruction-following.
• The model maintains strong performance with 8-bit quantization while highlighting challenges in aggressive pruning and cross-domain transfer.

AceGPT is an open-source LLM family grounded in a decoder-only Transformer architecture, with a focus on cultural, linguistic, and application-specific adaptation for Arabic and multilingual environments. Developed as a derivative of Meta’s LLaMA 2, AceGPT is available in at least two principal configurations (7B and 13B parameters), demonstrating leading performance on benchmarks of Arabic instruction-following, cultural value alignment, and multilingual reasoning. The design comprises targeted pretraining, extensive supervised fine-tuning in Arabic, and reinforcement learning using AI and human preference signals to produce models tailored for nuanced, context-aware responses in diverse settings (Huang et al., 2023, Ghazawi et al., 27 Jan 2025, Alshehhi et al., 25 Jul 2025).

1. Model Design and Pretraining Paradigm

AceGPT retains the architectural backbone of LLaMA 2, with no alterations to transformer depth, hidden size, or attention head count in either the 7B or 13B parameter variants—32/40 layers and hidden dimensions of ≈4 096 (7B) and ≈5 120 (13B) respectively (Huang et al., 2023). The original LLaMA 2 SentencePiece tokenizer (≈32 000 tokens) is leveraged; no native expansion to include specialized vocabulary was performed in initial releases. Pretraining employs a mixed-language corpus:

• Arabic Core: ~19.2B tokens (7B) / ~6B tokens (13B) from sources such as Arabic Text 2022, Arabic Wikipedia, CC100, and OSCAR.
• English Supplement: ~10.8B tokens (7B) / ~4B tokens (13B) chosen to preserve cross-lingual knowledge and mitigate catastrophic forgetting.

The pretraining objective is classical causal language modeling: $$L(\theta) = -\sum_{t=1}^T \log P_\theta(w_t \mid w_{<t})$$

Context length is set to 2 048 tokens. Optimization utilizes AdamW with cosine decay schedule and standard weight decay. This dual-language pretraining provides broad coverage of Arabic morphology, syntax, and idiomatic usage as well as robust generalization to English and multilingual settings (Huang et al., 2023, Alshehhi et al., 25 Jul 2025).

2. Localization and Cultural Alignment

The distinguishing feature of AceGPT is its multi-stage Arabic adaptation pipeline:

• Supervised Fine-Tuning (SFT). Native Arabic user queries (e.g., 43K Quora-Arabic instructions) receive GPT-4–authored Arabic responses. Additional instructional data (Alpaca, Evol-Instruct, Code-Alpaca) is translated to Arabic and answers regenerated with GPT-4 for fluency and relevance. All data are mixed (~630K examples), with Arabic-native instances up-sampled by 3× (Huang et al., 2023).
• Prompt Format. System prompts follow the LLaMA 2 “INST” framing for stable instruction parsing.
• Reinforcement Learning with AI Feedback (RLAIF). A preference-labeled corpus is built by sampling SFT-generated outputs and scoring pairwise preferences via GPT-4. This yields ~12K Arabic and ~12K English preference pairs. A reward model (initialized from Ziya-LLaMA) is trained with binary-logistic loss: $$\mathcal{L}_{\mathrm{RM}}(\phi) = -\mathbb{E}_{(x, y_c, y_r) \sim D}[\,\log \sigma \bigl(r_\phi(x, y_c) - r_\phi(x, y_r)\bigr)]$$ Proximal Policy Optimization (PPO) then tunes the policy under the reward signal and a KL regularization penalty.
• Cultural Value Benchmarks. The Arabic Cultural & Value Alignment (ACVA) set (≈8K yes/no queries, 2.5K clean) is used both for performance evaluation and as additional RL training signal.

This localized alignment yields marked improvement in Arabic knowledge, sensitivity to regional context, and avoidance of English-centric or culturally incongruent completions (Huang et al., 2023).

3. Compressed Model Variants: Quantization and Pruning

AceGPT’s deployability in resource-constrained settings is demonstrated via systematic study of post-training quantization and unstructured pruning (Alshehhi et al., 25 Jul 2025):

• Quantization. Both 4-bit and 8-bit weight quantization are implemented post-training:

$$q = \mathrm{round}\left(\frac{w}{S}\right) + Z;\quad \hat{w} = S\,(q-Z)$$

where $S$ is layer-wise scale and $Z$ is zero-point offset. Model accuracy is largely invariant under 8-bit compression and degrades modestly under 4-bit.

• Magnitude Pruning. Unstructured L1 pruning removes the $s$\% of smallest-magnitude weights per linear layer:

$w_{ij} = 0 \text{ if } |w_{ij}| < \tau$

with $\tau$ chosen by desired sparsity (20%, 40%, 80%).

Empirically, 8-bit quantization results in $<1$ percentage point (pp) loss on EnglishMMLU, ArabicMMLU, and Kannada benchmarks; pruning induces rapid collapse at $\geq 40$% sparsity, especially in the 13B variant used for high-reasoning tasks.

4. Multilingual and Task-Specific Benchmarking

AceGPT undergoes comparative benchmarking on EnglishMMLU (15.9K questions, 57 subjects), ArabicMMLU (14.6K questions, 40 subjects), and IndicBench (Kannada ARC-C-2.5K) (Alshehhi et al., 25 Jul 2025). Notable results:

Model	EnglishMMLU FP	ArabicMMLU FP	Kannada FP
AceGPT-13B	47.74	40.27	22.31
BLOOMZ-7.1B	44.7	41.7	34.4
LLaMA2-13B	30.6	29.1	14.2
Jais-13B	24.8	37.3	11.9

AceGPT-13B leads on EnglishMMLU; comparable performance is observed on ArabicMMLU, though BLOOMZ outperforms AceGPT by ~1.4 pp. Multilingual foundation models (BLOOMZ, XGLM) outperform AceGPT in low-resource Indic (Kannada) settings by 10–12 pp, highlighting limitations in cross-lingual transfer without explicit adaptation. Under compression, AceGPT’s accuracy remains strong up to 8-bit quantization but drops precipitously with $$\mathcal{L}_{\mathrm{RM}}(\phi) = -\mathbb{E}_{(x, y_c, y_r) \sim D}[\,\log \sigma \bigl(r_\phi(x, y_c) - r_\phi(x, y_r)\bigr)]$$0 pruning—especially in Arabic and English reasoning tasks (Alshehhi et al., 25 Jul 2025).

In the task of Arabic automated essay scoring (AES) using the AR-AES dataset, AceGPT-13B achieves a Quadratic Weighted Kappa (QWK) of 0.67 after fine-tuning, outperforming other open LLMs but not surpassing a BERT-based specialist (QWK = 0.88) (Ghazawi et al., 27 Jan 2025). Custom tokenization, including an Arabic SentencePiece vocabulary, reduces sequence length (from ~1,532 to ~410 tokens), improving both throughput and prediction consistency.

5. Cultural, Linguistic, and Technical Adaptation

AceGPT integrates Arabic-centric design in multiple technical facets:

• Tokenization: Although the default LLaMA tokenizer covers basic Arabic, character-level OOV mapping causes bloated sequence lengths and raises GPU memory demands. Integrating a custom, AR-AES-trained SentencePiece tokenizer dramatically reduces token count and computational cost while providing slight but consistent gains in QWK for AES tasks (Ghazawi et al., 27 Jan 2025).
• Prompting: Mixed-language prompting (English instructions + Arabic questions/rubrics + Arabic reference answers) outperforms mono-lingual setups, increasing zero-shot AES QWK from 0.20 (zero-shot) to 0.55 (few-shot) and fine-tuning to 0.67 (Ghazawi et al., 27 Jan 2025).
• Instruction Following and Cultural Response: RLHF with an Arabic reward model produces completions aligned with local social and religious context, as evidenced by higher ACVA F1 scores (AceGPT-13B: 75.0% on “all set,” approaching GPT-3.5 Turbo’s 75.55%) (Huang et al., 2023).

6. Error Profile, Limitations, and Future Directions

AceGPT displays robust instruction-following and cultural response generation, but several limitations persist:

• Error and Hallucination: ACEGPT can fabricate details (hallucinations), particularly when under-trained or after aggressive pruning. Confidence levels remain misleadingly high even when accuracy collapses under heavy pruning, representing a safety risk for deployment in low-resource languages (Alshehhi et al., 25 Jul 2025).
• Domain Variance: Performance varies across task domains and course contexts, indicating incomplete domain transfer and the need for more granular adaptation (Ghazawi et al., 27 Jan 2025).
• Comparison with Smaller Specialists: AraBERT (BERT-based) models outperform AceGPT on AES with limited task-specific data, though AceGPT’s open LLM performance remains state-of-the-art for generative models in Arabic (Ghazawi et al., 27 Jan 2025).
• Tokenization Coverage: Initial releases do not expand coverage for Arabic dialects, named entities, or morphological variants; this impacts both memory efficiency and performance stability (Huang et al., 2023).

Ongoing work centers on: extending tokenization for dialectal and named-entity coverage, scaling Arabic-only pretraining, integrating adversarial safety/misinformation assessment, and further enriching culturally grounded RL datasets (Huang et al., 2023).

7. Practical Recommendations for Deployment

Empirical findings suggest the following deployment guidelines:

• Default to 8-bit quantization for most applications as it preserves accuracy (loss $$\mathcal{L}_{\mathrm{RM}}(\phi) = -\mathbb{E}_{(x, y_c, y_r) \sim D}[\,\log \sigma \bigl(r_\phi(x, y_c) - r_\phi(x, y_r)\bigr)]$$11 pp) while reducing resource requirements.
• Limit unstructured pruning to ≤40% sparsity to maintain model reliability; more aggressive sparsity induces catastrophic performance drops.
• For high-fidelity usage in low-resource languages, supplement instruction-tuned LLMs with synthetic or back-translated corpora and dialectal data augmentation.
• Monitor not only accuracy but also the calibration of output confidence, especially after compression, to guard against error-aversion failure modes in critical language tasks.
• For Arabic-specific downstream tasks, employ Arabic-centric Reward Model RLHF, custom tokenization, and domain-specific fine-tuning as indicated by the variation in zero/few-shot vs task-adapted performance (Alshehhi et al., 25 Jul 2025, Ghazawi et al., 27 Jan 2025, Huang et al., 2023).

AceGPT thus represents the first end-to-end open-source pipeline for localizing LLMs in Arabic, demonstrating peak instruction-following and value alignment among generative LLMs for Arabic and strong baseline performance in multilingual evaluation. Its limitations in low-resource and domain transfer scenarios demarcate the frontiers for further research in culturally adapted, compressed, and specialized LLMs.