Peacock: Arabic-Centric Multimodal LLMs

• Peacock is a family of Arabic multimodal large language models that integrate vision encoders with Arabic LLM decoders using a translation-and-filtering pipeline.
• The system features two backbone variants (InstructBlip and LLaVA) paired with two Arabic decoders (AceGPT-7B-chat and AraLLaMA-7B) to assess architecture and language impacts.
• Peacock is evaluated on standard visual reasoning benchmarks and the culturally-aware Henna dataset, demonstrating improved performance on Arabic cultural and dialect-specific tasks.

Peacock is a family of Arabic multimodal LLMs (MLLMs) introduced as a systematic effort to build and evaluate Arabic-centric multimodal systems with strong vision and language capabilities. The family comprises four variants formed by combining two multimodal backbones—InstructBlip-style and LLaVA-style—with two Arabic LLM decoders, AceGPT-7B and AraLLaMA-7B. The same work also introduces Henna, an Arabic-culture-aware benchmark for evaluating images depicting Arabic food, monuments, events, and customs across 11 Arab countries, thereby coupling model development with culturally specific evaluation (Alwajih et al., 2024).

1. Positioning within Arabic multimodal modeling

Multimodal LLMs had already shown effectiveness on tasks requiring complex reasoning and linguistic comprehension, but the reported motivation for Peacock is that high-quality multimodal resources were concentrated in English. This imbalance was identified as a central obstacle to developing comparable systems for languages such as Arabic, despite Arabic’s large speaker population (Alwajih et al., 2024).

Within that setting, Peacock is framed as the first systematic effort to build and evaluate Arabic-centric multimodal LLMs. The project is not limited to transferring an English-centric architecture into Arabic. It combines Arabic decoders, an Arabic translation-and-filtering pipeline for image-text supervision, standard zero-shot evaluation on Arabic benchmark translations, and a benchmark explicitly designed for Arabic cultural content. This suggests that the work treats Arabic multimodality as a joint problem of linguistic coverage, visual reasoning, and cultural grounding rather than as a narrow localization exercise.

2. Architecture and model variants

Peacock contains two backbone families. The InstructBlip variant uses a ViT/G-14 vision encoder pre-trained on ImageNet, with patch size 14, and feeds output features into a Q-Former with 12 Transformer layers and learnable query vectors as in BLIP-2. The LLaVA variant uses a CLIP-Large image encoder with $336\times336$ inputs that produce 576 visual tokens, followed by a two-layer MLP projection. The language decoders are AceGPT-7B-chat, described as a LLaMA2-7B fine-tuned for Arabic dialogue, and AraLLaMA-7B, described as an LLaMA2-7B further pre-trained on a large Arabic corpus and instruction-tuned in Arabic (Alwajih et al., 2024).

In the alignment modules, the InstructBlip-Peacock configuration keeps the image encoder and Q-Former frozen and trains only the final linear projection head and LoRA adapters in the LLM. The LLaVA-Peacock configuration keeps the image encoder frozen and trains only the two-layer MLP and LoRA adapters in the LLM. The resulting four variants are the following.

Variant	Vision-side design	Arabic decoder
IB-AraLLaMA	InstructBlip + Q-Former	AraLLaMA-7B
IB-AceGPT	InstructBlip + Q-Former	AceGPT-7B-chat
LLaVA-AraLLaMA	CLIP-Large + 2-layer MLP	AraLLaMA-7B
LLaVA-AceGPT	CLIP-Large + 2-layer MLP	AceGPT-7B-chat

This design isolates two orthogonal choices: the multimodal backbone and the Arabic decoder. A plausible implication is that the benchmark results can be read not only as end-to-end system comparisons but also as evidence about how decoder quality and alignment strategy interact in Arabic multimodal transfer.

3. Data construction, preprocessing, and optimization

The training pipeline begins with English multimodal resources and converts them into Arabic through translation and filtering. The source data are COCO-2017, with 118K images and 5 English captions each, and LCS-558K, consisting of web-scraped English image-caption pairs. The captions are translated into Arabic using Google Cloud Translate. To filter poor translations, the pipeline computes LaBSE multilingual embedding similarity and keeps only pairs satisfying $\cos(\mathrm{emb}_{en}, \mathrm{emb}_{ar}) > 0.80$, dropping approximately 20% of translations (Alwajih et al., 2024).

Alignment pretraining uses approximately 916K filtered Arabic image-caption pairs from COCO and LCS. The objective is standard next-token cross-entropy over response tokens $\{y_t\}$, conditioned on visual features $V$ and prompt tokens $x$, with image caption instructions prepended:

$$L_{\mathrm{pre}} = - \sum_{t \in Y_{\mathrm{resp}}} \log P(y_t \mid y_{<t}, x, V)$$

During this stage, only the projection head—linear in the InstructBlip-style model or MLP in the LLaVA-style model—is trained, while both the backbone and the LLM are frozen. Visual instruction fine-tuning then uses three translated sources: the LLaVA-150K multimodal instruction dataset, translated VQAv2 reformatted into instruction-response format, and 60K multiple-choice items from the LLaVA1.5 instruction mix. In this stage, the trainable components are the projection head and LoRA adapters in the LLM, plus the MLP in the LLaVA variant, with the same next-token cross-entropy objective applied to response tokens only (Alwajih et al., 2024).

The pipeline is notable for its economy. Rather than collecting a native Arabic multimodal corpus from scratch, it translates existing English resources and then introduces a semantic filter intended to remove low-quality translations. This suggests a practical recipe for multilingual MLLM construction when native multimodal supervision is scarce, though the paper also reports that translation artifacts remain a limitation.

4. Performance on standard visual reasoning benchmarks

On standard VQA tasks evaluated zero-shot on Arabic translations, Peacock is compared against mBlip-BLOOMZ-7B using exact-match accuracy on filtered validation sets. The reported scores are: mBlip-BLOOMZ-7B at 55.7% on VQAv2, 23.3% on OKVQA, and 54.85% on GQA; IB-AraLLaMA at 56.15%, 29.77%, and 58.05%; IB-AceGPT at 51.20%, 16.82%, and 57.60%; LLaVA-AraLLaMA at 52.45%, 25.57%, and 49.75%; and LLaVA-AceGPT at 56.65%, 26.36%, and 52.20%. The best-performing system in that table is IB-AraLLaMA, reported as outperforming mBlip by approximately 4–6 points on all tasks (Alwajih et al., 2024).

On LLaVA-Bench, where GPT-4 provides scores for Conversation, Detailed Description, and Complex Reasoning, the average scores are 52.9 for mBlip-BLOOMZ-7B, 82.3 for IB-AraLLaMA, 69.1 for IB-AceGPT, 71.1 for LLaVA-AraLLaMA, and 73.6 for LLaVA-AceGPT. The detailed sub-scores for IB-AraLLaMA are 84.6 in Conversation, 80.0 in Detailed Description, and 82.1 in Complex Reasoning (Alwajih et al., 2024).

On SEED-Bench, which is summarized as covering eight visual-reasoning dimensions, IB-AraLLaMA is reported to excel in Instance Interaction at 58.8%, Visual Reasoning at 68.9%, and Scene Understanding at 65.5%, while lagging on Spatial Relations at 38.2%. LLaVA-AraLLaMA is described as generally stronger on attributes like Visual Reasoning but weaker on instance counting and location (Alwajih et al., 2024).

These results establish a consistent empirical pattern inside the family: AraLLaMA-based configurations are stronger than AceGPT-based configurations, and the InstructBlip-style branch is especially competitive. The paper’s own comparison therefore links Arabic multimodal performance not only to visual alignment design but also to the quality of the Arabic decoder.

5. Henna and Arabic cultural evaluation

Henna is introduced as an Arabic-culture-aware multimodal benchmark intended to evaluate understanding of images depicting Arabic food, monuments, events, and customs across 11 Arab countries. The dataset contains 1,132 manually reviewed image-question pairs, with images selected from public sources and Wikipedia. Ground-truth answers are generated by prompting GPT-4V with contextual Arabic Wikipedia text, and model responses are evaluated by GPT-4 on four dimensions—Helpfulness, Relevance, Accuracy, and Level of Details—using a 1–10 scale (Alwajih et al., 2024).

On Henna, the paper reports average GPT-4 scores for mBlip-mt0-XL-5B of 34.1 for Helpfulness, 39.2 for Relevance, 35.1 for Accuracy, and 20.7 for Detail. IB-AraLLaMA receives 62.3, 69.0, 49.7, and 49.8 respectively, and is described as nearly doubling the baseline on cultural knowledge (Alwajih et al., 2024).

The benchmark is significant because it formalizes a failure mode that is often obscured by standard VQA evaluation: a model may perform adequately on generic visual reasoning while remaining weak on culturally specific visual-linguistic interpretation. Henna therefore shifts the assessment target from generic multimodal competence toward culturally aware Arabic MLLMs. A plausible implication is that benchmarking localized multimodal systems requires domain-specific evaluation artifacts rather than only translated versions of English benchmarks.

6. Dialectal behavior, qualitative findings, and limitations

The qualitative analysis emphasizes both cultural specificity and dialectal behavior. In a Yemeni-culture example, IB-AraLLaMA correctly identifies a “ma‘sha” dish and explains its preparation, whereas mBlip produces generic captions. In an Egyptian dialect case study, the authors fine-tune InstructBlip models on 1K professionally translated Egyptian-dialect instruction samples. Before dialectal fine-tuning, the models understand dialectal questions but respond in Modern Standard Arabic; after dialectal LoRA-tuning, they answer in Egyptian slang (Alwajih et al., 2024).

The human evaluation for the Egyptian dialect setting uses 20 image-question pairs and 4 Egyptian-native annotators. IB-AraLLaMA receives approximately 8.2/10 on dialect authenticity versus approximately 6.5/10 for GPT-4V when instructed, while answer accuracy is approximately 7.3/10 for IB-AraLLaMA versus approximately 8.0/10 for GPT-4V. The authors characterize the outcome as an emergent behavior in which a small dialectal fine-tune yields strong local flavor while retaining general visual reasoning (Alwajih et al., 2024).

The reported limitations are also explicit. Peacock exhibits occasional object hallucinations, propagation of translation artifacts, and inability to read text within images. These constraints matter for interpretation of the benchmark results. The gains reported for Arabic multimodal reasoning and cultural knowledge do not eliminate standard MLLM failure modes; instead, they coexist with them. This suggests that Arabic-centric multimodal modeling remains bounded by both data-construction noise and unresolved perceptual limitations.

7. Significance and research implications

Peacock sets new baselines for Arabic MLLMs by combining translated-and-filtered multimodal resources with Arabic LLM decoders, particularly AraLLaMA, and by coupling model release with the Henna benchmark (Alwajih et al., 2024). Within the study’s own empirical frame, the strongest configuration is IB-AraLLaMA, which leads on standard Arabic VQA tasks, LLaVA-Bench, several SEED-Bench dimensions, and the Henna cultural benchmark.

The broader contribution lies in the combination of three elements. First, the work demonstrates that Arabic multimodal systems can be assembled from existing English resources through translation and semantic filtering rather than requiring exclusively native multimodal corpora at the outset. Second, it shows that Arabic decoder choice materially affects multimodal performance. Third, it provides concrete evidence that dialectal adaptation can be elicited through LoRA-based fine-tuning on a relatively small professionally translated dialect dataset. Together, these results position Peacock as a reference point for research on culturally aware and dialect-sensitive Arabic multimodal systems.