3LM Arabic STEM & Code Benchmark

• 3LM is a benchmark suite that evaluates Arabic LLMs for STEM and code generation using rigorously curated datasets.
• It combines native and synthetic STEM QA with translated code tasks to assess domain-specific problem-solving and coding skills.
• The suite employs advanced OCR, LLM pipelines, and iterative quality assurance to ensure robust, reproducible performance evaluations.

3LM is a publicly released benchmark suite designed to evaluate LLMs for Arabic in the domains of STEM (Science, Technology, Engineering, and Mathematics) and code generation. Unlike prior Arabic benchmarks, which predominantly cover linguistic, cultural, or religious content, 3LM specifically targets underrepresented yet practically crucial domains: STEM problem-solving and code synthesis. The suite comprises three benchmarks—Native STEM QA, Synthetic STEM QA, and Code Generation—collectively addressing model performance on naturally sourced questions, automatically generated challenging problems, and the ability to comprehend and generate code from Arabic prompts (Boussaha et al., 21 Jul 2025).

1. Composition and Design of the 3LM Suite

3LM is structured around three complementary evaluation tasks:

1. Native STEM QA: Curated from PDFs of worksheets, exams, question banks, and textbooks scraped from educational websites spanning middle- and high-school curricula. Sources are filtered for STEM domains (Biology, Chemistry, Physics, General Science, Mathematics, Geography) using regular expressions. A two-stage OCR pipeline, incorporating Pix2Tex, enables extraction of multilingual mathematical content into LaTeX. The QA extraction uses Qwen3-235B-A22B for self-contained QA pairs, categorized into multiple-choice (MCQ), completion, generative, or other. All items are standardized to a 4-option MCQ format before manual quality assurance by STEM-competent native-Arabic educators.
2. Synthetic STEM QA: Generated from the same educational sources using a modified YourBench pipeline. The process ingests and structures documents, summarizes and chunks content for coverage of both single-hop and multi-hop reasoning, and generates MCQs with answer choices labeled (“أ”, “ب”, “ج”, “د”). Questions requiring external visuals are discarded, and only those rated with difficulty ≥ 6 are retained; additional clustering is enforced to maximize topical and structural diversity.
3. Code Generation: Derived from machine-translated HumanEval+ and MBPP+ benchmarks using an automated GPT-40 translation pipeline. Post-translation, round-trip backtranslation and ROUGE-L F₁ scoring (threshold ≥ 0.85/0.80) filter out semantically inadequate prompts, which are subsequently corrected and validated in multiple review passes by native-Arabic Python experts. Code prompt content (docstrings/descriptions) is adapted to Arabic, but function and variable identifiers remain in English, preserving technical fidelity while enabling programmatic evaluation.

2. Data Sources, Extraction, and Generation Pipelines

Native STEM QA

Data is scraped via keyword/API access. After title and content filtering for STEM domains, a two-tier OCR approach processes both standard text and embedded mathematical notation (via Pix2Tex). Qwen3-235B-A22B is prompted to extract QA pairs, which are then stringently filtered, labeled by type (“MCQ”, “Completion”, “Generative”, “Other”), assigned a domain and difficulty score (1–10), and revealed to be exclusively text-based (visual questions are discarded). All completion-type or fill-in-the-blank questions are reformulated as MCQs, and answer positions are randomized. Native educators perform manual quality control on the final set.

Synthetic STEM QA

The YourBench-based pipeline progresses from document structuring to LLM-driven summarization and chunking, identifying semantically coherent units. Multi-hop chunks serve as seeds for generating complex MCQs, with each question scored for difficulty. Only questions rated ≥ 6 and not requiring visuals are kept. Embedding-based clustering reduces near-duplicate items while maximizing interdisciplinary and topical breadth across STEM.

Code Generation

English prompts from HumanEval+ and MBPP+ are translated to Arabic using GPT-40. Each prompt is backtranslated, and ROUGE-L F₁ is calculated using the formula:

$$\mathrm{ROUGE}\text{-}L\, F_1 = \frac{2\cdot (P\cdot R)}{P + R}$$

where $P$ and $R$ are derived from the longest common subsequence ratios between original, translated, and backtranslated texts. Items falling below threshold are manually corrected, with iterative rounds until all prompts pass quality standards. System/response prompts are technically matched to their English originals.

3. Benchmark Statistics and Content Distribution

Benchmark	# Prompts	Domains	Format/Types
Native STEM QA	865	Biology, Chemistry, Physics, Math, Geo	MCQ (all standardized)
Synthetic STEM QA	1,744	Math, Physics, Chemistry, Biology, Sci	MCQ (difficulty ≥ 6)
Code Generation	542	General Programming (by task description)	HumanEval+ / MBPP+

• The Native STEM QA set has a median prompt length of ≈ 12 words, maxing at 48.
• The Synthetic set shows a median prompt length ≈ 20 words, with mathematical questions as the longest.
• Synthetic QA maintains question-type diversity: conceptual (31.7%), analytical (32.1%), factual (28.9%), application-based (4.8%).
• Code benchmarks: HumanEval-Arabic (164 prompts, plus version 748 tests/task); MBPP-Arabic (378 prompts, plus version 105 tests/task).

4. Evaluation Protocols and Metrics

STEM Benchmarks

• MCQ Format: Models produce one of four answer letters. Evaluated by

$$\mathrm{Accuracy_{MCQ}} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}[\hat y_i = y_i]$$

• Completion Format: Evaluated by selecting the highest-likelihood choice under unconditional generation and scoring as above, with sequence length normalization applied.

Code Benchmarks

• pass@1: Fraction of problems where the first generated solution passes all unit tests:

$$\mathrm{pass@1} = \frac{\# \text{ successful first-try solutions}}{T}$$

Robustness under Distractor Perturbation (RDP)

• In 25% of native MCQs: 20% have the correct answer replaced by a “none of the above” variant; 5% have an additional “none of the above” distractor. Performance drop is measured as

$$\Delta\mathrm{Accuracy} = \mathrm{Accuracy}(\text{original}) - \mathrm{Accuracy}(\text{perturbed})$$

5. Experimental Results and Correlations

Across ∼40 models (base and instruct-tuned; multilingual and Arabic-centric):

• MCQ tasks consistently yield higher accuracy than completion.
• Top performers: On MCQ, Qwen2.5-72B achieves 94.5%; Gemma3-27B achieves 67.6% on completion.
• Instruct-tuned variants (IT): Qwen2.5-72B-IT reaches 93.1% (MCQ), while Gemma3-27B-IT scores 67.4% on completion.
• Domain difficulty varies, with mathematics and physics proving easier than biology and general science. Synthetic STEM QA is ∼5–10 points more challenging than native QA.
• Code pass@1: Qwen3-14B-Base (70.7%), Qwen2.5-72B (67.7%) for base; Qwen3-30B-A30B (72.3%), Qwen2.5-72B-Instruct (72.4%) for instruct-tuned.
• There is a strong correlation $r = 0.97$ between English and Arabic code performance (EvalPlus/ EvalPlus-Ar). Moderate correlation $(r \approx 0.42–0.45)$ is observed between Arabic NLP and code tasks, with a weaker connection for instruct-tuned models.

Robustness: Under RDP, base models lose up to 15 points, while instruct-tuned models degrade by Δ ≈ 8 points, signifying improved robustness.

Ablation results: Lowering synthetic QA difficulty from ≥6 to ≥4 raises scores by 6–8 points but diminishes question quality. Reducing backtranslation threshold from 0.85 to 0.75 increases prompt count by 20% but reduces pass@1 by 5%.

6. Limitations and Future Developments

Current 3LM coverage is restricted to middle- and high-school STEM content; university/graduate-level STEM domains are not represented. Synthetic question generation is subject to LLM-induced bias, especially from Qwen3-235B. Code tasks retain English variable/function names, precluding full evaluation of Arabicized code generation. Diagrammatic or image-based questions are excluded.

Planned extensions include expanding to advanced STEM (e.g., calculus proofs, organic chemistry mechanisms), incorporating multimodal LLMs to support visual content, generating fully Arabicized code prompts, and refining synthetic QA pipelines through adversarial filtering.

7. Significance and Outlook

3LM fills a critical gap by providing high-quality, well-validated benchmarks for evaluating Arabic LLMs in specialized and technical fields—domains overlooked by previous Arabic benchmarks. Its rigorous extraction, synthesis, and quality assurance pipelines ensure reliability for comparative evaluation across LLM architectures. By enabling domain-level, format-specific, and robustness-oriented assessments—coupled with transparent statistical reporting and diagnostic ablations—3LM serves as an authoritative foundation for advancing Arabic LLM research in STEM and computational disciplines (Boussaha et al., 21 Jul 2025).