MizanQA: Moroccan Legal QA Benchmark

• MizanQA is a specialized benchmark designed to assess LLM performance on Moroccan legal question answering tasks with complex linguistic and jurisprudential nuances.
• It comprises 1,776 multi-answer multiple-choice questions that blend Modern Standard Arabic, Islamic Maliki jurisprudence, Moroccan customary law, and French legal elements.
• Evaluation metrics reveal significant gaps in strict accuracy and calibration, emphasizing the need for domain adaptation, multi-metric assessment, and enhanced legal reasoning.

Searching arXiv for the specified paper and any directly relevant supporting work on Arabic legal QA benchmarks. {"query":"(Bahaj et al., 22 Aug 2025) MizanQA Benchmarking LLMs on Moroccan Legal Question Answering"} MizanQA (ميزان), pronounced Mizan and literally meaning “scale” in Arabic, is a benchmark designed to evaluate LLMs on Moroccan legal question answering tasks characterized by rich linguistic and legal complexity (Bahaj et al., 22 Aug 2025). The benchmark targets a low-resource legal setting in which Modern Standard Arabic is intertwined with Islamic Maliki jurisprudence, Moroccan customary law, and French legal influences. Comprising 1,776 multiple-choice questions, including a large proportion of multi-answer items, MizanQA is structured to capture the nuances of authentic legal reasoning rather than only single-label classification, and its benchmarking results expose substantial performance gaps in current multilingual and Arabic-focused LLMs.

1. Definition and scope

MizanQA is presented as a balancing tool for measuring LLM performance on Moroccan legal reasoning, an area described as rich in linguistic and jurisprudential complexity yet historically under-resourced in NLP (Bahaj et al., 22 Aug 2025). The benchmark is explicitly situated in Arabic legal NLP, but not in a monolingual or mono-jurisprudential sense. Its source material combines Modern Standard Arabic text with Islamic Maliki jurisprudence terms, Moroccan customary-law expressions, and French civil-law remnants together with international-law borrowings.

This configuration makes MizanQA a domain-specific legal QA benchmark rather than a generic Arabic reading-comprehension resource. The legal questions are not limited to statutory recall: the benchmark is designed around multiple-choice reasoning, including multi-answer formats with up to 10 correct options. This suggests an emphasis on structured legal judgment under lexical, doctrinal, and procedural variation.

The benchmark’s symbolic naming is itself programmatic. “Scale” evokes the universal symbol of justice, and the paper links that symbolism directly to evaluation: MizanQA is meant to measure LLM performance in a jurisdictionally specific legal environment where both language and legal sources deviate from standard high-resource NLP assumptions.

2. Dataset composition and legal-linguistic sources

The dataset contains 1,776 multiple-choice items distributed across 14 categories (Bahaj et al., 22 Aug 2025). Its question design is heterogeneous in both answer cardinality and textual length. Approximately 30% of the items are single-correct-answer MCQs, while approximately 70% are multi-answer MCQs. Multi-answer items may contain up to 10 correct options drawn from option sets of size 2–12.

The benchmark’s general statistics are as follows.

Statistic	Value
Number of questions	1776
Number of categories	14
Options per question (min–max)	2–12
Correct options per question (min–max)	1–10
Words per question (min–max)	1–63
Words per option (min–max)	1–71

The reported category counts identify the major topical concentrations.

Category	Count
Criminal Law	847
Civil Procedure	460
Exam	131
The Judicial System of the Kingdom	88
The Moroccan Constitution	70
Family Law	66
The Justice Sector	39
Family Code	38
Law of Obligations and Contracts	37

The paper also notes “plus 5 smaller categories,” but does not enumerate them in the provided summary.

The linguistic profile of the benchmark is central to its difficulty. The text is described as Modern Standard Arabic infused with Maliki jurisprudence terminology, customary-law expressions, and French-derived legal vocabulary. The resulting mixture includes archaic or region-specific phrasing absent from standard Arabic corpora. This has direct implications for representation learning: tokenization, lexical normalization, and semantic grounding become materially harder when the benchmark combines doctrinal Arabic, jurisdiction-specific formulae, and legal borrowings.

An example item cited from the paper illustrates the structure. A question beginning “If it is alleged that a Pasha, … has committed a felony or misdemeanor while performing their duties, then …” is followed by options A–F, where option F states “The answers are correct,” and the correct answer is F. This example shows that the benchmark includes procedural legal reasoning and answer structures that are not reducible to superficial lexical matching.

3. Task structure and difficulty profile

MizanQA’s difficulty is tied to two explicit variables: the number of correct options and the density of domain-specific jargon (Bahaj et al., 22 Aug 2025). The summary states that difficulty correlates with the number of correct options, because multi-answer sets are intrinsically harder, and with domain-specific jargon, with Family Code and Criminal Law questions scoring lowest in model evaluations.

The multi-answer formulation is particularly consequential. In a single-answer MCQ, model failure is binary and localized. In MizanQA, by contrast, the model must often recover a set of legally valid options from a larger candidate set. Missing one correct option or adding one incorrect option changes the evaluation outcome substantially. This yields a combinatorial decision surface rather than a single-label classification problem.

The benchmark’s legal heterogeneity compounds that difficulty. Code-switching between formal Arabic and French-derived legal terms such as “pourvoi en cassation” is reported to confuse tokenization. Maliki-jurisprudence and Moroccan customary rules introduce concepts absent in training corpora; the paper gives “hima” protected zones as an example. These features are not incidental linguistic noise. They are constitutive of the legal domain being modeled, and they force evaluation to reflect jurisdiction-specific legal competence rather than generic multilingual fluency.

This suggests that MizanQA operationalizes legal QA as a joint problem of doctrinal recall, lexical disambiguation, and set prediction. The benchmark is therefore notable not merely for being in Arabic, but for embedding Moroccan legal pluralism into the task definition itself.

4. Benchmarking protocol and evaluation metrics

The evaluated models are Allam-2 (7 B parameters), Gemini-1.5-flash, Gemini-2.0-flash, Llama-3.3 (70 B), Llama-4-maverick (17 B), and Llama-4-scout (17 B) (Bahaj et al., 22 Aug 2025). Allam-2 is identified as Arabic-centric; the Gemini and Llama variants are multilingual, with Gemini models additionally described as vision+language, and Llama-4-maverick as having improved calibration. All models were accessed via the Groq API or Google’s Gemini API and were prompted with a template instructing them to output a list of chosen option labels accompanied by confidence scores from 1 to 100.

The benchmark uses four families of evaluation signals: Strict Accuracy, Precision/Recall/F1-Like scoring, Partial Match Penalized Accuracy, and Expected Calibration Error.

Strict Accuracy is defined as exact set match:

$\mathrm{Acc} = \frac{1}{|\mathcal{Q}|}\sum_{i=1}^{|\mathcal{Q}|} \mathbbm{1}_{[S_i = C_i]}.$

For question $i$, with predicted set $S_i$ and correct set $C_i$, the paper defines:

• $TP_i = |C_i \cap S_i|$
• $FP_i = |S_i \setminus C_i|$
• $FN_i = |C_i \setminus S_i|$

Per-question precision and recall are then:

$$\mathrm{Precision}_{(i)} = \frac{TP_i}{TP_i + FP_i}, \qquad \mathrm{Recall}_{(i)} = \frac{TP_i}{TP_i + FN_i}.$$

The F1-like score is averaged across questions:

$$\mathrm{F1} = \frac{1}{|\mathcal{Q}|}\sum_{i=1}^{|\mathcal{Q}|} \frac{2\,P_i\,R_i}{P_i+R_i}.$$

Partial Match Penalized Accuracy introduces explicit false-positive penalization:

$$\mathrm{PMPA}_\beta = \frac{1}{|\mathcal{Q}|}\sum_{i=1}^{|\mathcal{Q}|} \max\!\Bigl(0,\min\!\bigl(1,\tfrac{TP_i - \beta\,FP_i}{|C_i|}\bigr)\Bigr).$$

Here, $i$0 controls the penalty applied to incorrect choices.

Calibration is measured with Expected Calibration Error by binning predictions by confidence and computing:

$i$1

ECE is reported in two settings: per-option and set-level joint confidence. This dual reporting matters because a legally useful system may appear reasonable on option-level confidence while remaining unreliable at the level of the full predicted answer set.

5. Empirical results

The aggregate results show consistent separation between strict exact-match performance and partial-credit metrics, with substantial variation across models (Bahaj et al., 22 Aug 2025).

Model	PMPA(1)	PMPA(0.5)	ACC
Allam-2 (7 B)	26.88	34.04	15.32
Gemini-1.5-flash	35.90	44.23	24.26
Gemini-2.0-flash	53.57	58.34	42.11
Llama-3.3 (70 B)	46.78	50.73	33.28
Llama-4-maverick (17 B)	49.97	55.53	36.83
Llama-4-scout (17 B)	44.06	49.01	31.27

A second set of reported metrics captures F1-like performance and calibration.

Model	F1(1)	F1(2)	ECE_set ↓
Allam-2 (7 B)	43.07	39.93	51.43
Gemini-1.5-flash	53.30	48.93	48.52
Gemini-2.0-flash	64.84	62.16	41.16
Llama-3.3 (70 B)	59.21	56.18	59.40
Llama-4-maverick (17 B)	64.90	61.29	29.10
Llama-4-scout (17 B)	59.51	55.60	61.78

The main reported findings are explicit. Strict accuracy is uniformly low on multi-answer items because missing or extra options are heavily penalized. The drop from PMPA(1) to PMPA(0.5) shows that models are sensitive to false positives. Gemini-2.0-flash leads with 42% strict accuracy and 53.6% PMPA(1), while Allam-2 lags at approximately 15% ACC.

The calibration results are also nontrivial. Llama-4-maverick has the lowest reported ECE values, with ECE_opt of 17.64 and ECE_set of 29.10 in the paper’s full table, whereas several other models exhibit much worse set-level calibration. This indicates that raw answer quality and confidence reliability are not perfectly aligned. A model can be relatively strong on answer selection while still being poorly calibrated at the set level.

The paper includes illustrative success and failure cases. Gemini-2.0-flash correctly selected all three required options in a Moroccan Constitution question on electoral procedures, assigning high confidences of at least 90. By contrast, Llama-3.3 missed one Maliki-jurisprudence clause in a Family Code question, selecting only 2 of 4 correct options and adding an incorrect one. The paper uses this failure to illustrate the difficulty of religious-law fusion.

6. Interpretation, limitations, and projected development paths

The benchmark’s discussion emphasizes cultural and domain-specific challenges rather than purely architectural limitations (Bahaj et al., 22 Aug 2025). Off-the-shelf multilingual LLMs are reported to capture generic legal patterns, such as Contracts and Constitution, better than culturally bound domains such as Family Code. Poor calibration in set-level confidence is identified as a problem for trust in automated legal support tools.

A common oversimplification would be to read the benchmark only through strict accuracy. The reported metric design makes clear that exact-set correctness, partial recovery of correct options, false-positive sensitivity, and calibration are separable properties. This suggests that model assessment for Moroccan legal QA requires a multi-metric view: ACC alone understates partial competence, while PMPA or F1-like scores alone may obscure exact-answer reliability and confidence miscalibration.

The paper recommends four improvement strategies. The first is domain adaptation through fine-tuning on Moroccan statutes, exam banks, and case law in order to internalize local idioms. The second is data augmentation through paraphrase and back-translation to expand coverage of rare terms. The third is retrieval augmented generation using a Moroccan-law knowledge base during inference to ground answers. The fourth is multi-task curriculum training jointly with other Arabic legal QA tasks, including Saudi Arabian benchmarks, to share cross-jurisdictional knowledge.

The overall conclusion is that MizanQA exposes significant performance gaps in current LLMs on Moroccan legal QA, especially in multi-answer and culturally specific domains. Even the best reported model, Gemini-2.0-flash, reaches only approximately 42% strict accuracy and approximately 65% F1-like. A plausible implication is that progress in this area will depend less on generic multilingual scaling alone than on specialized pretraining, grounded retrieval, and richer calibration methods aligned with low-resource Arabic legal settings.