BAREC 2025 Shared Task: Arabic Readability

• BAREC 2025 Shared Task is a benchmark for Arabic readability assessment with a fine-grained 19-level ordinal scale mapping text difficulty to learner levels.
• It encompasses sentence-level and document-level prediction tasks, with the document score determined by the most challenging sentence.
• The task employs Quadratic Weighted Kappa as the primary metric and encourages diverse system methodologies across strict, constrained, and open tracks.

Searching arXiv for the specified BAREC 2025 Shared Task papers and related shared-task context. BAREC 2025 Shared Task is a shared evaluation on automatic Arabic readability assessment at fine granularity, centered on a 19-level ordinal scale intended to align texts with appropriate learner levels. It comprises sentence-level and document-level prediction, and the label space has official mappings to coarser 7/5/3-level schemes for secondary reporting. In the document task, the overall document level is defined by the most difficult sentence, reflecting an emphasis on avoiding texts that are too advanced. Across the task, the primary evaluation metric is Quadratic Weighted Cohen’s Kappa (QWK), so distant ordinal errors are penalized more heavily than near misses (Abdou, 18 Sep 2025, Basem et al., 12 Sep 2025).

1. Task definition and track structure

The shared task targets fine-grained Arabic readability assessment over a 19-level ordinal scale, with levels 1–19 and higher values corresponding to harder text. Two subtasks are defined. The sentence-level subtask predicts one readability label for each sentence. The document-level subtask predicts one label for an entire document, with the task definition specifying that the document level is the maximum sentence-level difficulty. This design encodes a conservative pedagogical criterion: a document is treated as difficult if any sentence is too advanced.

The organizational structure comprises six tracks, with three tracks for each subtask. The Strict track uses only BAREC. The Constrained track uses BAREC plus SAMER resources, specifically the SAMER Corpus and SAMER Lexicon. The Open track allows any additional resources and augmentation. The task is organized by Elmadani, Alhafni, Taha, and Habash, and BAREC emphasizes Modern Standard Arabic (MSA), although participating systems may draw on backbones that also cover dialectal Arabic in order to capture stylistic and register variability (Abdou, 18 Sep 2025, Basem et al., 12 Sep 2025).

2. Corpus, labels, and annotation regime

The shared task is built on the BAREC corpus, described as a manually annotated resource with over 69,000 sentences and more than one million words, and elsewhere as a corpus of 1M+ words, 68k+ sentences, and 1,900+ documents. It provides 19 fine-grained readability levels together with mappings to 7/5/3-level schemes. The official sentence-level splits are specified as Train: 54.8k sentences, Dev: 7.3k sentences, Test: 7.3k sentences, and Blind test: 3.4k sentences. The official train/dev/test splits are provided by the organizers, while the blind test is supplied separately for evaluation (Abdou, 18 Sep 2025, Basem et al., 12 Sep 2025).

The corpus is annotated under official guidelines intended to ensure annotation consistency. The distribution is severely imbalanced: mid-level labels such as 12 and 14 are frequent, while extreme labels such as 1, 18, and 19 are rare. That imbalance is consequential because the label space is strictly ordinal. A plausible implication is that systems must optimize not only class discrimination but also proximity-aware behavior, since misclassifying an extreme or tail label by several levels incurs a much larger penalty under the official metric.

3. Evaluation framework and ordinal metrics

The primary metric is Quadratic Weighted Kappa. Let $K$ be the number of levels, with $K = 19$. Using the observed agreement matrix $O$ and the expected agreement matrix $E$ under independence, QWK is defined as

$$\kappa = 1 - \frac{\sum_{i,j} w_{ij} O_{ij}}{\sum_{i,j} w_{ij} E_{ij}},$$

with quadratic weights

$w_{ij} = \frac{(i-j)^2}{(K-1)^2}.$

In the formulation reported by the task papers, $O_{ij}$ is the count of items with true label $i$ and predicted label $j$, while $E_{ij} = (r_i c_j)/N$ is derived from row and column marginals under label independence. No deviation from the standard quadratic weights is reported (Abdou, 18 Sep 2025, Basem et al., 12 Sep 2025).

Secondary metrics are also reported. These include exact accuracy, adjacent accuracy at $K = 19$0 level, MAE or Average Distance, and coarse-grained accuracies after mapping to 7-, 5-, or 3-level schemes. In one task description these appear as Exact accuracy, $K = 19$1 adjacent accuracy, MAE, and Acc3, Acc5, Acc7; in another they appear as Accuracy at 19/7/5/3 label granularities, Adjacent Accuracy ($K = 19$2 Acc19), and Average Distance with

$K = 19$3

A common source of confusion is the relation between exact accuracy and QWK. The task formulations explicitly show that a method can improve QWK even when exact accuracy dips slightly, because shrinking large-gap errors into near misses reduces the squared penalty term that dominates the metric.

4. Representative system methodologies

Two reported systems illustrate distinct methodological responses to the task’s ordinal structure. The mucAI system proposes a simple, model-agnostic post-processing technique for fine-grained Arabic readability classification. It applies Inductive Conformal Prediction (ICP) with standard multiclass nonconformity scores, using a calibration split to estimate a threshold $K = 19$4 such that, under exchangeability, the true label is included in the prediction set with probability at least $K = 19$5:

$K = 19$6

Three scoring functions are evaluated: naïve inverse probability, Adaptive Prediction Sets (APS), and Regularized APS (RAPS), with $K = 19$7 in the experiments. For a new input $K = 19$8, the conformal set is

$K = 19$9

Posterior probabilities from the base classifier are then restricted and renormalized within the conformal set,

$O$0

and the final prediction is the rounded posterior mean over ordinal labels,

$O$1

The reported base model is AraBERT-v2. Losses include Cross-Entropy, Squared Earth Mover’s Distance for ordered classes, an ordinal regression head, and a Focal Loss variant. Benchmark preprocessing uses four CAMeL pipelines—Word, Lex, D3Lex, and D3Tok—while blind evaluation uses Farasa segmentation. The official dev set is split stratifiably into dev-cal and dev-tune, with 4,981 instances for calibration and 2,329 for tuning, and $O$2 unless otherwise stated. APS and RAPS are preferred due to coverage robustness (Abdou, 18 Sep 2025).

The !MSA system adopts a confidence-weighted ensemble of four complementary transformer models: AraBERTv2, AraELECTRA, MARBERT, and CAMeLBERT. It trains multiple heads per backbone with distinct objectives: 19-way classification, regression with a continuous score in $O$3, and ordinal prediction using CORAL with $O$4 binary thresholds. The final system uses CE, MSE, and CORAL rather than focal or label-smoothed CE. For class imbalance it applies weights

$O$5

within CE and CORAL losses. Classification-head confidence is derived from maximum softmax probability, $O$6, while regression confidence uses inverse variance from validation residuals. Probabilistic fusion is

$O$7

and scalar fusion is

$O$8

Arabic-specific preprocessing includes regular-expression cleaning and D3TOK morphological tokenization. In the constrained track, SAMER is integrated through relabeling with the best BAREC-trained model after an initial min–max scaling attempt from 3–6 to 1–19 yielded only modest performance. In the open track, approximately 10,000 synthetic examples are generated via Gemini 2.5 Flash to bolster rare-level coverage. At the document level, the system reports targeted post-processing to correct skewed prediction distributions, including a high-label override heuristic when any model predicts 16 or 17 for a document (Basem et al., 12 Sep 2025).

5. Reported results and competitive outcomes

The mucAI submission reports consistent QWK improvements of 1–3 points across different base models in the strict track. At sentence level on the test split, the paper reports CE baseline: QWK 82.6 $O$9 CE+CP: 84.3 (+1.6), EMD baseline: 82.8 $E$0 EMD+CP: 83.9 (+1.1), and Focal baseline: 81.8 $E$1 Focal+CP: 84.4 (+2.6). Ensemble results are Average 84.9 and Most Common 84.6, while an Oracle upper bound that is explicitly described as non-deployable reaches 95.3 QWK. On the blind test, sentence-level results are CE+CP: 84.3, EMD+CP: 84.6, Focal+CP: 85.3, Regression baseline (point): 85.4, Ensemble Average: 85.7, and Most Common: 84.8. For document level on the blind test, the reported strict-track result is 73.3 QWK using the maximum over sentence-level predictions. The paper summarizes the strict-track submission as 84.9% QWK on test and 85.7% on blind test for sentence level, and 73.3% for document level (Abdou, 18 Sep 2025).

The !MSA system reports first place in six of six tracks. Headline QWK scores are Sentence-level Strict: 87.5 (Run 1, 1st/39), Constrained: 86.6 (Run 1, 1st/20), and Open: 86.4 (Run 1, 1st/22). Document-level results are Strict: 87.4 (Run 1, 1st/27), Constrained: 84.3 (Run 1, 1st/22), and Open: 82.2 (Run 1, 1st/19). Selected secondary metrics include Sentence Strict Run 1 with Acc19 43.5% and $E$2 Acc19 76.7%, and Document Strict Run 1 with Acc19 52.0% and $E$3 Acc19 94.0%. In its ablations, single models span QWK from 81.0% to 84.8%, with MARBERT+COR identified as the strongest single model, while confidence-weighted fusion raises QWK up to 87.5%. The document-level distribution-correction stage is reported to yield a +6.3% absolute QWK gain (Basem et al., 12 Sep 2025).

6. Error patterns, educational significance, and open directions

The task papers converge on the importance of locality in ordinal error. In the mucAI analysis, CP with APS at $E$4 achieves 94.88% coverage with average set size approximately 5. The decoding redistributes errors: some perfect predictions become near-misses, but many large errors shrink. On dev-tune, 86.7% of newly introduced errors are within $E$5 level, while 17.1% of originally incorrect predictions improve; among those improved cases, 80.6% shrink by 1 level, 14.7% by 2, 3.1% by 3, and 1.6% by 4 levels. Domain variation also matters: failures with the gold label outside the conformal set cluster more in Social Sciences (approximately 7.1%) and STEM (approximately 6.1%) than in Arts & Humanities (approximately 4.3%). The paper therefore suggests Mondrian CP for domain-aware calibration, as well as tighter calibration or adaptive thresholds to reduce set size while maintaining validity (Abdou, 18 Sep 2025).

The !MSA analysis identifies complementary challenges. Rare levels 1, 18, and 19 remain difficult because sparse training examples lead to under-coverage. Dialectal and orthographic variability, very short sentences, and very long sentences all increase uncertainty. The paper also notes that ordinal heads reduce inversions between neighboring levels but do not fully eliminate them. At the system-design level, it points to calibration across heterogeneous heads, more rigorous document aggregation beyond max or mean plus heuristics, and additional curated tail data as natural next steps. Taken together, these findings suggest that BAREC 2025 functions not only as a leaderboard task but also as a testbed for ordinal modeling, uncertainty quantification, Arabic-specific preprocessing, and aggregation strategies under pedagogically motivated risk constraints. Both reports also foreground practical educational use: compact conformal sets can offer a handful of plausible levels per sentence with formal coverage guarantees, while ensemble and post-processing methods can reduce high-risk assignments far from a learner’s ability. Reproducibility is supported by public code repositories for both systems and explicit training details, including single-NVIDIA-A100 training for mucAI and NVIDIA L40S training with mixed precision for !MSA (Abdou, 18 Sep 2025, Basem et al., 12 Sep 2025).