AraLongBench: Arabic Long-Context QA Benchmark

Updated 10 July 2026

AraLongBench is a comprehensive benchmark designed for evaluating long-context Arabic document QA across single-page and cross-page scenarios.
It employs a fully automated agentic pipeline combining PDF extraction, OCR, BiDi text correction, and layout parsing to generate evidence-grounded QA pairs.
The evaluation protocol uses adjustable difficulty thresholds and diverse domain documents to assess LVLM performance on fine-grained extraction and multi-page reasoning.

AraLongBench is a large-scale Arabic long-context question-answering benchmark introduced alongside the self-evolving workflow in "A-SEA3L-QA: A Fully Automated Self-Evolving, Adversarial Workflow for Arabic Long-Context Question-Answer Generation" (Wang et al., 2 Sep 2025). It is designed for evaluating long-context Arabic document understanding under both single-page and cross-page conditions, using raw multi-page Arabic documents collected and processed through a fully automated agentic pipeline. The benchmark couples document-scale multimodal inputs, explicit evidence grounding, and tunable difficulty through a judge-controlled acceptance threshold, positioning it as a testbed for Arabic Large Vision LLMs (LVLMs) on fine-grained extraction, multi-page aggregation, and long-context reasoning.

1. Corpus composition and scale

AraLongBench is derived from a candidate pool of 1,301 Arabic documents, from which a final retained corpus of 113 multi-page documents was selected, corresponding to an 8.6% acceptance rate (Wang et al., 2 Sep 2025). Documents are chunked in overlapping 50-page windows with a 5-page overlap for QA generation, and individual documents range from 60 to over 300 pages. After OCR and cleanup, token counts per document are approximately distributed as follows: mean 38,500, standard deviation 12,400, minimum 15,000, and maximum 75,000.

The benchmark contains 5,778 QA pairs. These are divided almost evenly between single-page and cross-page settings: approximately 2,900 single-page tasks and approximately 2,878 cross-page or multi-page tasks. This balance suggests an intentional evaluation regime in which local extraction and multi-page aggregation receive comparable weight.

Domain coverage spans 13 categories across the 113 documents.

Domain	Documents
Legal	14
Medical	12
Research (academic reports)	10
Finance	10
Policy/Government	9
Education	9
Technical Manuals	8
News	8
Literature	8
Business	7
Technology	7
Environment	6
History	5

The breadth of this domain distribution is important because long-context Arabic QA is not restricted to a single genre or formatting convention. A plausible implication is that benchmark performance reflects not only reasoning ability but also robustness to heterogeneous layout structures, terminology, and discourse organization.

2. Automated collection and preprocessing pipeline

AraLongBench is built through an autonomous “harvester” agent that accepts a high-level domain or topic query and executes three stages end-to-end without human intervention: repository selection, document retrieval, and multi-stage filtration (Wang et al., 2 Sep 2025). Repository selection spans APIs, web archives, and organizational portals. Retrieval includes both native PDFs and scanned images. Filtration includes license checking, minimum-length constraints, and QA-suitability screening.

The selection criteria are explicitly license-aware. Native PDFs are downloaded through public APIs or direct links; scanned documents in TIFF or JPEG format are included when freely licensed; and only CC-BY, public-domain, or institutional-open documents are retained. QA-suitability is assessed through LVLM judge prompts that test whether at least 80% of pages contain sufficient text or layout for automated QA. If structural anomalies are detected, page-level rejection and re-crawl are performed.

Preprocessing proceeds in five stages:

PDF text and layout extraction via pdfplumber
OCR on scanned pages with Tesseract, including binarization and deskewing
Unicode BiDi re-ordering to correct right-to-left text
Page rasterization with pdf2image for downstream LVLM visual encoding
Structural parsing with DocLayout-YOLO for segmenting headings, paragraphs, tables, and figures

This preprocessing stack indicates that AraLongBench is not a purely textual benchmark. It preserves a document-understanding formulation in which page images, layout cues, and structural segmentation are integral to downstream QA generation and evaluation.

3. Multi-LVLM annotation architecture

The benchmark is produced through a multi-role LVLM architecture rather than a conventional static annotation pipeline (Wang et al., 2 Sep 2025). The system comprises four functional components: a Question Generator, an Answer Generator Swarm, an Evaluator or Judge, and a Final Validator.

The Question Generator takes image chunks $I_c$ and layout annotations $L_c$ as input and outputs a draft question $Q_c$ with cognitive premises. Its policy $\pi$ enforces traceability to source content and a target difficulty level. The Answer Generator Swarm receives $(I_c, Q_c)$ and produces candidate answers $\{A_{c_i}\}_{i=1}^N$ , each grounded in document evidence. The Evaluator or Judge takes $(I_c, L_c, Q_c, \{A_{c_i}\}, \pi)$ and outputs a validated answer set $\{A'_{c_i}\}$ , a question-difficulty rating $d(Q_c)$ , and feedback $F$ consisting of correction and refinement hints. The Final Validator receives $L_c$ 0 and outputs fully validated QA pairs with explicit evidence pages and justifications.

A central property of the system is traceability. Questions are not meant to float free of the source document; they are tied to page evidence, evidence spans, and a justification field in the final serialized record. In practical terms, this constrains the benchmark toward document-grounded QA rather than open-domain answer generation.

4. Self-evolving adversarial cycle and difficulty control

AraLongBench is generated through a self-evolving adversarial loop rather than one-pass synthesis (Wang et al., 2 Sep 2025). The cycle operates as follows: the Question Generator proposes $L_c$ 1; the swarm proposes $L_c$ 2; the Judge computes QA accuracy and difficulty; if accuracy is below the confidence threshold $L_c$ 3, the system automatically regenerates either $L_c$ 4 or $L_c$ 5; when the Judge sees more than 50% answer accuracy for a batch, it instructs the Question Generator to raise question complexity by one level; the process repeats until all chunks are covered; and the Final Validator checks evidence alignment and prunes mismatches down to less than 5% error.

Confidence is assigned at the answer level. For each answer $L_c$ 6, the Judge assigns a score $L_c$ 7 based on overlap with evidence and answer consistency. If $L_c$ 8, the system automatically regenerates either the question or the answer swarm, depending on whether the failure is attributed to poor question formulation or low answer quality.

The formal acceptance rule is expressed as

$L_c$ 9

with acceptance only when

$Q_c$ 0

By varying $Q_c$ 1 over $Q_c$ 2, the benchmark yields question sets of increasing difficulty, with higher $Q_c$ 3 producing easier retained questions because only high-confidence items survive. A point of potential confusion is that the confidence-policy summary also lists typical thresholds as No Gate $Q_c$ 4, Medium Gate $Q_c$ 5, and High Gate $Q_c$ 6 only keep very high-confidence questions $Q_c$ 7 The benchmark evaluation table, however, is reported for $Q_c$ 8, $Q_c$ 9, and $\pi$ 0. The formal hyperparameter definition therefore governs the benchmark splits described in the main evaluation.

5. Task design and evaluation protocol

AraLongBench defines two principal task types and three context-length regimes (Wang et al., 2 Sep 2025). Single-Page (SP) tasks are questions whose answers lie entirely within one page. Cross-Page or Multi-Page (CP) tasks require aggregation of information across two or more pages. Document context is stratified by total length into SC (short, fewer than 100 pages), MC (medium, 100–200 pages), and LC (long, more than 200 pages).

In zero-shot evaluation, the reported metric is Accuracy, defined as the percentage of fully correct questions. For binary QA judgments, the benchmark description also notes that one can use Precision, Recall, and $\pi$ 1, although Accuracy is the headline metric in the reported results. The emphasis on fully correct questions is consequential: it makes the evaluation stricter than partially matched span-level scoring and aligns it with document-grounded correctness rather than approximate semantic overlap.

The benchmark’s JSONL format mirrors this grounding requirement. Each record contains: {"doc_id", "pages":[…], "question", "answer", "evidence_pages", "evidence_spans", "justification"}. This schema supports page-localized analysis, evidence auditing, and controlled separation of retrieval and reasoning errors.

6. Baseline LVLM performance and empirical patterns

AraLongBench was used to evaluate leading Arabic-capable LVLMs in a zero-shot setting under three gating conditions, $\pi$ 2, $\pi$ 3, and $\pi$ 4, across SC, MC, LC, and SP or CP subsets (Wang et al., 2 Sep 2025). An excerpted set of reported Accuracy values is shown below.

Model	SC @ No-Gate	CP @ No-Gate
GPT-4o	87.2	82.8
Gemini-2.0F	93.0	86.7
AIN (7B)	78.5	71.5
Qwen 2.5 VL	89.8	85.7

The same excerpt reports SC accuracy under tighter gating as follows: GPT-4o declines from 87.2 at No-Gate to 79.1 at 50% and 65.7 at 75%; Gemini-2.0F declines from 93.0 to 84.1 to 71.8; AIN (7B) declines from 78.5 to 69.1 to 58.2; and Qwen 2.5 VL declines from 89.8 to 79.6 to 69.4. On SP versus CP under No-Gate, GPT-4o scores 90.9 versus 82.8, Gemini-2.0F 94.3 versus 86.7, AIN (7B) 80.0 versus 71.5, and Qwen 2.5 VL 91.5 versus 85.7.

Three empirical regularities are explicitly identified. First, all models’ accuracy drops by 10–12 percentage points when moving from No-Gate to 50% gate, and by another 10 points when moving to 75%. Second, CP tasks consistently score 5–8 percentage points lower than SP tasks. Third, LC settings exhibit the steepest performance declines. Taken together, these observations indicate that the benchmark stresses not only answer correctness but also robustness to increasingly selective confidence-based filtering and extended context integration.

7. Access, research use, and relation to adjacent benchmarks

AraLongBench is released as JSONL under CC-BY-4.0 on GitHub at https://github.com/wangk0b/Self_Improving_ARA_LONG_Doc.git (Wang et al., 2 Sep 2025). The stated access procedure is: clone the repository, install Python dependencies with pip install -r requirements.txt, and download the benchmark through the provided script download_aralongbench.sh.

The benchmark documentation also specifies several research practices. It recommends using the gated splits $\pi$ 5 to design curriculum-learning schedules, evaluating SP and CP subsets separately to isolate multi-page reasoning capability, leveraging the domain metadata to probe domain transfer, and combining AraLongBench with existing short-form Arabic DU datasets such as Camel-Bench for mixed-context pretraining. These recommendations situate AraLongBench as both an evaluation resource and a dataset for training curricula.

Within the broader Arabic evaluation landscape, AraLongBench occupies a different niche from "AraLingBench A Human-Annotated Benchmark for Evaluating Arabic Linguistic Capabilities of LLMs" (Zbib et al., 18 Nov 2025). AraLingBench is a fully human-annotated multiple-choice benchmark spanning grammar, morphology, spelling, reading comprehension, and syntax through 150 expert-designed questions, whereas AraLongBench centers on long-context Arabic document QA over multi-page sources. This suggests a complementary division between foundational linguistic diagnosis and multimodal long-document reasoning.

A terminological complication arises from the distilled description attached to "AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies" (Ye et al., 2024), where the name “AraLongBench” is expanded as “Abstract and Long-context Analogies Benchmark.” In that source, the term refers to a two-part multiple-choice and retrieval benchmark for identifying abstract analogies in long contexts and large candidate pools. In current Arabic document-understanding usage, however, AraLongBench denotes the Arabic long-context QA benchmark introduced in A-SEA3L-QA. The coexistence of these usages makes explicit citation especially important in technical discussion.