AraEngLongBench: Arabic-English Long Document QA

• AraEngLongBench is a bilingual long-document QA benchmark built from visual, layout-rich PDFs averaging 77 pages and 84k tokens.
• It employs a fully automated multi-agent pipeline integrating OCR, layout analysis, and evidence-based question generation to ensure cross-page consistency.
• Empirical evaluations show significant challenges in cross-page reasoning and Arabic document understanding, highlighting gaps in current LVLM performance.

Searching arXiv for the benchmark paper and closely related long-context benchmarks to ground the article. AraEngLongBench is a bilingual Arabic–English long-document question–answering benchmark introduced in “Multi-Agent Interactive Question Generation Framework for Long Document Understanding” (Wang et al., 27 Jul 2025). It is built from long, multi-page, visual, layout-rich documents, with average length around 77 pages and approximately 84k tokens, and comprises 6,732 automatically generated question–answer pairs with evidence pages, evidence sources, justification, validation, and difficulty estimates. The benchmark is designed to test and train Large Vision-LLMs on long-context document understanding, particularly in settings where cross-page reasoning, complex layouts, OCR noise, and low-resource language constraints make standard short-context document understanding methods unreliable.

1. Motivation and problem setting

AraEngLongBench was created in response to three gaps identified in long-context document understanding research. First, Large Vision-LLMs such as GPT-4o, Gemini, Claude, LLaVA, and Qwen perform well on short-context document understanding benchmarks, including DocVQA, ChartQA, and InfographicVQA, but their accuracy drops to roughly 40% on long-context benchmarks such as DUDE, MMLongBench, LongDocURL, and M-LongDoc. The reported failure modes include difficulty with reasoning across multiple pages, tracking cross-page references, and handling dense long documents (Wang et al., 27 Jul 2025).

Second, existing long-document benchmarks are described as limited in page count, token density, or dependence on human annotation. Third, the paper identifies a scarcity of high-quality, fine-grained, multi-page Arabic question–answer data. Arabic is treated as a low-resource language in the LVLM ecosystem, and the lack of Arabic long-document supervision is presented as a direct obstacle to robust document understanding.

Within this framing, AraEngLongBench targets long-context visual question answering rather than short-span extraction. The benchmark stresses memory and attention across dozens to hundreds of pages, cross-page reasoning over text, tables, and figures, and layout-aware interpretation of PDFs rendered as page images. The Arabic component adds further pressure through right-to-left scripts, diacritic marks, dialects, OCR errors, and inconsistent encodings. This suggests that AraEngLongBench is intended not merely as a bilingual extension of existing benchmarks, but as a stress test for multilingual long-context reasoning under realistic document-layout constraints.

2. Automated multi-agent construction pipeline

AraEngLongBench is produced by a fully automated, multi-agent interactive question generation framework rather than by manual question–answer annotation (Wang et al., 27 Jul 2025). The pipeline begins with document preprocessing. Each PDF is rendered to page images $I$ using pdf2image, preserving layout and visual structure. OCR text $O$ is then obtained using LVLM-based OCR methods such as TextHawk2 and OceanOCR. Layout analysis is performed with a deep learning-based detector, described as YOLO-based, to identify headings, paragraphs, tables, figures, and related regions, producing structured layout annotations $L$. Documents are subsequently split into overlapping page chunks, with a 10-page overlap used explicitly to maintain coherence and support cross-chunk context.

The core generation loop is organized around five agents. Agent 1 performs question generation and later question refinement. Agent 2 performs question extraction and filtering. Agent 3 generates answers from document page images rather than from OCR text alone. Agent 4 assesses answers, produces refined reference answers, estimates depth and difficulty, and returns feedback. Agent 5 validates evidence and confirms that each final question is answerable from the specified document evidence.

The interaction is iterative. For each chunk $c$, Agent 1 generates candidate questions from $O_c$, $L_c$, and a policy $\pi$ specifying desired question types, difficulty, and cross-page requirements. Agent 2 removes irrelevant, redundant, or low-quality items. Agent 3 answers the filtered questions from the visual pages $I_c$. Agent 4 then evaluates correctness and difficulty; if Agent 3’s answer accuracy exceeds 40% for a subset, Agent 1 is triggered to increase question complexity, for example by shifting toward multi-hop or cross-page reasoning. After refinement, Agent 5 validates answerability and evidence support.

The paper does not define an explicit differentiable objective or loss function for this system. Instead, the framework is procedural, controlled by policies and heuristics, with the 40% accuracy threshold functioning as the main numeric rule. A plausible implication is that AraEngLongBench prioritizes controllable difficulty escalation and evidential grounding over optimization against a single scalar training objective.

3. Dataset profile and internal structure

The benchmark contains 6,732 question–answer pairs and is heavily skewed toward reasoning-intensive categories (Wang et al., 27 Jul 2025). The question distribution reported in the paper is as follows:

Question type	Count	Share
Reasoning	4,160	61.8%
Factual Recall	271	4.0%
Image-based Question	412	6.1%
Conceptual Understanding	272	4.0%
Hypothetical Reasoning	798	11.9%
Multi-hop Reasoning	281	4.2%
Unanswerable	454	6.7%

Additional categories are also present: Prediction Analysis (15), Step-by-step Explanation (36), Data Retrieval (paper_content OCR) (6), Experimental Design (21), and Argumentation (6). The predominance of Reasoning and Hypothetical Reasoning indicates that the benchmark is not organized around simple factual lookup.

Answer formats are similarly heterogeneous. Among 5,085 answers annotated by answer type, the benchmark includes Text (2,826), Integer (1,169), Code (624), Float (347), List (74), Boolean (31), Array (15), and JSON (7). This breadth is notable because it requires models to operate beyond free-form prose generation, including numerical, structured, and code-like outputs.

AraEngLongBench covers English and Arabic long documents, but the paper does not provide an explicit numerical language breakdown. It does state that the dataset is partitioned into English and Arabic portions and that both subsets include categories such as short, medium, and long context; single-page and cross-page; and unanswerable items. The documents span technical domains including Chemistry, Biology, Mathematics, and Computer Science. The collection process also scraped long-form PDFs and HTML from the web. The paper does not exhaustively enumerate document genres, but notes that the source material includes long, multi-page documents across diverse domains.

The benchmark is explicitly layout-aware. Questions can depend on figures, charts, diagrams, tables, multi-column layouts, and multi-page references linking text on one page to figures or tables elsewhere. This design choice distinguishes AraEngLongBench from purely text-serialized long-context benchmarks by retaining the visual structure of the document as part of the task definition.

4. Task formulation and evaluation protocol

The core task is open-ended visual question answering over long documents (Wang et al., 27 Jul 2025). A model receives document page images, or the entire document up to its context window, together with a question in English or Arabic, and must output a free-form answer. The benchmark stores, for each instance, the question, reference answer, evidence page indices, evidence type, justification, and validation information.

Question instances are categorized along two axes. The first is context length: SC for short context $(<100$ tokens), MC for medium context $(100\text{–}200)$ tokens, and LC for long context $O$0 tokens. The second is page span: SP for single-page questions and CP for cross-page questions. A separate UA category marks unanswerable cases, where the correct behavior is abstention or an explicit statement that the answer is not present in the document.

Accuracy is the only reported evaluation metric. It is computed per category for SC, MC, LC, SP, CP, and UA, and no F1, BLEU, or ROUGE score is reported for the main benchmark. In this paper, AraEngLongBench is used only for evaluation; no train–validation–test split or training protocol is described.

The evaluated models are run within their natural context-window limits: GPT-4o at 128K tokens, Gemini 2.0 Flash at 1M, Gemini 1.5 Pro at 2M, Qwen 2 VL at 32K, Qwen 2.5 VL at 128K, and Llama 3.2 VL at 128K. This places AraEngLongBench in the broader lineage of long-context evaluation frameworks that separate performance by context regime rather than treating a single aggregate score as sufficient, a design principle also emphasized in Ada-LEval (Wang et al., 2024).

5. Empirical findings

The paper evaluates three closed-source LVLMs—GPT-4o, Gemini-2.0 Flash, and Gemini-1.5 Pro—and three open-source models—Qwen 2 VL (7B), Qwen 2.5 VL (7B), and Llama 3.2 VL (11B)—on AraEngLongBench (Wang et al., 27 Jul 2025). The reported averages are uniformly far from saturation.

Model	Average	Brief note
Gemini-1.5 Pro	49.8%	Highest overall average
Gemini-2.0 Flash	49.6%	Close to Gemini-1.5 Pro
Qwen 2.5 VL	44.5%	Best open-source model
Qwen 2 VL	38.4%	Below Qwen 2.5 VL
Llama 3.2 VL	32.4%	Relatively poor reported result
GPT-4o	29.8%	Lowest reported average among listed major models

Gemini-1.5 Pro is described as state of the art on this benchmark, with especially strong English performance on long-context and cross-page questions. Gemini-2.0 Flash performs similarly, particularly on English single-page and cross-page cases. Qwen 2.5 VL is the best open-source model and improves substantially over Qwen 2 VL. GPT-4o underperforms the Gemini models overall, although the paper notes that it is relatively better on unanswerable questions.

A consistent pattern is that English accuracy exceeds Arabic accuracy for all evaluated models. The paper gives representative examples: for Qwen 2.5 VL, English LC is 54.9% while Arabic LC is 27.2%; for Gemini-1.5 Pro, English LC is 58.2% while Arabic LC is 42.3%. This supports the authors’ claim that Arabic long-context document understanding remains significantly harder.

The hardest category is unanswerable questions. Gemini-1.5 Pro records approximately 9% UA accuracy in both languages, Gemini-2.0 Flash roughly 9–10%, and Qwen 2.5 VL about 5–6%. GPT-4o is relatively stronger but still low at 18.1% in English and 12.5% in Arabic. The benchmark therefore functions not only as a test of retrieval and reasoning, but also as a test of calibrated abstention under long-context visual evidence.

Taken together, the results are used to justify the paper’s description of AraEngLongBench as “quite challenging.” Even the strongest evaluated models remain below 50–60% on most categories, while open-source systems degrade more sharply, especially on Arabic and unanswerable subsets.

6. Relation to other long-context benchmarks

AraEngLongBench is situated against several prior document-understanding and long-context benchmarks. The paper contrasts it with DocVQA, ChartQA, and InfographicVQA as primarily single-page and short-context, and with DUDE, MMLongBench-Doc, LongDocURL, and M-LongDoc as multi-page but shorter or more annotation-intensive (Wang et al., 27 Jul 2025). In comparative figures, AraEngLongBench is plotted at approximately 77 average pages and 84,000 average tokens, compared with about 47.5 pages and 21k tokens for MMLongBench.

In a broader benchmarking context, LongBench established a bilingual, multitask long-context benchmark for English and Chinese, standardized around $O$1 task triples and automatic evaluation across 21 datasets (Bai et al., 2023). Ada-LEval later argued for length-adaptable tasks and explicit per-length-bin evaluation, especially in ultra-long settings above 32k tokens (Wang et al., 2024). AraEngLongBench differs from both by focusing on visual, layout-rich, long-document question answering in Arabic and English, rather than on text-only multitask evaluation or length-adaptable synthetic-style tasks.

A plausible implication is that AraEngLongBench occupies a distinctive niche: it combines the bilingual ambition associated with LongBench, the long-context stress-testing motivation associated with Ada-LEval, and a fully automated evidence-grounded generation process tailored to page images, OCR, and layout analysis. Later work such as LongBench Pro emphasizes human–model collaborative construction over fully automated generation (Chen et al., 6 Jan 2026); this contrast further highlights the methodological choice made by AraEngLongBench to trade manual verification for scalable agentic synthesis.

7. Limitations, usage, and future directions

The paper identifies several limitations of AraEngLongBench (Wang et al., 27 Jul 2025). All questions and answers are generated automatically by LVLM-based agents rather than by humans, which may introduce stylistic biases or uneven coverage of reasoning types. Domain coverage is concentrated on academic and technical documents. Some question categories, including Argumentation, Data Retrieval, and Experimental Design, each account for less than 0.5% of the dataset. The paper also reports no formal human evaluation study or manual audit subset; quality control is entirely agent-based.

These limitations do not negate the benchmark’s utility, but they shape its interpretation. AraEngLongBench is best understood as an automatically generated, evidence-linked long-document benchmark whose principal contributions are scale, bilingual coverage, layout awareness, and difficulty calibration. It is not a manually curated gold-standard corpus in the conventional sense.

The benchmark and code are released through the repository https://github.com/wangk0b/Multi_Agentic_QA_Long_Doc.git. The paper presents three main uses: benchmarking LVLMs on English and Arabic long-context document understanding, fine-tuning or training models for long-context reading and cross-page reasoning, and ablation or failure analysis for unanswerable detection, cross-lingual robustness, and layout-aware reasoning.

Future work proposed by the authors includes exploring the relationship between document-understanding depth and the number of agent loops, adapting the framework to document-understanding tasks beyond question answering, improving cross-lingual generalization and confidence calibration, especially for Arabic and unanswerable cases, and scaling the same agentic pipeline to additional languages and domains. In that sense, AraEngLongBench is both a benchmark and a construction paradigm: a reusable automated workflow for generating long-context, bilingual, evidence-grounded document-understanding data under the constraints of modern LVLM evaluation.