AraTable: Arabic Tabular Reasoning Benchmark

• AraTable is a benchmark specifically designed to assess Arabic LLMs on tabular reasoning challenges using curated tables, hybrid generation, and native validation.
• The benchmark includes 615 human-validated items covering direct QA, reasoning QA, and fact verification from diverse sources such as Arabic Wikipedia and real-world datasets.
• Empirical findings show that while direct QA is largely solved, reasoning and complex inference remain challenging, highlighting gaps in arithmetic and multi-step processing.

AraTable is a benchmark specifically designed to evaluate LLMs on reasoning and understanding over Arabic tabular data. It targets three core tasks—direct question answering, fact verification, and complex reasoning—and was proposed to fill a documented gap: prior Arabic benchmarks focused on general NLP tasks such as sentiment analysis, natural language inference, and question answering over text, but not on structured or tabular input. The benchmark combines curated Arabic tables from multiple sources, a hybrid construction pipeline in which GPT‑4 and GPT‑4o generate candidate items that are then filtered and validated by native Arabic speakers, and an automated evaluation framework based on Assisted Self-Deliberation that is intended to align closely with human judgment (Alshaikh et al., 24 Jul 2025).

1. Motivation, scope, and linguistic setting

AraTable was proposed from a three-part motivation. First, table reasoning is treated as a distinct problem from ordinary text understanding: the model must interpret a two-dimensional structure rather than a linear sequence, align questions with rows, columns, and headers, and often perform symbolic and numerical operations such as aggregation, comparison, differences, ratios, and temporal ordering. Second, Arabic introduces additional challenges, including rich morphology and clitics, complex grammar and syntactic variation, right-to-left script directionality, variable punctuation and diacritics, and dialectal variation with Arabic–English code-switching in real data. Third, the existing Arabic benchmark landscape did not measure how well LLMs reason over Arabic tables, even though English tabular benchmarks were already well established (Alshaikh et al., 24 Jul 2025).

The benchmark therefore defines a multilingual-LLM evaluation problem that is not reducible to generic Arabic QA. High direct-answer performance would indicate that a model can parse Arabic CSV tables and align questions with cell values, whereas failures on reasoning and verification would indicate deficits in multi-step inference, arithmetic handling, and table-grounded entailment. This suggests that AraTable is not merely a language benchmark; it is also a structural-reasoning benchmark in which Arabic linguistic phenomena and tabular reasoning demands are jointly stressed.

2. Corpus design and hybrid dataset construction

AraTable contains 41 tables drawn from three sources and normalized into CSV format while preserving original Arabic names and content. For very large tables, the construction pipeline limited the input to max 40 rows, mainly to respect Jais’s context limits; smaller tables were kept as-is. The final benchmark contains 5 Direct QA, 5 Reasoning QA, and 5 Fact verification items per table, giving 615 human-validated items in total because $41 \times 15 = 615$ (Alshaikh et al., 24 Jul 2025).

Source	Tables	Typical content
Arabic Wikipedia	26	Tourism, sports, economy, demographics, education, transportation
Real-world data	10	Kaggle datasets and Saudi open data portals
LLM-generated tables	5	Student performance, payroll, expenses, leave records, product costs

The Arabic Wikipedia portion includes domains such as world museums, mosques, UNESCO sites, World Cup winners, Olympic host cities, company revenues, biggest profits and losses, population, transportation statistics, global university rankings, largest universities, busiest airports, and fastest cars. These tables range from small tables with 10 rows to substantially larger tables such as a 121×13 academic ranking table and a 422×7 table of World Cup-winning players. The real-world component is deliberately more heterogeneous: it includes Hotel Arabic Reviews (HARD) 93,700×7, Saudi used cars 8,248×14, Jamalon Arabic books 8,986×11, Saudi real estate (AQAR) 3,718×24, and additional Saudi open-data tables such as train schedules, road lengths, events, tenders, meteorological data, and household book genres. The synthetic component covers scenarios that were underrepresented in real data, with dimensions around 20–30 rows and 6–10 columns (Alshaikh et al., 24 Jul 2025).

Construction proceeded in two phases. In the first phase, GPT‑4 or GPT‑4o received a table in CSV form and a task-specific prompt, and produced 10 direct QA questions, 10 reasoning questions, and 10 fact-verification statements per table. In the second phase, native Arabic speakers reviewed all generated items, verified answers against the table using code, Excel, and manual checks, corrected ambiguous or poorly phrased instances, ensured that every question was answerable solely from the table, and then selected 5 items per task to maximize coverage of rows, columns, reasoning types, and difficulty. The released benchmark is publicly available at https://github.com/rana-alshaikh/AraTable-Benchmark (Alshaikh et al., 24 Jul 2025).

AraTable’s reasoning subset emphasizes numeric and comparative operations. The reported breakdown is 79 Mathematical Reasoning, 67 Comparative Reasoning, 42 Logical Reasoning, and 17 Temporal Reasoning questions. A plausible implication is that benchmark scores are especially sensitive to arithmetic and cross-row comparison rather than to temporal ordering alone.

3. Task taxonomy and annotation schema

AraTable organizes evaluation into three task families of increasing complexity. Direct Question Answering is a simple lookup task: the model extracts a specific value directly from the table without inference beyond locating the correct cell. Reasoning Question Answering requires multi-step inference and is explicitly labeled by reasoning type: temporal reasoning, mathematical reasoning, comparative reasoning, or logical or conditional reasoning. Fact Verification is formulated as table-based entailment: the input is an Arabic declarative statement about the table, and the output is True or False depending on whether the table supports it (Alshaikh et al., 24 Jul 2025).

The reasoning task is structurally richer than the direct QA task. For each reasoning item, the internal CSV includes the Arabic question, a binary label for “Reasoning” or “Non-Reasoning,” a reasoning-type label such as “Mathematical reasoning,” the answer, and a short description of how to derive it. Those descriptions are used internally for human validation and are not part of the benchmark input. In the fact-verification task, the generation prompt requires 10 statements per table, with 5 “True” and 5 “False”, and each statement must involve complex reasoning such as combining multiple conditions, cross-referencing multiple columns, or logical deduction (Alshaikh et al., 24 Jul 2025).

This task design makes AraTable broader than a simple table lookup corpus. The benchmark spans surface retrieval, typed reasoning, and verification under free-form model outputs. In the broader Table QA literature, the survey literature classifies systems into semantic-parsing-based, generative, extractive, matching-based, and retriever-reader-based families (Jin et al., 2022). AraTable is compatible with all of those evaluation regimes, but its task mixture especially foregrounds the failure modes that arise when Arabic structural understanding, arithmetic manipulation, and entailment must be handled in a single benchmark.

4. Human evaluation rubric and Assisted Self-Deliberation

AraTable does not rely on exact string match because models were allowed to answer freely. Instead, it uses a two-layer evaluation framework: human judges with a detailed rubric, and LLMs-as-judges improved by an Assisted Self-Deliberation (ASD) mechanism. Human evaluation involved nine native Arabic speakers divided into three rounds: one group labeled model answers as correct or incorrect with respect to the gold answer, a second group reviewed and corrected mislabels, and a third group performed a final review (Alshaikh et al., 24 Jul 2025).

The rubric is deliberately relaxed but tightly table-grounded. It accepts semantic equivalence over exact match, so answers such as “40”, “حوالي 40”, and “أربعون سنة” can be treated as equivalent. For fact verification, clear affirmative and negative variants are accepted; the benchmark explicitly lists True, Yes, صحيح, نعم, and بيان صحيح as acceptable positive forms, and False, No, خطأ, غير صحيح, and بيان كاذب as acceptable negative forms. Numeric answers are accepted if they are within ±0.005 of the gold answer, and formatting variants such as 25% versus 0.25 or unit-preserving paraphrases such as 38000 versus 38 ألف are allowed. For list answers, order, punctuation, and minor formatting are ignored if the item set is the same. A particularly strict rule applies to Arabic entity names: if the gold answer is an Arabic entity name, only Arabic forms are accepted, because an English form may indicate that the model relied on latent world knowledge rather than on the provided table (Alshaikh et al., 24 Jul 2025).

The automated layer uses Qwen and GPT‑4o as judges. Each judge sees the table, the question, the ground-truth answer, and the evaluation rubric, but not the original model identities; systems are anonymized as Model 1–5. ASD then operates in four steps. First, Qwen and GPT‑4o independently score all answers as True or False. Second, a disagreement report is prepared for each judge, listing cases where its label differs from the other judge’s label. Third, for each disagreement case, the judge performs self-deliberation: it justifies its original decision, explains why the other judge might have chosen the opposite label, identifies which rubric rules support or contradict both decisions, and decides whether to maintain or revise its label. Crucially, each judge sees only the other judge’s label, not the other judge’s reasoning. Fourth, the revised labels are compared against the human baseline, and the paper reports signed differences Δ vs human before and after ASD (Alshaikh et al., 24 Jul 2025).

The significance of ASD is methodological as much as evaluative. The paper reports that after ASD, Qwen’s scores align almost perfectly with human accuracy: on Wikipedia and real-world subsets, Δ=0.00 for all models, while on LLM-generated tables the residuals are 0.00 to +0.04. GPT‑4o also improves, but remains less perfectly aligned, including a persistent −0.05 underestimate for Jais-Full on the Wikipedia and real-world subsets. The paper therefore treats a single LLM judge (Qwen) plus ASD as a practical replacement for human judging on AraTable (Alshaikh et al., 24 Jul 2025).

5. Experimental setup and empirical findings

AraTable evaluates Llama 3.3 70B, DeepSeek‑V3, Mistral Large, and Jais 70B in two forms: the original verbose output and a concise answer extracted by human annotators. All systems are tested in a zero-shot setting with a prompt that includes a task description, the table in CSV format, and the Arabic question. No strict output format is imposed except that fact-verification prompts explicitly request True or False. Because Jais often produces long, verbose, or digressive responses, two annotators independently extracted core answers to create the “Jais concise” variant (Alshaikh et al., 24 Jul 2025).

The main empirical pattern is consistent across all three table sources: Direct QA is relatively easy, fact verification is intermediate, and reasoning is the principal bottleneck. On Wikipedia tables, DeepSeek‑V3 is the best overall system with 0.79 overall accuracy, including 0.96 on Direct QA, 0.81 on Fact Verification, and 0.59 on Reasoning. On real-world tables, DeepSeek‑V3 again leads with 0.75 overall, while reasoning drops to 0.48; Llama’s reasoning falls to 0.20, and Jais remains weak at 0.14. On LLM-generated tables, DeepSeek‑V3 reaches 1.00 on Direct QA and 0.75 overall, while DeepSeek‑V3 and Mistral Large tie at 0.48 on Reasoning (Alshaikh et al., 24 Jul 2025).

Table source	Best overall model	Key result
Wikipedia	DeepSeek‑V3	0.79 overall; 0.59 reasoning
Real-world	DeepSeek‑V3	0.75 overall; 0.48 reasoning
LLM-generated	DeepSeek‑V3	0.75 overall; 1.00 direct QA

Several benchmark-wide conclusions follow directly from these numbers. First, direct QA is relatively solved for the top models, with performance at or above 0.9 and, in one synthetic setting, 1.0. Second, reasoning remains difficult, with best performance below 0.6 everywhere and as low as 0.20 for Llama on real-world reasoning. Third, fact verification remains non-trivial, with top-model accuracy generally in the 0.6–0.8 range. Fourth, Jais 70B, despite being Arabic-centric, is substantially weaker than multilingual frontier models, which the paper interprets as evidence that Arabic language specialization alone is insufficient without exposure to structured/tabular patterns and reasoning tasks in pre-training or fine-tuning (Alshaikh et al., 24 Jul 2025).

The source effect is also important. Real-world tables are explicitly described as messier and more heterogeneous than Wikipedia tables, with different schemas, numeric-heavy columns, and domain-specific structure, and the reported results show the sharpest degradation on this subset. This suggests that AraTable is not dominated by Wikipedia-style regularity; it also probes schema heterogeneity and operational tabular reasoning in more realistic Arabic data settings.

6. Significance, limitations, and relation to broader table research

AraTable’s stated contributions are fivefold: a benchmark dataset for Arabic tabular QA with 615 high-quality instances; a hybrid creation method combining LLM generation with rigorous human vetting; a three-source design spanning Wikipedia, real-world datasets, and synthetic tables; an automatic evaluation framework based on ASD; and a public release of dataset and code for future work (Alshaikh et al., 24 Jul 2025). The benchmark therefore functions both as an evaluation suite and as an infrastructure contribution for Arabic structured-data research.

The benchmark also has explicit limitations. The authors tested only a limited set of open-source, Arabic-capable models and did not include GPT‑4-class proprietary systems as QA models. Evaluation is zero-shot only; there are no few-shot or fine-tuned experiments. Tables were truncated to ≤ 40 rows because of Jais’s context limitation, so AraTable does not yet evaluate very large tables or multi-table reasoning. The benchmark’s 41 tables and 615 QAs are substantial for Arabic tabular QA but still smaller than large English benchmarks, and some reasoning types—especially temporal reasoning—remain underrepresented (Alshaikh et al., 24 Jul 2025).

The paper’s future directions are correspondingly concrete: few-shot and fine-tuned settings; richer table structures such as multi-table, hierarchical, or relational schemas; more domains and more reasoning types; longer tables; and improved prompting and reasoning scaffolds, including Chain-of-Table, Tab-CoT, and symbolic decomposition (Alshaikh et al., 24 Jul 2025). In the surrounding literature, several technical directions are directly relevant to systems that might later be evaluated on AraTable. T‑RAG proposes an end-to-end retrieval-augmented generation architecture for open-domain Table QA (Pan et al., 2022). ReAcTable shows that ReAct-style tool use with SQL and Python executors can improve table QA without training a new model (Zhang et al., 2023). ALTER targets large-table reasoning through query and table augmentation and reports strong performance in large-table scenarios (Zhang et al., 2024). TAMO treats tables as an independent modality integrated with LLMs and reports an average relative gain of 42.65% across several benchmarks (Li et al., 30 Nov 2025). ASTRA reconstructs complex tables into Logical Semantic Trees and combines textual navigation with symbolic execution for complex table QA (Guo et al., 10 Apr 2026).

AraTable does not implement those methods; it provides an Arabic benchmark on which such approaches could be compared. This suggests a broader role for the benchmark: as a focal point for bringing Arabic table reasoning into the same methodological conversation that already includes retrieval-augmented generation, tool-augmented execution, large-table filtering, multimodal structure encoding, and semantic-tree representations. In that sense, AraTable marks the transition of Arabic tabular understanding from an unmeasured capability to a benchmarked research problem (Alshaikh et al., 24 Jul 2025).