DATABench: Table QA Benchmark

• DATABench is a benchmark for table question answering that tests schema understanding, evidence localization, and precise answer formatting.
• It evaluates systems over both full and context-reduced settings using diverse real-world datasets with heterogeneous table structures.
• Key challenges include complex numerical reasoning, categorical misclassification, and the need for iterative error correction in programmatic approaches.

DATABench, as used in SemEval 2025 Task 8, is a benchmark for question answering over tabular data in which systems must answer natural-language questions by retrieving and reasoning over structured tables rather than generating open-ended responses. It is described as a large, real-world benchmark with heterogeneous table shapes, data types, and domains, and it is evaluated in two settings: DataBench QA, where the full dataset is available, and DataBench Lite QA, where each dataset is reduced to a sampled context with at most 20 rows (Tyagi et al., 11 Sep 2025). In the shared-task literature, DATABench serves as a stress test for schema understanding, row and column selection, numerical and categorical reasoning, and strict output formatting, with answers that may be boolean, number, category, list[category], or list[number] (Site et al., 1 Aug 2025).

1. Benchmark definition and problem formulation

The benchmark frames table question answering as a structured retrieval-and-reasoning problem. One system paper formalizes each instance as a pair $(T, Q)$, where $T$ is a structured table with columns $\{\mathrm{col}_1,\dots,\mathrm{col}_c\}$ and rows $\{\mathrm{row}_1,\dots,\mathrm{row}_r\}$, and $Q$ is a natural-language question whose answer may be directly present in the table or require simple computation over table values (Evangelatos et al., 1 Mar 2025). Another paper states the central assumption more operationally: the appropriate solution strategy is to translate the question into SQL, execute it on the provided table, and post-process the result into the required answer format, rather than answer directly from the table text (Tyagi et al., 11 Sep 2025).

This formulation distinguishes DATABench from free-form QA. The central difficulty is not merely linguistic interpretation, but the coordinated execution of schema understanding, evidence localization, filtering, aggregation, and output normalization. The shared-task papers repeatedly emphasize that even simple-looking questions may depend on a single key column hidden in a large table, several interacting columns, numeric aggregation, list extraction, or text-column operations such as uniqueness tests or substring checks (Evangelatos et al., 1 Mar 2025).

2. Corpus structure, scale, and answer space

The benchmark is described as comprising 65 diverse real-world datasets with heterogeneous table shapes and data types, and one open-source system paper reports 1,308 English question–answer pairs across 65 domains (Tyagi et al., 11 Sep 2025, Site et al., 1 Aug 2025). A separate system paper gives the shared-task split explicitly as 988 questions from 49 tables for train, 320 questions from 16 tables for dev, and 522 questions from 15 tables for test (Evangelatos et al., 1 Mar 2025). These descriptions are complementary in that they jointly portray a benchmark built from many real tables and evaluated through a held-out test split.

Questions may return several output forms. The shared-task papers enumerate boolean, number, category, list[category], and list[number], while another description groups these more coarsely as numerical values, categorical values, Boolean judgments, and lists (Site et al., 1 Aug 2025, Tyagi et al., 11 Sep 2025). This answer heterogeneity is important because it makes correctness depend not only on the selected evidence but also on exact answer typing and formatting.

A later large-scale TQA paper, "TableZoomer" (Xiong et al., 1 Sep 2025), characterizes DataBench as an English benchmark with 80 real-world tables spanning five domains—business, health, social, sports, and travel—and five QA types. In its reported test split, that paper states that DataBench contains 15 datasets / 522 question-answer pairs, with 513,359 average table cells in the test set. This later characterization underscores the role of DATABench as a large-table benchmark, although the shared-task papers primarily emphasize the 65-dataset SemEval construction.

3. Evaluation settings and metrics

The benchmark defines two official task settings.

Setting	Description
DataBench QA	Full dataset or full table setting
DataBench Lite QA	Sampled context with at most 20 rows

The Lite setting is described in two closely aligned ways: as a sampled context with at most 20 rows, and as a simplified version built from 20 sampled entries from the original dataset (Tyagi et al., 11 Sep 2025, Site et al., 1 Aug 2025). In both cases, the point is to preserve table structure while reducing context size.

The shared-task papers describe the official metric as Accuracy, and systems are ranked by answer accuracy (Tyagi et al., 11 Sep 2025, Site et al., 1 Aug 2025). By contrast, the AILS-NTUA system paper reports a relaxed accuracy metric in which semantically equivalent answers such as "Yes" for True are accepted and numeric answers are compared after truncation to two decimals; it also reports post-submission human review for official scores (Evangelatos et al., 1 Mar 2025). A later evaluation paper on TableZoomer reports Exact Match (EM) Accuracy using the official benchmark implementations and normalization typical for EM scoring (Xiong et al., 1 Sep 2025). This suggests that later implementations operationalize benchmark accuracy through normalized exact-match procedures.

4. Dominant methodological paradigms on DATABench

Work on DATABench converges on programmatic intermediate representations rather than direct answer generation. Representative approaches include an NL-to-SQL agentic pipeline, zero-shot Python/Pandas code generation, two-stage language-to-code prompting with error fixing, and the collaborative TableZoomer framework (Tyagi et al., 11 Sep 2025, Site et al., 1 Aug 2025, Evangelatos et al., 1 Mar 2025, Xiong et al., 1 Sep 2025).

Paradigm	Representative components
NL-to-SQL agentic pipeline	Example selection; SQL generation; answer extraction and formatting; answer verification; iterative reprocessing/refinement
Zero-shot Python/Pandas code generation	Schema preprocessing; constrained code generation; controlled execution; iterative error correction
Language-to-code prompting	Main Module; CUAT inference; executable Python generation; Error-Fixing Module
Collaborative agent framework	Table Describer; Query Planner; Table Refiner; Code Generator; Answer Formatter

The NL-to-SQL system of "Agentic LLMs for Question Answering over Tabular Data" uses GPT-4o, GPT-4o-mini, and DeepSeek-v2:16b, manually curates 25 example question–SQL pairs, retrieves the two most similar examples using text-embedding-ada-002 and ChromaDB, generates PostgreSQL-oriented SQL, executes it on a SQLite database, performs Chain-of-Thought (CoT) prompting for answer extraction, and applies a GPT-4o verification prompt followed by iterative repair if the answer fails format or relevance checks (Tyagi et al., 11 Sep 2025).

The ITUNLP system adopts a zero-shot Python code generation framework in which datasets are normalized, compact schemas are built with 5 unique sample values per column and the total number of unique values, and open-source code models generate executable Pandas code. Execution failures trigger an error-correction loop that feeds the faulty code and error message back to the model for up to two iterations in the main system (Site et al., 1 Aug 2025).

The AILS-NTUA system likewise treats DATABench as language-to-code translation, but uses a two-step prompt in which the model first infers Columns Used and Answer Type (CUAT) and then generates a Python function answer(df: pd.DataFrame). If execution fails, an Error-Fixing Module rewrites the function using the original prompt and the interpreter’s error message (Evangelatos et al., 1 Mar 2025).

TableZoomer extends the programmatic line of work to very large tables. Its pipeline replaces full-table verbalization with a schema representation, performs query-aware table zooming through Column Selection, Entity Linking, and Table Zooming, and generates executable Python under a Program-of-Thoughts (PoT) regime embedded in a ReAct loop. The paper explicitly states a complexity reduction from $O(M\times N)$ for fully verbalized tables to $O(N)$ for schema-based representation (Xiong et al., 1 Sep 2025).

5. Reported results and competitive landscape

Across published systems, DATABench has functioned as a benchmark on which program execution substantially outperforms the organizer baseline. The baseline reported in multiple shared-task papers is 26.00 on DataBench QA and 27.00 on DataBench Lite QA (Tyagi et al., 11 Sep 2025, Evangelatos et al., 1 Mar 2025).

System or model	DataBench QA / Subtask I	DataBench Lite QA / Subtask II
Baseline	26.00	27.00
GPT-4o	70.50	71.65
DeepSeek-V3	84.67	80.84
DeepSeek-R1	84.09	85.05
Claude 3.5 Sonnet	85.63	87.93
Claude 3.5 Sonnet (official after human review)	89.85	88.89

The NL-to-SQL agentic submission reports 70.50% on DataBench QA and 71.65% on DataBench Lite QA with GPT-4o, ranking 10th on the proprietary-model leaderboard for DataBench and 9th for DataBench Lite (Tyagi et al., 11 Sep 2025). ITUNLP reports the strongest open-source test performance with DeepSeek-V3 = 84.67% on Subtask I and DeepSeek-R1 = 85.05% on Subtask II, with official ranks of 8th place in Subtask I and 6th place in Subtask II among the 30 systems that beat the baseline in the open-source category (Site et al., 1 Aug 2025). AILS-NTUA reports Claude 3.5 Sonnet at 85.63% on DataBench test and 87.93% on DataBench Lite test, with official post-review scores of 89.85% and 88.89%, yielding 1st place in both subtasks in the proprietary-model category (Evangelatos et al., 1 Mar 2025).

Post-competition work uses DATABench to evaluate large-table reasoning architectures. TableZoomer reports that, with Qwen3-8B, conventional PoT yields 67.82% average EM on DataBench, whereas TableZoomer reaches 87.16%, an absolute gain of 19.34%; it also reports gains for Qwen3-14B, Qwen2.5-32B, Qwen3-32B, QwQ-32B, TeleChat2.5-35B, and Llama3.1-70B (Xiong et al., 1 Sep 2025). A plausible implication is that DATABench has become not only a shared-task benchmark but also a post hoc testbed for large-table localization and reasoning strategies.

6. Failure modes, future directions, and nomenclature

Published system papers identify several recurring failure modes. The NL-to-SQL submission explicitly notes complex numerical reasoning errors, especially in ranking, aggregation, and filtering chains, and categorical misclassification, where the system returns a semantically related but incorrect category (Tyagi et al., 11 Sep 2025). The ITUNLP paper groups failures into Runtime, Syntax, and Degenerate Loop errors, and states that runtime errors dominate initially; it further reports that the iterative correction loop reduces code execution errors by nearly half on average (Site et al., 1 Aug 2025). The AILS-NTUA paper adds that many remaining mistakes are semantic rather than syntactic, including wrong column selection, confusing maximum with most frequent, and incorrect question interpretation (Evangelatos et al., 1 Mar 2025).

The proposed remedies are correspondingly structural rather than purely model-scaling based. The "Future Work" section of the NL-to-SQL paper suggests stronger contextual reasoning, better token representations for tables, more explicit schema understanding, and improved verification/refinement (Tyagi et al., 11 Sep 2025). TableZoomer’s ablation results similarly identify Table Schema representation and Table Zooming as the most critical contributors, with Entity Linking and ReAct providing smaller but consistent gains (Xiong et al., 1 Sep 2025). This suggests that DATABench rewards pipelines that explicitly separate schema compression, evidence localization, executable reasoning, and answer formatting.

The name itself requires disambiguation. In 2025–2026 literature, DATABench also denotes a benchmark framework for data lakes that targets structured, semi-structured, and unstructured data, with workloads such as keyword search over IMDb, DBLP, and Apache HTTP logs (Lyu et al., 27 Jan 2026). It also denotes an adversarial benchmark for dataset auditing consisting of 17 evasion attacks, 5 forgery attacks, and 9 representative auditing methods (Shao et al., 8 Jul 2025). These are unrelated to the SemEval table-QA benchmark. Historically adjacent but also distinct is BigDataBench, a scalable and unified benchmark suite for big data and AI whose methodology is organized around eight data motifs and whose scope includes 13 representative real-world data sets, 47 benchmarks, 7 workload types, and 17 software stacks (Gao et al., 2018).