---
title: 'DATABench: Table QA Benchmark'
url: https://www.emergentmind.com/topics/databench-b329257e-ce62-4f47-b9ae-2741187649a0
type: topic
---

# DATABench: Table QA Benchmark

DATABench, as used in SemEval 2025 Task 8, is a benchmark for question answering over tabular data in which systems must answer natural-language questions by retrieving and reasoning over structured tables rather than generating open-ended responses. It is described as a large, real-world benchmark with heterogeneous table shapes, data types, and domains, and it is evaluated in two settings: **DataBench QA**, where the full dataset is available, and **DataBench Lite QA**, where each dataset is reduced to a sampled context with at most 20 rows [2509.09234]. In the shared-task literature, DATABench serves as a stress test for schema understanding, row and column selection, numerical and categorical reasoning, and strict output formatting, with answers that may be `boolean`, `number`, `category`, `list[category]`, or `list[number]` [2508.00762].

## 1. Benchmark definition and problem formulation

The benchmark frames table question answering as a structured retrieval-and-reasoning problem. One system paper formalizes each instance as a pair $(T, Q)$, where $T$ is a structured table with columns $\{\mathrm{col}_1,\dots,\mathrm{col}_c\}$ and rows $\{\mathrm{row}_1,\dots,\mathrm{row}_r\}$, and $Q$ is a natural-language question whose answer may be directly present in the table or require simple computation over table values [2503.00435]. Another paper states the central assumption more operationally: the appropriate solution strategy is to translate the question into SQL, execute it on the provided table, and post-process the result into the required answer format, rather than answer directly from the table text [2509.09234].

This formulation distinguishes DATABench from free-form QA. The central difficulty is not merely linguistic interpretation, but the coordinated execution of schema understanding, evidence localization, filtering, aggregation, and output normalization. The shared-task papers repeatedly emphasize that even simple-looking questions may depend on a single key column hidden in a large table, several interacting columns, numeric aggregation, list extraction, or text-column operations such as uniqueness tests or substring checks [2503.00435].

## 2. Corpus structure, scale, and answer space

The benchmark is described as comprising **65 diverse real-world datasets** with heterogeneous table shapes and data types, and one open-source system paper reports **1,308 English question–answer pairs across 65 domains** [2509.09234] [2508.00762]. A separate system paper gives the shared-task split explicitly as **988 questions from 49 tables** for train, **320 questions from 16 tables** for dev, and **522 questions from 15 tables** for test [2503.00435]. These descriptions are complementary in that they jointly portray a benchmark built from many real tables and evaluated through a held-out test split.

Questions may return several output forms. The shared-task papers enumerate `boolean`, `number`, `category`, `list[category]`, and `list[number]`, while another description groups these more coarsely as numerical values, categorical values, Boolean judgments, and lists [2508.00762] [2509.09234]. This answer heterogeneity is important because it makes correctness depend not only on the selected evidence but also on exact answer typing and formatting.

A later large-scale TQA paper, "TableZoomer" [2509.01312], characterizes DataBench as an **English benchmark with 80 real-world tables** spanning **five domains**—business, health, social, sports, and travel—and **five QA types**. In its reported test split, that paper states that DataBench contains **15 datasets / 522 question-answer pairs**, with **513,359 average table cells** in the test set. This later characterization underscores the role of DATABench as a large-table benchmark, although the shared-task papers primarily emphasize the 65-dataset SemEval construction.

## 3. Evaluation settings and metrics

The benchmark defines two official task settings.

| Setting | Description |
|---|---|
| **DataBench QA** | Full dataset or full table setting |
| **DataBench Lite QA** | Sampled context with at most **20 rows** |

The Lite setting is described in two closely aligned ways: as a sampled context with at most **20 rows**, and as a simplified version built from **20 sampled entries** from the original dataset [2509.09234] [2508.00762]. In both cases, the point is to preserve table structure while reducing context size.

The shared-task papers describe the official metric as **Accuracy**, and systems are ranked by answer accuracy [2509.09234] [2508.00762]. By contrast, the AILS-NTUA system paper reports a **relaxed accuracy** metric in which semantically equivalent answers such as `"Yes"` for `True` are accepted and numeric answers are compared after truncation to **two decimals**; it also reports post-submission human review for official scores [2503.00435]. A later evaluation paper on TableZoomer reports **Exact Match (EM) Accuracy** using the official benchmark implementations and normalization typical for EM scoring [2509.01312]. This suggests that later implementations operationalize benchmark accuracy through normalized exact-match procedures.

## 4. Dominant methodological paradigms on DATABench

Work on DATABench converges on programmatic intermediate representations rather than direct answer generation. Representative approaches include an **NL-to-SQL agentic pipeline**, **zero-shot Python/Pandas code generation**, **two-stage language-to-code prompting with error fixing**, and the collaborative **TableZoomer** framework [2509.09234] [2508.00762] [2503.00435] [2509.01312].

| Paradigm | Representative components |
|---|---|
| **NL-to-SQL agentic pipeline** | Example selection; SQL generation; answer extraction and formatting; answer verification; iterative reprocessing/refinement |
| **Zero-shot Python/Pandas code generation** | Schema preprocessing; constrained code generation; controlled execution; iterative error correction |
| **Language-to-code prompting** | Main Module; CUAT inference; executable Python generation; Error-Fixing Module |
| **Collaborative agent framework** | Table Describer; Query Planner; Table Refiner; Code Generator; Answer Formatter |

The NL-to-SQL system of "Agentic LLMs for Question Answering over Tabular Data" uses **GPT-4o**, **GPT-4o-mini**, and **DeepSeek-v2:16b**, manually curates **25 example question–SQL pairs**, retrieves the **two most similar examples** using **text-embedding-ada-002** and **ChromaDB**, generates PostgreSQL-oriented SQL, executes it on a **SQLite database**, performs **Chain-of-Thought (CoT) prompting** for answer extraction, and applies a **GPT-4o verification prompt** followed by iterative repair if the answer fails format or relevance checks [2509.09234].

The ITUNLP system adopts a zero-shot **Python code generation framework** in which datasets are normalized, compact schemas are built with **5 unique sample values per column** and the total number of unique values, and open-source code models generate executable Pandas code. Execution failures trigger an error-correction loop that feeds the faulty code and error message back to the model for up to **two iterations** in the main system [2508.00762].

The AILS-NTUA system likewise treats DATABench as language-to-code translation, but uses a two-step prompt in which the model first infers **Columns Used and Answer Type (CUAT)** and then generates a Python function `answer(df: pd.DataFrame)`. If execution fails, an **Error-Fixing Module** rewrites the function using the original prompt and the interpreter’s error message [2503.00435].

TableZoomer extends the programmatic line of work to very large tables. Its pipeline replaces full-table verbalization with a schema representation, performs **query-aware table zooming** through **Column Selection**, **Entity Linking**, and **Table Zooming**, and generates executable Python under a **Program-of-Thoughts (PoT)** regime embedded in a **ReAct** loop. The paper explicitly states a complexity reduction from **\(O(M\times N)\)** for fully verbalized tables to **\(O(N)\)** for schema-based representation [2509.01312].

## 5. Reported results and competitive landscape

Across published systems, DATABench has functioned as a benchmark on which program execution substantially outperforms the organizer baseline. The baseline reported in multiple shared-task papers is **26.00** on DataBench QA and **27.00** on DataBench Lite QA [2509.09234] [2503.00435].

| System or model | DataBench QA / Subtask I | DataBench Lite QA / Subtask II |
|---|---:|---:|
| **Baseline** | 26.00 | 27.00 |
| **GPT-4o** | 70.50 | 71.65 |
| **DeepSeek-V3** | 84.67 | 80.84 |
| **DeepSeek-R1** | 84.09 | 85.05 |
| **Claude 3.5 Sonnet** | 85.63 | 87.93 |
| **Claude 3.5 Sonnet (official after human review)** | 89.85 | 88.89 |

The NL-to-SQL agentic submission reports **70.50%** on **DataBench QA** and **71.65%** on **DataBench Lite QA** with **GPT-4o**, ranking **10th** on the proprietary-model leaderboard for DataBench and **9th** for DataBench Lite [2509.09234]. ITUNLP reports the strongest open-source test performance with **DeepSeek-V3 = 84.67%** on Subtask I and **DeepSeek-R1 = 85.05%** on Subtask II, with official ranks of **8th place in Subtask I** and **6th place in Subtask II** among the **30 systems that beat the baseline** in the open-source category [2508.00762]. AILS-NTUA reports **Claude 3.5 Sonnet** at **85.63%** on DataBench test and **87.93%** on DataBench Lite test, with official post-review scores of **89.85%** and **88.89%**, yielding **1st place in both subtasks in the proprietary-model category** [2503.00435].

Post-competition work uses DATABench to evaluate large-table reasoning architectures. TableZoomer reports that, with **Qwen3-8B**, conventional PoT yields **67.82%** average EM on DataBench, whereas TableZoomer reaches **87.16%**, an absolute gain of **19.34%**; it also reports gains for **Qwen3-14B**, **Qwen2.5-32B**, **Qwen3-32B**, **QwQ-32B**, **TeleChat2.5-35B**, and **Llama3.1-70B** [2509.01312]. A plausible implication is that DATABench has become not only a shared-task benchmark but also a post hoc testbed for large-table localization and reasoning strategies.

## 6. Failure modes, future directions, and nomenclature

Published system papers identify several recurring failure modes. The NL-to-SQL submission explicitly notes **complex numerical reasoning errors**, especially in ranking, aggregation, and filtering chains, and **categorical misclassification**, where the system returns a semantically related but incorrect category [2509.09234]. The ITUNLP paper groups failures into **Runtime**, **Syntax**, and **Degenerate Loop** errors, and states that runtime errors dominate initially; it further reports that the iterative correction loop reduces code execution errors by **nearly half on average** [2508.00762]. The AILS-NTUA paper adds that many remaining mistakes are semantic rather than syntactic, including wrong column selection, confusing maximum with most frequent, and incorrect question interpretation [2503.00435].

The proposed remedies are correspondingly structural rather than purely model-scaling based. The "Future Work" section of the NL-to-SQL paper suggests **stronger contextual reasoning**, **better token representations for tables**, **more explicit schema understanding**, and **improved verification/refinement** [2509.09234]. TableZoomer’s ablation results similarly identify **Table Schema representation** and **Table Zooming** as the most critical contributors, with **Entity Linking** and **ReAct** providing smaller but consistent gains [2509.01312]. This suggests that DATABench rewards pipelines that explicitly separate schema compression, evidence localization, executable reasoning, and answer formatting.

The name itself requires disambiguation. In 2025–2026 literature, **DATABench** also denotes a **benchmark framework for data lakes** that targets structured, semi-structured, and unstructured data, with workloads such as keyword search over **IMDb**, **DBLP**, and **Apache HTTP logs** [2601.19176]. It also denotes an adversarial benchmark for **dataset auditing** consisting of **17 evasion attacks**, **5 forgery attacks**, and **9 representative auditing methods** [2507.05622]. These are unrelated to the SemEval table-QA benchmark. Historically adjacent but also distinct is **BigDataBench**, a scalable and unified benchmark suite for big data and AI whose methodology is organized around **eight data motifs** and whose scope includes **13 representative real-world data sets**, **47 benchmarks**, **7 workload types**, and **17 software stacks** [1802.08254].

Source: https://www.emergentmind.com/topics/databench-b329257e-ce62-4f47-b9ae-2741187649a0