SemEval 2025 Task 8: DataBench Overview

• DataBench is a benchmark for evaluating natural-language question answering over structured, single-table data with diverse, realistic datasets.
• The task challenges systems with operations like filtering, aggregation, and multi-step reasoning on tables that may contain noisy and heterogeneous data.
• Evaluation relies on relaxed accuracy metrics and answer normalization for booleans, numbers, and list types in both full-table and Lite settings.

Searching arXiv for the SemEval 2025 Task 8 / DataBench shared task papers and related work. SemEval 2025 Task 8, commonly referred to as DataBench in the cited system papers, is a shared task on question answering over tabular data in which a system must answer a natural-language question $Q$ given a structured table $T$. In the formulations reported by participating teams, each instance is a pair $(T, Q)$, where $T$ contains columns and rows with values that may be numeric, categorical, textual, date-valued, boolean, or missing, and the required answer must be one of خمسة accepted types: boolean, category or string, number, list[category or string], or listnumber. Across the participating-system descriptions, the task is characterized as an evaluation of realistic table reasoning under domain diversity and large-table conditions, with a full-table setting and a reduced-row setting called DataBench Lite (Evangelatos et al., 1 Mar 2025, Site et al., 1 Aug 2025). The task has also been discussed in connection with dynamic benchmark construction for shared evaluations, where document-grounded generation and contamination control are proposed as ways to refresh evaluation sets in rapidly changing domains (Shashidhar et al., 2 Apr 2025).

1. Task definition and benchmark scope

SemEval 2025 Task 8 assesses whether a model can answer a natural-language question over a single structured table. One detailed formalization states that each table $T$ has $c$ columns $\{col_1, \dots, col_c\}$ and $r$ rows $\{row_1, \dots, row_r\}$, with row entries $\{row_{i,1}, \dots, row_{i,c}\}$, and that the target answer $T$0 belongs to one of the accepted answer types enumerated above (Evangelatos et al., 1 Mar 2025). Participating papers consistently describe the benchmark as single-table rather than multi-table; one zero-shot system explicitly notes that “no multi-table joins” are required in the task (Site et al., 1 Aug 2025).

A recurring emphasis in the system papers is that DataBench was designed to move beyond prior benchmarks with narrow topical coverage or unrealistically small tables. One description attributes to DataBench “65 real-world datasets from varied domains” and tables ranging “up to several hundred thousand rows,” with the aim of making evaluation closer to practical analytics scenarios (Evangelatos et al., 1 Mar 2025). Another team describes the benchmark as aggregating “65 real-world datasets and over 1,300 manually crafted question–answer pairs across diverse domains,” with heterogeneous CSV-like tables containing numeric, categorical, and textual columns, as well as missing values and inconsistent encodings (Lagos et al., 28 May 2025). This suggests that the benchmark’s central challenge is not merely symbolic aggregation, but robust reasoning under schema heterogeneity, noisy values, and varied answer types.

The system papers also describe the repertoire of operations required by the task. These include filtering rows, aggregations such as count, mean, and max, grouping and ranking, categorical or text lookup, simple string predicates, arithmetic over numeric columns, and multi-step reasoning that may combine several of these operations (Evangelatos et al., 1 Mar 2025, Lagos et al., 28 May 2025). In practical terms, the task sits at the intersection of semantic parsing, program synthesis, and table-centered reasoning.

2. Dataset composition, splits, and subtasks

A detailed account of the dataset organization reports the following train, development, and test structure: 988 questions over 49 tables for train, 320 questions over 16 tables for dev, and 522 questions over 15 tables for test, for a train-plus-dev total of 1,308 questions over 65 tables (Evangelatos et al., 1 Mar 2025). MRT reports the same split sizes in its own description of the released data (Lagos et al., 28 May 2025). These matching counts across independent submissions provide a consistent picture of the benchmark scale.

Two subtasks are described in the participating papers. Subtask I is “DataBench QA,” in which full tables of any size are available with each question (Evangelatos et al., 1 Mar 2025). Subtask II is “DataBench Lite QA,” where the same questions are paired with sampled tables containing at most 20 rows, thereby reducing input length and shifting emphasis toward reasoning over smaller contexts (Evangelatos et al., 1 Mar 2025). Another paper describes Subtask II as a simplified version of Subtask I and notes that a “sample_answer” is given for 20 sampled entries from the original dataset (Site et al., 1 Aug 2025). Because the two descriptions are not identical in wording, the safest synthesis is that Lite uses sampled small-table views intended to simplify context size while preserving the core QA setting.

The system literature also records schema information available in the released data. For train and dev, organizers precompute the columns required to answer each question, their types, and the expected answer type; the test set omits these annotations to assess generalization (Evangelatos et al., 1 Mar 2025). ITUNLP likewise reports that train and dev include fields such as answer type, columns_used, column_types, and dataset name, whereas test includes only question and dataset (Site et al., 1 Aug 2025). This division reflects a conventional shared-task design in which development aids are available for analysis and prompt construction, but test evaluation requires generalization without privileged schema-to-answer mappings.

3. Evaluation protocol and answer semantics

The most detailed description of official scoring comes from the AILS-NTUA paper, which states that the task uses a relaxed accuracy metric provided by the databench_eval package (Evangelatos et al., 1 Mar 2025). According to that description, the metric normalizes booleans by case and format, compares numbers after truncation to two decimal places, and applies type-aware equivalences for categorical values (Evangelatos et al., 1 Mar 2025). The same paper presents the evaluation in exact-match form after normalization:

$T$1

For list outputs, it specifies that list[category] and list[number] are scored by equality after normalization, and that token-level precision, recall, and $T$2 are not used by the organizers (Evangelatos et al., 1 Mar 2025).

Other participating papers use the simpler term “Accuracy” for the official metric. ITUNLP defines it as

$T$3

and reports leaderboard rankings under that metric for both subtasks (Site et al., 1 Aug 2025). MRT also reports Accuracy as the official metric, while noting that the paper itself does not reproduce the exact details of the organizer’s script and instead focuses on its own answer-normalization layers designed to align outputs with evaluator expectations (Lagos et al., 28 May 2025). Taken together, these accounts indicate that official evaluation is answer-level rather than rationale-level, with normalization rules playing a substantial role in numeric, boolean, and typed-list comparisons.

A practical implication, repeatedly visible in system design, is that answer formatting is not a peripheral concern. Several systems therefore add explicit output