SemEval 2025 Task 8: Tabular QA Challenge

• The paper introduces a novel language-to-code framework that converts table–question pairs into executable Python code using Pandas, achieving significant accuracy gains through iterative error correction and prompt tuning.
• SemEval 2025 Task 8 is a shared task that standardizes tabular QA over heterogeneous data, emphasizing schema extraction, multi-step reasoning, and handling of diverse answer types.
• The task underscores the effectiveness of modular pipelines combining schema normalization, structured prompt construction, and LLM-driven code generation to optimize QA performance.

SemEval 2025 Task 8 is a shared task focused on question answering (QA) over tabular data, evaluated on the DataBench benchmark. It operationalizes tabular QA as a language-to-code problem, challenging systems to synthesize executable code that answers natural-language questions about diverse, real-world tables. The task design, data, and evaluation protocols provide a rigorous framework for assessing the effectiveness of LLMs and code generation heuristics in structured information extraction and reasoning.

1. Task Specification and Dataset Characteristics

SemEval 2025 Task 8 is divided into two subtasks: DataBench QA (Subtask I) and DataBench Lite QA (Subtask II). Both require systems to ingest a table $T$ (in Parquet format, with up to hundreds of thousands of rows and up to 123 columns) and a natural language question $Q$, then output an answer $a$ in one of five types: boolean, number, category, list[number], or list[category]. Subtask II restricts tables to at most 20 rows to support rapid evaluation and development. The DataBench dataset spans 65 domains—including sports, survey statistics, and patent filings—with 1,308 questions evenly distributed over answer types. Table schemas are structurally heterogeneous, containing numeric, categorical, bool, datetime, and text columns, often with large cardinalities and significant missingness (Evangelatos et al., 1 Mar 2025, Lagos et al., 28 May 2025, Site et al., 1 Aug 2025).

Most QA queries conform to select–filter–aggregate patterns, requiring multi-step reasoning but seldom involving multi-table joins or deep aggregation. The evaluation protocol leverages the official databench_eval Python package, using (macro)-accuracy as the key metric: $$\mathrm{Accuracy} = \frac{1}{N} \sum_{i=1}^N \mathbf{1}\{\hat y_i = y_i\}$$ where $\hat y_i$ is the system output and $y_i$ the ground-truth answer.

2. System Pipelines and LLM-Driven Code Generation

State-of-the-art submissions universally employ LLM-based code generation, translating $(T, Q)$ into Python, typically using the Pandas library for dataframe manipulations and statistical operations. The dominant modeling paradigm decomposes the QA process into multiple interpretable stages:

• Schema Extraction: Systems extract column names/types, unique counts, and example values. Some pipelines normalize/standardize schemas (e.g., lowercasing, deduplication, truncation of example values) to improve prompt consistency.
• Prompt Construction: Prompts include the schema and question, and are engineered for explicitness—e.g., structured JSON descriptors, column dictionaries, or concise schema-summary templates. In-context learning with few-shot exemplars, or chain-of-thought (CoT) comments, is frequently used to improve instruction-following and interpretability.
• Code Generation: LLMs synthesize a Python function or code block that, when executed on $T$ (as a Pandas DataFrame), should return $a$. Constraints include: composable function signatures, banned use of certain Pandas operators (e.g., groupby()), explicit casting, and consistent variable-naming conventions.
• Execution and Iterative Error Handling: Generated code is sandboxed and executed. On exception (syntax or runtime), systems capture tracebacks and re-prompt the LLM for corrections, often limiting to 2–3 attempts for efficiency.
• Post-Processing: Answers are normalized to fit types accepted by the evaluator (boolean, int/float, str, or lists thereof), with simple output reformatting, e.g., converting floats to ints, tuples to lists, or string-literals to native types.

A representative example is the five-stage pipeline of “MRT” (Maximizing Recovery from Tables with Multiple Steps), in which each stage (Column Descriptor, Explainer, Coder/Runner, Interpreter, Formatter) is a well-defined module, combining LLM calls and lightweight heuristics (Lagos et al., 28 May 2025). Analogously, “AILS-NTUA” leverages prompt decomposition with an error-fixing loop, and “ITUNLP” adopts a zero-shot approach with schema-rich prompting and minimal correction iterations (Evangelatos et al., 1 Mar 2025, Site et al., 1 Aug 2025).

3. Model Choices, Prompt Engineering, and Error-Fixing

Model selection is a central dimension of system design. Submissions benchmarked proprietary LLMs (e.g., Claude 3.5 Sonnet) against open-source options (e.g., Llama-3.x, Qwen2.5-Coder, DeepSeek-R1/V3, Phi-4-14B). Empirically, code-specialized and larger models outperform general-purpose or smaller ones by sizable margins—gaps between 8B and 400B models can reach ≈20 accuracy points (Evangelatos et al., 1 Mar 2025, Lagos et al., 28 May 2025, Site et al., 1 Aug 2025).

Prompt design is shown to be critical: ablation studies demonstrate that prompt templates combining few-shot exemplars, CoT annotations, and schema details can yield accuracy improvements exceeding 10% relative to partial prompts. For example, inclusion of explicit schema fields (type, frequency, unique count), task descriptions, and example-driven reasoning steps enhances LLM consistency. Small prompt tweaks—such as instructions to cast columns or avoid certain Pandas idioms—reduce code exceptions by ≈15% (Lagos et al., 28 May 2025).

Error-fixing loops are effectively universal. Upon code failure, LLMs are prompted with the faulty code and exception details, and are tasked to return a revised, error-free version. Up to three retries are common. This mechanism recovers 1–3 accuracy points overall, with higher marginal gain for smaller/less reliable models, and resolves up to 40% of errors for the largest models (Evangelatos et al., 1 Mar 2025, Site et al., 1 Aug 2025).

4. Experimental Results and Comparative Analysis

Performance varies by model family, subtask, and system design. Table 1 presents representative accuracy scores for leading models on the DataBench test splits, as reported:

Model	Subtask I Test (%)	Subtask II Test (%)
Claude 3.5 Sonnet	85.63 (89.85*)	87.93 (88.89*)
Llama 3.1 405B	83.33 (87.16)	77.78 (78.54)
DeepSeek-R1	84.09	85.05
Qwen2.5-Coder	83.90	81.99
MRT (Qwen2.5-14B)	70.50	—
Baseline	26.00	27.00

(* After human review.) (Evangelatos et al., 1 Mar 2025, Lagos et al., 28 May 2025, Site et al., 1 Aug 2025)

The error-fixing module raises system accuracy by 1–3 points overall, while prompt engineering and model scaling account for much larger absolute gains. Notably, the macro-accuracy of the best open-source zero-shot system ("ITUNLP") reached 84–85%, outperforming the majority of baselines and competitive with proprietary models (Site et al., 1 Aug 2025).

Ablation studies confirm:

• Prompt compositions with chain-of-thought, task/column descriptions, and exemplars perform best.
• Completion of just one prompt example produces a 10% accuracy increase.
• Code-specialized LLMs (e.g., Qwen2.5-Coder, DeepSeek) outperform general-purpose models of similar parameter scale.
• Larger models both make fewer mistakes and are more effective at error correction (Evangelatos et al., 1 Mar 2025, Lagos et al., 28 May 2025, Site et al., 1 Aug 2025).

5. Error Analysis and Interpretability

Manual error breakdowns enumerate primary sources of failure:

• Incorrect or unnecessary step generation by LLMs (“Explainer” module): 37.66%
• Cell-value filtering errors (token mismatch): 14.29%
• Code exceptions (syntax/runtime): 14.29%
• Formatting/answer-type mismatches: <7% each
• Others/ambiguous: ≈23%

Runtime errors are most prevalent, with most (≈70%) related to key errors, value mismatches, or type conversion faults. Error correction via iterative feedback resolves ≈80% of syntax errors and ≈60% of runtime faults within two retries. Some value normalization problems (“Biden” vs. “Joe Biden”) persist, implicating a need for pre-alignment or fuzzy-matching modules. Interpretability is supported by explicit CoT-style comments and schema-aware prompt construction, facilitating both automatic and human debugging. Notably, a modular multi-stage design allows for granular error attribution and targeted improvements (Lagos et al., 28 May 2025, Evangelatos et al., 1 Mar 2025, Site et al., 1 Aug 2025).

6. Limitations, Open Challenges, and Future Directions

Current systems exhibit several limitations:

• Most remaining errors arise in the instruction-generation (explainer) phase, motivating research in either human-in-the-loop verification or re-ranking of LLM-generated step plans.
• Value normalization and semantic entity matching remain unresolved for up to 15% of queries, especially in categorical data.
• Formatter stages occasionally fail to conform to official evaluator expectations; integrating evaluation-aware feedback into prompts is a target for future work.
• Scaling to multi-table joins or free-form aggregation is an unsolved problem, with only single-table select-filter-aggregate queries currently supported.
• Code debugging, though effective, yields diminishing returns for larger models; improvements may come via error taxonomy refinement or dedicated “debugger” agents (Lagos et al., 28 May 2025, Evangelatos et al., 1 Mar 2025).

Future research directions highlighted in participant papers include:

• Hybrid “linguistic-first” strategies for ambiguous or entity-centric queries.
• Enhanced type and intermediate representation (IR) checking prior to code execution.
• Decomposition strategies for lightweight models (retrieval- or explain-first pipelines).
• Extension of the pipeline to handle open-ended text columns (e.g., sentiment analysis, NER) in conjunction with table reasoning (Lagos et al., 28 May 2025, Evangelatos et al., 1 Mar 2025).

7. Significance and Recommendations for Practitioners

The outcomes of SemEval 2025 Task 8 demonstrate that pure LLM-based prompting, combined with error-fixing and schema-aware prompt engineering, enables high-accuracy tabular QA on heterogeneous, large-scale benchmarks—without fine-tuning. Key strategies for practical deployment include:

• Schema normalization and inclusion of example values.
• Use of few-shot, chain-of-thought exemplars in prompt templates.
• Integration of iterative error correction loops.
• Model selection benchmarking on domain-specific dev sets.
• Explicit output normalization and evaluator-aware output formatting.

Public implementations and templates are available for replication and further study (Evangelatos et al., 1 Mar 2025, Lagos et al., 28 May 2025, Site et al., 1 Aug 2025). SemEval 2025 Task 8 establishes a robust foundation for evaluating and improving LLM-driven, program-synthesis-based approaches to data-centric QA, with continuing work required to address compositional, entity-normalization, and multi-table reasoning challenges.