---
title: 'DSQABench: Dynamic Sports QA Benchmark'
url: https://www.emergentmind.com/topics/dynamic-sport-question-answering-benchmark-dsqabench
type: topic
---

# DSQABench: Dynamic Sports QA Benchmark

Dynamic Sport Question Answering Benchmark, usually written as **DSQABench**, is an evaluation dataset introduced alongside **SportSQL**, an interactive system for natural-language querying and visualization over **live, temporally evolving sports data** with a focus on **English Premier League (EPL)** and **Fantasy Premier League (FPL)** data. It is designed for the task of **Dynamic Sports Question Answering (DSQA)**: answering natural-language questions over **real-time, evolving sports databases** rather than over a static corpus or a fixed database snapshot. In DSQABench, each example is tied to executable structure rather than only to text, with a **manually authored SQL query**, a **gold answer**, and the **database snapshot at the time of execution**, so that evaluation is explicitly conditioned on temporal state [2508.17157].

## 1. Origin and problem setting

DSQABench was created to fill a gap in both sports QA and text-to-SQL evaluation. The motivating claim is that most prior text-to-SQL benchmarks assume **static schemas and immutable contents**, while sports QA datasets often rely on **unstructured text** or **multiple-choice formats** rather than executable structured queries. DSQABench instead targets settings in which the underlying data is continuously updated and the correct answer may vary depending on when the query is asked [2508.17157].

The benchmark is therefore organized around a different problem formulation from conventional sports QA. The relevant challenges include **temporal grounding**, **schema linking**, **entity resolution**, and **execution against the correct database snapshot**. The underlying motivation is specific to sports data: player and team statistics change, fixtures move from future to past, season-over-season records matter, derived indicators such as form and strength are queried, and users often refer to entities through aliases or nicknames rather than canonical database strings [2508.17157].

Within the associated system, DSQABench serves as the evaluation companion to **SportSQL**, whose pipeline ingests live FPL data, normalizes it into a **MariaDB-backed relational database**, resolves entities such as player and team names, prompts an LLM to generate SQL using only the schema, executes the SQL over the live database, and optionally produces visualizations for trend, ranking, and comparison queries. The benchmark is thus not merely a question collection; it is built to measure whether symbolic QA remains correct under a live relational backend [2508.17157].

## 2. Domain scope and dataset composition

DSQABench is focused on **EPL** and **FPL** data, and the underlying database is referred to as **“FutSQL_FPL.”** The system and benchmark involve multiple classes of soccer information, including **players**, **teams**, **fixtures**, **player history**, **player past-game statistics**, and **future fixtures**. The benchmark is explicitly described as operating over a **live, dynamically updated relational database** rather than a frozen extract [2508.17157].

The dataset contains **1,793 questions** derived from **180 base templates**, with each template rephrased in **three distinct ways** and instantiated with real-world values. It is divided into **1,395 scalar-answer questions**, which produce strings or numbers, and **398 tabular-output questions**, which require table-shaped answers. The benchmark also includes **398 questions involving dynamic queries** to player-specific tables fetched just in time from the API: **`player_past`** with **270 queries**, **`player_history`** with **72 queries**, and **`player_future`** with **54 queries** [2508.17157].

Question construction combines **templated question generation** and **manual authoring**. The process begins with **manually written question templates**, which are then rephrased into multiple natural-language variants, instantiated with real-world entities such as team and player names, and filled with context-appropriate numerical thresholds or conditions. In addition, the benchmark contains **manually crafted challenging questions** meant to probe harder reasoning patterns, including **multi-hop** and **compositional** cases. The intended result is a dataset with both **lexical diversity** and **semantic control** [2508.17157].

## 3. Annotation model and executable semantics

Each DSQABench example includes three annotations: a **gold SQL program**, a **gold answer**, and a **database snapshot**. The SQL is **manually authored**, then **executed and validated** against the MariaDB database to ensure correctness. This makes DSQABench an **executable benchmark**, not just a natural-language QA dataset with reference answers detached from system execution [2508.17157].

The role of the **database snapshot** is central. In a dynamic sports setting, the correct answer can change as the season progresses, so the benchmark binds each question to the state of the database **at execution time**. That temporal anchoring is what allows the same natural-language form to remain evaluable even when the live database is continuously refreshed. The benchmark is therefore designed not only to test text-to-SQL generation, but also to test whether execution occurs against the correct temporal context [2508.17157].

This annotation scheme distinguishes DSQABench from benchmarks built only from passages, commentary, or multiple-choice options. A question in DSQABench is evaluated through an executable program operating on a temporally specified relational state. In practice, that means error can arise from several distinct sources—schema misunderstanding, entity grounding failure, incorrect temporal interpretation, or SQL that is syntactically valid but semantically misaligned with the live database—even when the natural-language question itself is well understood [2508.17157].

## 4. Dynamicity and reasoning requirements

The defining novelty of DSQABench is its treatment of **dynamic data**. The underlying database is continuously refreshed from the **FPL API**; some tables are updated nightly, while others are fetched **just in time** during query execution. Queries may therefore depend on current season statistics, live player or team status, recent matches, upcoming fixtures, or historical season data. Some dynamic tables are not always persisted and may instead be loaded on demand and discarded afterward, and DSQABench contains queries that explicitly stress this runtime behavior [2508.17157].

The benchmark is designed to stress several difficult subproblems. For **schema linking**, systems must map informal language to the correct schema columns and tables; the paper gives the example **“team position” \(\leftrightarrow\) `league_rank`**. For **entity resolution**, the system must ground aliases and nicknames such as **“CR7”** and **“Donatello”** to canonical entities. For **executable query generation**, the requirement is not merely syntactic validity but SQL that is actually executable and semantically correct on the live database [2508.17157].

Query difficulty is also framed in terms of six reasoning primitives:
\[
\text{Calculate},\ \text{Compare},\ \text{Filter},\ \text{Order},\ \text{Manipulate},\ \text{Retrieve}.
\]
The reported analysis states that simple **single-primitive** queries perform very well, while performance degrades rapidly when primitives are combined, especially in the presence of **aggregation**, **joins**, or **multi-step manipulations**. DSQABench is therefore intended to capture genuine **compositional stress** rather than shallow lookup behavior [2508.17157].

## 5. Evaluation protocol and reported performance

DSQABench is used to evaluate two LLMs in the SportSQL pipeline: **GPT-4o** and **Gemini-2.0 Flash**. Because outputs can be either scalar strings or tables, evaluation is **type-aware**. **String answers** are scored by **exact match**, while **table answers** are evaluated with **TabEval**, which converts tables into atomic natural-language statements and computes entailment-based metrics via **RoBERTa-MNLI**. For tables, the reported metrics are **Correctness** (precision), **Completeness** (recall), and **Overall** (F1) [2508.17157].

The reported results are as follows. For **Gemini-2.0 Flash**, **String EM** is **76.23**, **Table Correctness** is **0.64**, **Table Completeness** is **0.76**, and **Table Overall** is **0.69**. For **GPT-4o**, **String EM** is **80.48**, **Table Correctness** is **0.70**, **Table Completeness** is **0.81**, and **Table Overall** is **0.75**. The authors note that **GPT-4o outperforms Gemini-2.0 Flash by about 4.2 points in exact match and 0.05 macro-F1** [2508.17157].

The benchmark also supports qualitative demonstrations involving **top goal scorers**, **historical player trends**, **team rankings**, and visualization-oriented outputs such as **bar charts** and **line plots**. Those demonstrations are part of the broader SportSQL setting, where a successful system must not only answer questions over evolving soccer data but also support exploratory interaction over time-sensitive statistics [2508.17157].

## 6. Relation to other sports QA benchmarks

DSQABench occupies a specific position within a broader landscape of sports QA benchmarks. **SportQA** is a large benchmark for sports understanding in LLMs with exactly **70,592 multiple-choice questions** split into **three difficulty levels**: **“Foundational Sports Knowledge”** with **21,385 questions**, **“Rules and Tactics Comprehension”** with **45,685 questions** across **35 distinct sports types**, and **“Advanced Scenario-Based Understanding”** with **3,522 questions** focusing on six sports. In SportQA, the hardest level is “dynamic” in the sense of **evolving game situations, rule dependencies, and scenario-conditioned inference**, not in the sense of a live database [2402.15862].

**LiveQA** defines a different kind of dynamic sports QA. It is a multiple-choice QA dataset built from **play-by-play live NBA broadcasts** on Hupu, with **117,050 questions** over **1,670 NBA games**. Its questions are inserted into the broadcast timeline, so the answer may depend on **when** in the game the question is asked. The dataset stresses **timeline understanding**, **event tracking**, **comparison**, **calculation**, **inference**, and **tracking**, but it remains passage-centered rather than SQL-centered [2010.00526].

A separate 2024 paper also uses the name **DSQABENCH** for a broader sports understanding benchmark spanning **text and video**. In that usage, the benchmark is organized into **Basic Sports Understanding**, **Advanced Scenario Analysis**, and **Video-based Sports Recognition**, and is built by consolidating **SportQA**, **BIG-bench Sports Understanding**, and **Sports-QA** rather than by constructing a live relational database benchmark from scratch. That parallel usage is important because it places the acronym within a wider and partially overlapping benchmark literature, but it refers to a different artifact from the executable EPL/FPL benchmark introduced with SportSQL [2406.14877].

These comparisons clarify DSQABench’s distinctive contribution. Relative to multiple-choice sports QA, its emphasis is on **schema-driven SQL generation** and **execution correctness**. Relative to live broadcast QA, its dynamicity resides in **temporally indexed relational data** rather than in commentary streams. Relative to multimodal sports understanding, it targets **natural-language querying over evolving databases** rather than video recognition or action understanding [2508.17157].

## 7. Significance, scope, and methodological implications

DSQABench is best understood as a **dynamic, executable sports QA benchmark** for evaluating systems that answer questions over **live EPL/FPL data**. Its contribution is to operationalize a setting in which sports QA depends on **live data access**, **time-aware answers**, **schema-driven SQL generation**, **entity grounding**, and support for both **scalar** and **tabular** outputs. In that sense, it fills a gap left by static text-to-SQL benchmarks and by sports QA datasets centered on unstructured text or fixed multiple-choice inventories [2508.17157].

Its scope is nevertheless deliberately narrow in domain terms: the benchmark is focused on **soccer**, specifically EPL/FPL data. A plausible implication is that its primary value lies in making temporal database execution a first-class evaluation target rather than in maximizing sport coverage. That focus contrasts with SportQA’s coverage of **35 sports types** and with CourtSI’s sports-focused spatial intelligence benchmark for VLMs, which evaluates metric spatial reasoning over **badminton**, **tennis**, and **table tennis** scenes rather than database querying [2402.15862][2603.09896].

More broadly, the emergence of DSQABench, SportQA, LiveQA, and CourtSI indicates that “dynamic sports QA” has become an umbrella for several technically distinct evaluation regimes: **scenario-based rule reasoning**, **timeline-based broadcast understanding**, **live relational querying**, and **metric spatial reasoning in sports scenes**. DSQABench’s specific role within that landscape is the structured-query regime, where correctness depends on aligning natural language with an evolving relational state and producing executable, temporally grounded answers [2508.17157].

Source: https://www.emergentmind.com/topics/dynamic-sport-question-answering-benchmark-dsqabench