DSQABench: Dynamic Sports QA Benchmark
- DSQABench is a dynamic sports QA benchmark that links natural language queries to manually authored, executable SQL over live EPL/FPL databases.
- It addresses challenges such as temporal grounding, schema linking, and multi-hop reasoning by binding each query to a specific database snapshot.
- Evaluation metrics include exact match for scalar answers and entailment-based scoring for table outputs, highlighting improved performance of models like GPT-4o.
Dynamic Sport Question Answering Benchmark, usually written as DSQABench, is an evaluation dataset introduced alongside SportSQL, an interactive system for natural-language querying and visualization over live, temporally evolving sports data with a focus on English Premier League (EPL) and Fantasy Premier League (FPL) data. It is designed for the task of Dynamic Sports Question Answering (DSQA): answering natural-language questions over real-time, evolving sports databases rather than over a static corpus or a fixed database snapshot. In DSQABench, each example is tied to executable structure rather than only to text, with a manually authored SQL query, a gold answer, and the database snapshot at the time of execution, so that evaluation is explicitly conditioned on temporal state (Martinez et al., 23 Aug 2025).
1. Origin and problem setting
DSQABench was created to fill a gap in both sports QA and text-to-SQL evaluation. The motivating claim is that most prior text-to-SQL benchmarks assume static schemas and immutable contents, while sports QA datasets often rely on unstructured text or multiple-choice formats rather than executable structured queries. DSQABench instead targets settings in which the underlying data is continuously updated and the correct answer may vary depending on when the query is asked (Martinez et al., 23 Aug 2025).
The benchmark is therefore organized around a different problem formulation from conventional sports QA. The relevant challenges include temporal grounding, schema linking, entity resolution, and execution against the correct database snapshot. The underlying motivation is specific to sports data: player and team statistics change, fixtures move from future to past, season-over-season records matter, derived indicators such as form and strength are queried, and users often refer to entities through aliases or nicknames rather than canonical database strings (Martinez et al., 23 Aug 2025).
Within the associated system, DSQABench serves as the evaluation companion to SportSQL, whose pipeline ingests live FPL data, normalizes it into a MariaDB-backed relational database, resolves entities such as player and team names, prompts an LLM to generate SQL using only the schema, executes the SQL over the live database, and optionally produces visualizations for trend, ranking, and comparison queries. The benchmark is thus not merely a question collection; it is built to measure whether symbolic QA remains correct under a live relational backend (Martinez et al., 23 Aug 2025).
2. Domain scope and dataset composition
DSQABench is focused on EPL and FPL data, and the underlying database is referred to as “FutSQL_FPL.” The system and benchmark involve multiple classes of soccer information, including players, teams, fixtures, player history, player past-game statistics, and future fixtures. The benchmark is explicitly described as operating over a live, dynamically updated relational database rather than a frozen extract (Martinez et al., 23 Aug 2025).
The dataset contains 1,793 questions derived from 180 base templates, with each template rephrased in three distinct ways and instantiated with real-world values. It is divided into 1,395 scalar-answer questions, which produce strings or numbers, and 398 tabular-output questions, which require table-shaped answers. The benchmark also includes 398 questions involving dynamic queries to player-specific tables fetched just in time from the API: player_past with 270 queries, player_history with 72 queries, and player_future with 54 queries (Martinez et al., 23 Aug 2025).
Question construction combines templated question generation and manual authoring. The process begins with manually written question templates, which are then rephrased into multiple natural-language variants, instantiated with real-world entities such as team and player names, and filled with context-appropriate numerical thresholds or conditions. In addition, the benchmark contains manually crafted challenging questions meant to probe harder reasoning patterns, including multi-hop and compositional cases. The intended result is a dataset with both lexical diversity and semantic control (Martinez et al., 23 Aug 2025).
3. Annotation model and executable semantics
Each DSQABench example includes three annotations: a gold SQL program, a gold answer, and a database snapshot. The SQL is manually authored, then executed and validated against the MariaDB database to ensure correctness. This makes DSQABench an executable benchmark, not just a natural-language QA dataset with reference answers detached from system execution (Martinez et al., 23 Aug 2025).
The role of the database snapshot is central. In a dynamic sports setting, the correct answer can change as the season progresses, so the benchmark binds each question to the state of the database at execution time. That temporal anchoring is what allows the same natural-language form to remain evaluable even when the live database is continuously refreshed. The benchmark is therefore designed not only to test text-to-SQL generation, but also to test whether execution occurs against the correct temporal context (Martinez et al., 23 Aug 2025).
This annotation scheme distinguishes DSQABench from benchmarks built only from passages, commentary, or multiple-choice options. A question in DSQABench is evaluated through an executable program operating on a temporally specified relational state. In practice, that means error can arise from several distinct sources—schema misunderstanding, entity grounding failure, incorrect temporal interpretation, or SQL that is syntactically valid but semantically misaligned with the live database—even when the natural-language question itself is well understood (Martinez et al., 23 Aug 2025).
4. Dynamicity and reasoning requirements
The defining novelty of DSQABench is its treatment of dynamic data. The underlying database is continuously refreshed from the FPL API; some tables are updated nightly, while others are fetched just in time during query execution. Queries may therefore depend on current season statistics, live player or team status, recent matches, upcoming fixtures, or historical season data. Some dynamic tables are not always persisted and may instead be loaded on demand and discarded afterward, and DSQABench contains queries that explicitly stress this runtime behavior (Martinez et al., 23 Aug 2025).
The benchmark is designed to stress several difficult subproblems. For schema linking, systems must map informal language to the correct schema columns and tables; the paper gives the example “team position” league_rank. For entity resolution, the system must ground aliases and nicknames such as “CR7” and “Donatello” to canonical entities. For executable query generation, the requirement is not merely syntactic validity but SQL that is actually executable and semantically correct on the live database (Martinez et al., 23 Aug 2025).
Query difficulty is also framed in terms of six reasoning primitives: The reported analysis states that simple single-primitive queries perform very well, while performance degrades rapidly when primitives are combined, especially in the presence of aggregation, joins, or multi-step manipulations. DSQABench is therefore intended to capture genuine compositional stress rather than shallow lookup behavior (Martinez et al., 23 Aug 2025).
5. Evaluation protocol and reported performance
DSQABench is used to evaluate two LLMs in the SportSQL pipeline: GPT-4o and Gemini-2.0 Flash. Because outputs can be either scalar strings or tables, evaluation is type-aware. String answers are scored by exact match, while table answers are evaluated with TabEval, which converts tables into atomic natural-language statements and computes entailment-based metrics via RoBERTa-MNLI. For tables, the reported metrics are Correctness (precision), Completeness (recall), and Overall (F1) (Martinez et al., 23 Aug 2025).
The reported results are as follows. For Gemini-2.0 Flash, String EM is 76.23, Table Correctness is 0.64, Table Completeness is 0.76, and Table Overall is 0.69. For GPT-4o, String EM is 80.48, Table Correctness is 0.70, Table Completeness is 0.81, and Table Overall is 0.75. The authors note that GPT-4o outperforms Gemini-2.0 Flash by about 4.2 points in exact match and 0.05 macro-F1 (Martinez et al., 23 Aug 2025).
The benchmark also supports qualitative demonstrations involving top goal scorers, historical player trends, team rankings, and visualization-oriented outputs such as bar charts and line plots. Those demonstrations are part of the broader SportSQL setting, where a successful system must not only answer questions over evolving soccer data but also support exploratory interaction over time-sensitive statistics (Martinez et al., 23 Aug 2025).
6. Relation to other sports QA benchmarks
DSQABench occupies a specific position within a broader landscape of sports QA benchmarks. SportQA is a large benchmark for sports understanding in LLMs with exactly 70,592 multiple-choice questions split into three difficulty levels: “Foundational Sports Knowledge” with 21,385 questions, “Rules and Tactics Comprehension” with 45,685 questions across 35 distinct sports types, and “Advanced Scenario-Based Understanding” with 3,522 questions focusing on six sports. In SportQA, the hardest level is “dynamic” in the sense of evolving game situations, rule dependencies, and scenario-conditioned inference, not in the sense of a live database (Xia et al., 2024).
LiveQA defines a different kind of dynamic sports QA. It is a multiple-choice QA dataset built from play-by-play live NBA broadcasts on Hupu, with 117,050 questions over 1,670 NBA games. Its questions are inserted into the broadcast timeline, so the answer may depend on when in the game the question is asked. The dataset stresses timeline understanding, event tracking, comparison, calculation, inference, and tracking, but it remains passage-centered rather than SQL-centered (Liu et al., 2020).
A separate 2024 paper also uses the name DSQABENCH for a broader sports understanding benchmark spanning text and video. In that usage, the benchmark is organized into Basic Sports Understanding, Advanced Scenario Analysis, and Video-based Sports Recognition, and is built by consolidating SportQA, BIG-bench Sports Understanding, and Sports-QA rather than by constructing a live relational database benchmark from scratch. That parallel usage is important because it places the acronym within a wider and partially overlapping benchmark literature, but it refers to a different artifact from the executable EPL/FPL benchmark introduced with SportSQL (Yang et al., 2024).
These comparisons clarify DSQABench’s distinctive contribution. Relative to multiple-choice sports QA, its emphasis is on schema-driven SQL generation and execution correctness. Relative to live broadcast QA, its dynamicity resides in temporally indexed relational data rather than in commentary streams. Relative to multimodal sports understanding, it targets natural-language querying over evolving databases rather than video recognition or action understanding (Martinez et al., 23 Aug 2025).
7. Significance, scope, and methodological implications
DSQABench is best understood as a dynamic, executable sports QA benchmark for evaluating systems that answer questions over live EPL/FPL data. Its contribution is to operationalize a setting in which sports QA depends on live data access, time-aware answers, schema-driven SQL generation, entity grounding, and support for both scalar and tabular outputs. In that sense, it fills a gap left by static text-to-SQL benchmarks and by sports QA datasets centered on unstructured text or fixed multiple-choice inventories (Martinez et al., 23 Aug 2025).
Its scope is nevertheless deliberately narrow in domain terms: the benchmark is focused on soccer, specifically EPL/FPL data. A plausible implication is that its primary value lies in making temporal database execution a first-class evaluation target rather than in maximizing sport coverage. That focus contrasts with SportQA’s coverage of 35 sports types and with CourtSI’s sports-focused spatial intelligence benchmark for VLMs, which evaluates metric spatial reasoning over badminton, tennis, and table tennis scenes rather than database querying (Xia et al., 2024, Yang et al., 10 Mar 2026).
More broadly, the emergence of DSQABench, SportQA, LiveQA, and CourtSI indicates that “dynamic sports QA” has become an umbrella for several technically distinct evaluation regimes: scenario-based rule reasoning, timeline-based broadcast understanding, live relational querying, and metric spatial reasoning in sports scenes. DSQABench’s specific role within that landscape is the structured-query regime, where correctness depends on aligning natural language with an evolving relational state and producing executable, temporally grounded answers (Martinez et al., 23 Aug 2025).
