DP-Bench: Data Product Benchmark

• DP-Bench is a benchmark for evaluating end-to-end data product creation, where systems select relevant tables and generate derived columns with explicit SQL provenance.
• It integrates data from BIRD and ELT-Bench, aligning natural language business requests with schema curation through both automated and manual validation.
• The evaluation focuses on column-level accuracy, derived-column similarity, and downstream text-to-SQL performance, offering a practical measure of schema optimization.

Searching arXiv for the DP-Bench paper and closely related benchmark papers mentioned in the provided material. DP-Bench is a benchmark for evaluating automatic data product creation systems: systems that take a natural-language description of a business need and a database schema, and output a data product consisting of a curated subset of tables and columns together with any derived columns needed to support analysis or question answering for that business use case. It is positioned as the first benchmark of its kind for this task, addressing a setting in which existing benchmarks focus on adjacent problems such as text-to-SQL or ELT pipeline construction rather than end-to-end data product generation. In the benchmark formulation, a system must select relevant tables and columns, create derived columns when required, and preserve provenance for those derived columns by providing SQL showing how they are computed from source data (Chowdhury et al., 16 Dec 2025).

1. Definition of the task

DP-Bench treats a data product as a purpose-driven, semantically curated schema designed to answer a specific business request rather than as raw data. The benchmark formalizes automatic data product creation as a task in which the input is a set of databases, optionally unstructured documents, and a natural-language data product request (DPR) describing a business use case. The output is a data product that contains a set of tables over which questions about the DPR can be answered, includes columns that are either directly selected from the input database, derived from input columns, or optionally extracted from unstructured text, and provides provenance for every derived column (Chowdhury et al., 16 Dec 2025).

The released benchmark deliberately narrows this broader notion of a data product to two required capabilities: table selection and derived column creation. It does not require dashboards, reports, or pre-built queries, even though those can also be part of real data products. The benchmark also recognizes that there is no perfect one-to-one mapping between a DPR and a data product. Different granularities of derived columns or different modeling choices can be acceptable. Accordingly, the benchmark checks coverage by verifying whether original database columns are included either directly or indirectly through derivation provenance (Chowdhury et al., 16 Dec 2025).

This task formulation distinguishes DP-Bench from ordinary schema retrieval. The target is not merely to identify columns relevant to a single question, but to construct a business-level abstraction that can support a family of questions associated with the same use case. A plausible implication is that evaluation must account both for schema selection and for transformation logic.

2. Source benchmarks and benchmark construction

DP-Bench is constructed by leveraging two existing benchmarks: BIRD, described as a large cross-domain text-to-SQL benchmark with 95 databases and 12,751+ question-SQL pairs, and ELT-Bench, described as a benchmark for evaluating AI agents on ELT pipelines with 100 pipelines for 100 databases, 835 source tables, and 203 data models. Neither benchmark directly evaluates creation of a data product from a business request, but together they provide database schemas, natural-language questions, SQL, and higher-level data models with derived columns that can be repurposed into gold-standard data products and DPRs (Chowdhury et al., 16 Dec 2025).

The construction process begins by aligning the two sources. The authors identified 78 ELT-Bench pipelines corresponding to 78 BIRD databases. For these aligned databases, BIRD contributes 10,928 unique question-SQL pairs, while the 78 BIRD databases contain 4,306 columns and the aligned ELT-Bench pipelines contain 122 data models and 921 columns. Using an LLM and manual verification, ELT-Bench data models were aligned to BIRD tables; among the aligned columns, 634/921 columns (69%) were derived columns, and the remaining 287 columns were non-derived, directly mappable to source database columns (Chowdhury et al., 16 Dec 2025).

For derived columns, DP-Bench requires explicit provenance. The benchmark therefore prompts an LLM to generate SQL provenance from the source schema using the template:

Generate only SQL to find {COL_DESC} from the following database schema. {DB}

Here, {DB} includes the complete schema with table names, column names, descriptions, types, primary keys, and foreign keys. The resulting SQL is manually validated, and derived columns whose provenance is wrong or unusable are discarded. After filtering, 582 derived columns remain (Chowdhury et al., 16 Dec 2025).

Preliminary data products are then created on a per-database basis by grouping the 582 derived columns and 287 non-derived columns by table names in the corresponding BIRD database. For each ELT-Bench data model, non-derived columns are placed in the aligned BIRD table, while some derived columns are manually moved to different tables when that is more semantically appropriate (Chowdhury et al., 16 Dec 2025).

Candidate DPRs are generated from the resulting data products in two prompt-based stages. First, for each data product, an LLM produces 3 candidate business-use summaries from table metadata, each beginning with the phrase “This table contains data that enables analysis of ...” and limited to 100 words. Second, each summary is reformulated into a candidate DPR. This yields 234 candidate DPRs for 78 data products. These are then manually curated in a two-phase annotation process: for each data product, 5 BIRD questions are randomly selected; in phase 1, 2 annotators review each candidate DPR and edit it if it is not broad and relevant enough to support those questions; in phase 2, a third annotator adjudicates disagreements and produces the final gold DPR. The paper reports that 71% of candidate DPRs required no edits, while 67 candidate DPRs across 27 data products were edited (Chowdhury et al., 16 Dec 2025).

The benchmark is subsequently expanded using BIRD questions. Of the original 10,928 question-SQL pairs, 9,218/10,928 are described as noise-free and traceable to the schema, involving 4,331 unique table-column pairs. The benchmark uses IBM’s slate.30m.english.rtrvr embedding model with a cosine similarity threshold of 0.80 to retrieve BIRD questions relevant to each DPR, yielding 836 unique questions relevant to the 234 DPRs. Missing tables and columns required by these relevant SQLs are added to the data products (Chowdhury et al., 16 Dec 2025).

3. Dataset contents and semantic annotation

Each DP-Bench instance contains a business use case description in the form of a DPR, the corresponding data product, the source DB schema, natural-language questions associated with the use case, SQL provenance for derived columns, and topics annotated for both the DPR and the data product (Chowdhury et al., 16 Dec 2025).

The released benchmark comprises the following aggregate counts:

Component	Count
Data products	78
DPRs	78
Tables across final data products	383
Non-derived columns	1,458
Derived columns	582
Topics from data products	564
Topics from DPRs	461

A Hard subset is defined separately, consisting of 30 databases/data products, each with at least 50 columns. This subset is used for more challenging evaluation and is reported independently in the empirical results (Chowdhury et al., 16 Dec 2025).

Topic annotation adds an explicit semantic layer to the benchmark. Topics are generated for both the data product and the DPR using manually crafted 3-shot prompts with Llama-3.3-70B-Instruct, after which the topics are manually filtered. The paper describes several intended uses for these topics: semantic indexing and search, quality evaluation of generated data products, and measurement of abstraction alignment between DPRs and data products. The provided example contrasts data-product-side topics such as “User reviews,” “Mobile apps,” and “Sentiment analysis” with more business-oriented DPR topics such as “App performance evaluation” and “Mobile app reputation” (Chowdhury et al., 16 Dec 2025).

This organization suggests that DP-Bench is not limited to literal schema reconstruction. It also encodes a semantic relation between request language and curated schema content, although the benchmark’s formal scoring is still centered on columns, derived columns, and downstream executability.

4. Evaluation methodology

DP-Bench evaluates systems along three axes: column selection quality, derived-column similarity, and downstream text-to-SQL performance (Chowdhury et al., 16 Dec 2025).

The first axis is column-level precision, recall, and F1. Given a DPR, a gold data product, and a predicted data product, the benchmark computes Precision, Recall, and F1 over the set of predicted columns relative to the gold columns. This is reported under two settings: non-derived only, where only directly selected columns count, and including derived provenance, where source columns used to create derived columns are also included. Scores are macro-averaged over all DPRs (Chowdhury et al., 16 Dec 2025).

The second axis is derived-column similarity. Since exact match between generated and gold derived columns is considered too strict, DP-Bench defines a softer comparison based on SQL AST similarity. For each derived column, the SQL provenance is converted into an AST using sqlglot, with normalization including canonical ordering, alias elimination, and conversion to DNF. The similarity between two derived columns $x$ and $y$ is defined as

$sim(x,y)=1-\frac{diff(x,y)}{totaldiff(x,y)}$

where diff(x,y) is the AST edit distance from sqlglot.diff, and totaldiff(x,y) is the total number of AST comparison steps. Pairwise similarities are computed between predicted and gold derived columns; a maximum-similarity unique matching is then found, and these similarities are treated as soft true positives. The benchmark reports soft precision, soft recall, and soft F1, again macro-averaged over the dataset (Chowdhury et al., 16 Dec 2025).

The third axis is text-to-SQL execution accuracy, designed to measure whether a predicted data product is sufficient for downstream question answering. For each DPR, the corresponding questions are answered using the predicted data product as schema context. The generated SQL is executed and compared against the ground-truth SQL result following a BIRD-style evaluation. Three scenarios are reported: Oracle, which assumes perfect text-to-SQL and tests only whether the data product contains enough information for the query to be executable; Llama 4 Maverick as the text-to-SQL model; and GPT OSS 120b as the text-to-SQL model. These results are reported on the Hard subset only, containing 900 total questions across 30 databases (Chowdhury et al., 16 Dec 2025).

The benchmark also includes two comparison settings for downstream evaluation: No Search, which uses the entire database schema as context, and Ground-Truth DP, which uses the curated gold data product. This enables explicit comparison between schema reduction and unrestricted schema exposure (Chowdhury et al., 16 Dec 2025).

5. Baseline systems

DP-Bench provides two baseline families: a hybrid retrieval baseline for column selection and a family of LLM-based data product creation methods (Chowdhury et al., 16 Dec 2025).

The hybrid search baseline embeds table-column descriptions using both a dense sentence embedding model and BM25 sparse retrieval. Each table-column pair is represented as a sentence formed from the table name and column name; this representation is compared to the DPR embedding. The final relevance score is a weighted combination with dense similarity weight 0.8 and sparse similarity weight 0.2, and a cutoff threshold of 0.75 determines whether a column is selected. If the predicted data product contains tables connected by foreign keys, the foreign-key columns are added as well. This baseline cannot generate derived columns (Chowdhury et al., 16 Dec 2025).

The LLM-based baseline family prompts LLMs with the full database schema in JSON, the DPR, and chain-of-thought instructions, and asks them to identify relevant columns and generate derived columns together with SQL for them. Hallucinated tables or columns are filtered out, and generation is repeated when the output JSON is malformed. The evaluated models are GPT OSS 20b, Granite 4 Small, Qwen 3 30b Thinking, Qwen 2.5 72b Instruct, GPT OSS 120b, Llama 3.3 70b Instruct, and Llama 4 Maverick. The paper describes these as spanning dense, MoE, and “thinking” architectures, with parameter sizes ranging from around 21B to 400B total parameters (Chowdhury et al., 16 Dec 2025).

These baselines reflect the two main subproblems encoded in the benchmark. The retrieval baseline targets relevance estimation over existing schema elements, whereas the LLM baselines address both relevance estimation and synthesis of derived columns with provenance.

6. Empirical findings and limitations

On the full benchmark, the No Search condition already attains relatively strong column-level F1 because many databases are small: 0.648 for non-derived columns and 0.715 when including derived provenance. Among the reported baselines, Llama 4 Maverick achieves the best overall F1 with 0.704 non-derived F1 and 0.762 F1 including derived provenance. The paper also notes that GPT OSS models show very high recall but somewhat lower precision, that Llama 3.3 and Llama 4 perform best in F1, and that Qwen 3 30b is strong on standard column selection but not on derived columns (Chowdhury et al., 16 Dec 2025).

Performance drops on the Hard subset, as expected. In that setting, No Search reaches 0.459 non-derived F1 and 0.534 F1 including derived provenance, while the best reported baseline remains Llama 4 Maverick at 0.557 non-derived F1 and 0.612 F1 including derived provenance (Chowdhury et al., 16 Dec 2025).

Derived columns are markedly harder than plain column selection. On the full benchmark, the best reported soft-F1 values remain low: Llama 3.3 70b reaches 0.254, GPT OSS 120b reaches 0.267, and Qwen 3 30b attains 0.047. The paper notes that derived-column quality on the Hard subset is nearly the same as on the full set, which it interprets as evidence that the difficulty is fundamental rather than primarily an artifact of database size (Chowdhury et al., 16 Dec 2025).

Topic-based quality is also reported. Using topic embeddings, the best baseline, GPT OSS 120b, achieves an average semantic quality score of 0.594. The paper reports little correlation between original database size and topic-based data-product quality, and lower quality when the DPR is vague or underspecified (Chowdhury et al., 16 Dec 2025).

The downstream text-to-SQL results are structurally significant. The Ground-Truth DP improves text-to-SQL execution accuracy compared with No Search, confirming that a well-curated data product can help downstream question answering. On the Hard subset, both Oracle settings achieve 100% execution accuracy for No Search and Ground-Truth DP, while Llama 4 Maverick improves from 45.6% under No Search to 49.8% under Ground-Truth DP, and GPT OSS 120b improves from 37.7% to 41.4%. At the same time, all automatically generated data products perform worse than No Search in downstream execution accuracy, which the paper attributes to predicted data products sometimes dropping critical columns and thereby hurting recall. For predicted data products, oracle accuracies range from roughly 43.4% to 84.3% (Chowdhury et al., 16 Dec 2025).

DP-Bench also reports average input prompt lengths for text-to-SQL: 8,432 tokens on average in the No Search condition versus 2,192 tokens with the Ground-Truth DP. This indicates that high-quality data products can reduce schema context size substantially while preserving executability (Chowdhury et al., 16 Dec 2025).

The benchmark’s stated limitations are explicit. It currently focuses on single-DB data products rather than cross-database products; it simplifies data products to tables and derived columns, excluding dashboards and reports; and although the task definition mentions broader generalization to unstructured text, that aspect is not central to the released benchmark. The paper identifies cross-DB data products and agentic optimization of generated data products as future directions. Taken together, these constraints show that DP-Bench is a benchmark for a deliberately scoped but nontrivial version of automatic data product creation, in which the dominant open difficulty is not merely selecting relevant schema elements but generating correct derived transformations with usable provenance (Chowdhury et al., 16 Dec 2025).