---
title: 'DP-Bench: Data Product Benchmark'
url: https://www.emergentmind.com/topics/dp-bench
type: topic
---

# DP-Bench: Data Product Benchmark

Searching arXiv for the DP-Bench paper and closely related benchmark papers mentioned in the provided material.
DP-Bench is a benchmark for evaluating automatic data product creation systems: systems that take a natural-language description of a business need and a database schema, and output a data product consisting of a curated subset of tables and columns together with any derived columns needed to support analysis or question answering for that business use case. It is positioned as the first benchmark of its kind for this task, addressing a setting in which existing benchmarks focus on adjacent problems such as text-to-SQL or ELT pipeline construction rather than end-to-end data product generation. In the benchmark formulation, a system must select relevant tables and columns, create derived columns when required, and preserve provenance for those derived columns by providing SQL showing how they are computed from source data [2512.15798].

## 1. Definition of the task

DP-Bench treats a data product as a purpose-driven, semantically curated schema designed to answer a specific business request rather than as raw data. The benchmark formalizes automatic data product creation as a task in which the input is a set of databases, optionally unstructured documents, and a natural-language **data product request (DPR)** describing a business use case. The output is a data product that contains a set of tables over which questions about the DPR can be answered, includes columns that are either directly selected from the input database, derived from input columns, or optionally extracted from unstructured text, and provides provenance for every derived column [2512.15798].

The released benchmark deliberately narrows this broader notion of a data product to two required capabilities: **table selection** and **derived column creation**. It does not require dashboards, reports, or pre-built queries, even though those can also be part of real data products. The benchmark also recognizes that there is no perfect one-to-one mapping between a DPR and a data product. Different granularities of derived columns or different modeling choices can be acceptable. Accordingly, the benchmark checks coverage by verifying whether original database columns are included either directly or indirectly through derivation provenance [2512.15798].

This task formulation distinguishes DP-Bench from ordinary schema retrieval. The target is not merely to identify columns relevant to a single question, but to construct a business-level abstraction that can support a family of questions associated with the same use case. A plausible implication is that evaluation must account both for schema selection and for transformation logic.

## 2. Source benchmarks and benchmark construction

DP-Bench is constructed by leveraging two existing benchmarks: **BIRD**, described as a large cross-domain text-to-SQL benchmark with **95 databases** and **12,751+ question-SQL pairs**, and **ELT-Bench**, described as a benchmark for evaluating AI agents on ELT pipelines with **100 pipelines for 100 databases**, **835 source tables**, and **203 data models**. Neither benchmark directly evaluates creation of a data product from a business request, but together they provide database schemas, natural-language questions, SQL, and higher-level data models with derived columns that can be repurposed into gold-standard data products and DPRs [2512.15798].

The construction process begins by aligning the two sources. The authors identified **78 ELT-Bench pipelines** corresponding to **78 BIRD databases**. For these aligned databases, BIRD contributes **10,928 unique question-SQL pairs**, while the 78 BIRD databases contain **4,306 columns** and the aligned ELT-Bench pipelines contain **122 data models** and **921 columns**. Using an LLM and manual verification, ELT-Bench data models were aligned to BIRD tables; among the aligned columns, **634/921 columns (69%)** were **derived columns**, and the remaining **287 columns** were **non-derived**, directly mappable to source database columns [2512.15798].

For derived columns, DP-Bench requires explicit provenance. The benchmark therefore prompts an LLM to generate SQL provenance from the source schema using the template:

```text
Generate only SQL to find {COL_DESC} from the following database schema. {DB}
```

Here, `{DB}` includes the complete schema with table names, column names, descriptions, types, primary keys, and foreign keys. The resulting SQL is manually validated, and derived columns whose provenance is wrong or unusable are discarded. After filtering, **582 derived columns** remain [2512.15798].

Preliminary data products are then created on a per-database basis by grouping the **582 derived columns** and **287 non-derived columns** by table names in the corresponding BIRD database. For each ELT-Bench data model, non-derived columns are placed in the aligned BIRD table, while some derived columns are manually moved to different tables when that is more semantically appropriate [2512.15798].

Candidate DPRs are generated from the resulting data products in two prompt-based stages. First, for each data product, an LLM produces **3 candidate business-use summaries** from table metadata, each beginning with the phrase “This table contains data that enables analysis of ...” and limited to 100 words. Second, each summary is reformulated into a candidate DPR. This yields **234 candidate DPRs** for **78 data products**. These are then manually curated in a **two-phase annotation process**: for each data product, **5 BIRD questions** are randomly selected; in phase 1, **2 annotators** review each candidate DPR and edit it if it is not broad and relevant enough to support those questions; in phase 2, a **third annotator** adjudicates disagreements and produces the final gold DPR. The paper reports that **71%** of candidate DPRs required **no edits**, while **67 candidate DPRs** across **27 data products** were edited [2512.15798].

The benchmark is subsequently expanded using BIRD questions. Of the original **10,928** question-SQL pairs, **9,218/10,928** are described as noise-free and traceable to the schema, involving **4,331 unique table-column pairs**. The benchmark uses IBM’s `slate.30m.english.rtrvr` embedding model with a cosine similarity threshold of **0.80** to retrieve BIRD questions relevant to each DPR, yielding **836 unique questions** relevant to the **234 DPRs**. Missing tables and columns required by these relevant SQLs are added to the data products [2512.15798].

## 3. Dataset contents and semantic annotation

Each DP-Bench instance contains a **business use case description** in the form of a DPR, the corresponding **data product**, the **source DB schema**, **natural-language questions** associated with the use case, SQL **provenance** for derived columns, and **topics** annotated for both the DPR and the data product [2512.15798].

The released benchmark comprises the following aggregate counts:

| Component | Count |
|---|---:|
| Data products | 78 |
| DPRs | 78 |
| Tables across final data products | 383 |
| Non-derived columns | 1,458 |
| Derived columns | 582 |
| Topics from data products | 564 |
| Topics from DPRs | 461 |

A **Hard** subset is defined separately, consisting of **30 databases/data products**, each with **at least 50 columns**. This subset is used for more challenging evaluation and is reported independently in the empirical results [2512.15798].

Topic annotation adds an explicit semantic layer to the benchmark. Topics are generated for both the data product and the DPR using manually crafted **3-shot prompts** with **Llama-3.3-70B-Instruct**, after which the topics are manually filtered. The paper describes several intended uses for these topics: semantic indexing and search, quality evaluation of generated data products, and measurement of abstraction alignment between DPRs and data products. The provided example contrasts data-product-side topics such as “User reviews,” “Mobile apps,” and “Sentiment analysis” with more business-oriented DPR topics such as “App performance evaluation” and “Mobile app reputation” [2512.15798].

This organization suggests that DP-Bench is not limited to literal schema reconstruction. It also encodes a semantic relation between request language and curated schema content, although the benchmark’s formal scoring is still centered on columns, derived columns, and downstream executability.

## 4. Evaluation methodology

DP-Bench evaluates systems along three axes: column selection quality, derived-column similarity, and downstream text-to-SQL performance [2512.15798].

The first axis is **column-level precision, recall, and F1**. Given a DPR, a gold data product, and a predicted data product, the benchmark computes **Precision**, **Recall**, and **F1** over the set of predicted columns relative to the gold columns. This is reported under two settings: **non-derived only**, where only directly selected columns count, and **including derived provenance**, where source columns used to create derived columns are also included. Scores are **macro-averaged** over all DPRs [2512.15798].

The second axis is **derived-column similarity**. Since exact match between generated and gold derived columns is considered too strict, DP-Bench defines a softer comparison based on SQL AST similarity. For each derived column, the SQL provenance is converted into an AST using `sqlglot`, with normalization including canonical ordering, alias elimination, and conversion to DNF. The similarity between two derived columns \(x\) and \(y\) is defined as

\[
sim(x,y)=1-\frac{diff(x,y)}{totaldiff(x,y)}
\]

where `diff(x,y)` is the AST edit distance from `sqlglot.diff`, and `totaldiff(x,y)` is the total number of AST comparison steps. Pairwise similarities are computed between predicted and gold derived columns; a maximum-similarity unique matching is then found, and these similarities are treated as soft true positives. The benchmark reports **soft precision**, **soft recall**, and **soft F1**, again macro-averaged over the dataset [2512.15798].

The third axis is **text-to-SQL execution accuracy**, designed to measure whether a predicted data product is sufficient for downstream question answering. For each DPR, the corresponding questions are answered using the predicted data product as schema context. The generated SQL is executed and compared against the ground-truth SQL result following a BIRD-style evaluation. Three scenarios are reported: **Oracle**, which assumes perfect text-to-SQL and tests only whether the data product contains enough information for the query to be executable; **Llama 4 Maverick** as the text-to-SQL model; and **GPT OSS 120b** as the text-to-SQL model. These results are reported on the **Hard subset** only, containing **900 total questions** across **30 databases** [2512.15798].

The benchmark also includes two comparison settings for downstream evaluation: **No Search**, which uses the entire database schema as context, and **Ground-Truth DP**, which uses the curated gold data product. This enables explicit comparison between schema reduction and unrestricted schema exposure [2512.15798].

## 5. Baseline systems

DP-Bench provides two baseline families: a hybrid retrieval baseline for column selection and a family of LLM-based data product creation methods [2512.15798].

The **hybrid search baseline** embeds table-column descriptions using both a **dense sentence embedding** model and **BM25** sparse retrieval. Each table-column pair is represented as a sentence formed from the table name and column name; this representation is compared to the DPR embedding. The final relevance score is a weighted combination with dense similarity weight **0.8** and sparse similarity weight **0.2**, and a cutoff threshold of **0.75** determines whether a column is selected. If the predicted data product contains tables connected by foreign keys, the foreign-key columns are added as well. This baseline cannot generate derived columns [2512.15798].

The **LLM-based baseline family** prompts large language models with the full database schema in JSON, the DPR, and chain-of-thought instructions, and asks them to identify relevant columns and generate derived columns together with SQL for them. Hallucinated tables or columns are filtered out, and generation is repeated when the output JSON is malformed. The evaluated models are **GPT OSS 20b**, **Granite 4 Small**, **Qwen 3 30b Thinking**, **Qwen 2.5 72b Instruct**, **GPT OSS 120b**, **Llama 3.3 70b Instruct**, and **Llama 4 Maverick**. The paper describes these as spanning dense, MoE, and “thinking” architectures, with parameter sizes ranging from around **21B** to **400B** total parameters [2512.15798].

These baselines reflect the two main subproblems encoded in the benchmark. The retrieval baseline targets relevance estimation over existing schema elements, whereas the LLM baselines address both relevance estimation and synthesis of derived columns with provenance.

## 6. Empirical findings and limitations

On the full benchmark, the **No Search** condition already attains relatively strong column-level F1 because many databases are small: **0.648** for non-derived columns and **0.715** when including derived provenance. Among the reported baselines, **Llama 4 Maverick** achieves the best overall F1 with **0.704** non-derived F1 and **0.762** F1 including derived provenance. The paper also notes that GPT OSS models show **very high recall** but somewhat lower precision, that **Llama 3.3** and **Llama 4** perform best in F1, and that **Qwen 3 30b** is strong on standard column selection but not on derived columns [2512.15798].

Performance drops on the **Hard subset**, as expected. In that setting, **No Search** reaches **0.459** non-derived F1 and **0.534** F1 including derived provenance, while the best reported baseline remains **Llama 4 Maverick** at **0.557** non-derived F1 and **0.612** F1 including derived provenance [2512.15798].

Derived columns are markedly harder than plain column selection. On the full benchmark, the best reported soft-F1 values remain low: **Llama 3.3 70b** reaches **0.254**, **GPT OSS 120b** reaches **0.267**, and **Qwen 3 30b** attains **0.047**. The paper notes that derived-column quality on the Hard subset is nearly the same as on the full set, which it interprets as evidence that the difficulty is fundamental rather than primarily an artifact of database size [2512.15798].

Topic-based quality is also reported. Using topic embeddings, the best baseline, **GPT OSS 120b**, achieves an average semantic quality score of **0.594**. The paper reports little correlation between original database size and topic-based data-product quality, and lower quality when the DPR is vague or underspecified [2512.15798].

The downstream text-to-SQL results are structurally significant. The **Ground-Truth DP** improves text-to-SQL execution accuracy compared with **No Search**, confirming that a well-curated data product can help downstream question answering. On the **Hard subset**, both **Oracle** settings achieve **100%** execution accuracy for **No Search** and **Ground-Truth DP**, while **Llama 4 Maverick** improves from **45.6%** under No Search to **49.8%** under Ground-Truth DP, and **GPT OSS 120b** improves from **37.7%** to **41.4%**. At the same time, all automatically generated data products perform worse than **No Search** in downstream execution accuracy, which the paper attributes to predicted data products sometimes dropping critical columns and thereby hurting recall. For predicted data products, oracle accuracies range from roughly **43.4% to 84.3%** [2512.15798].

DP-Bench also reports average input prompt lengths for text-to-SQL: **8,432 tokens** on average in the **No Search** condition versus **2,192 tokens** with the **Ground-Truth DP**. This indicates that high-quality data products can reduce schema context size substantially while preserving executability [2512.15798].

The benchmark’s stated limitations are explicit. It currently focuses on **single-DB** data products rather than cross-database products; it simplifies data products to **tables and derived columns**, excluding dashboards and reports; and although the task definition mentions broader generalization to unstructured text, that aspect is not central to the released benchmark. The paper identifies **cross-DB data products** and **agentic optimization** of generated data products as future directions. Taken together, these constraints show that DP-Bench is a benchmark for a deliberately scoped but nontrivial version of automatic data product creation, in which the dominant open difficulty is not merely selecting relevant schema elements but generating correct derived transformations with usable provenance [2512.15798].

Source: https://www.emergentmind.com/topics/dp-bench