DeKeyNLU: Dataset for NL2SQL Interpretation
- DeKeyNLU is a specialized dataset with structured annotations for task decomposition and keyword extraction, tailored for NL2SQL generation.
- The dataset addresses recurrent retrieval and SQL generation errors by providing a clear, SQL-grounded interpretation layer.
- DeKeyNLU is integral to the DeKeySQL pipeline, demonstrably improving SQL execution accuracy on benchmarks like BIRD and Spider.
DeKeyNLU is a dataset for natural language understanding in natural language to SQL generation that was introduced to supervise two upstream capabilities in modular NL2SQL systems: task decomposition and keyword extraction. It contains 1,500 annotated question-answer pairs derived from the BIRD training set, and it was proposed together with DeKeySQL, a Retrieval-Augmented Generation pipeline whose User Question Understanding module is fine-tuned on DeKeyNLU. The dataset’s stated purpose is to address recurrent failures in recent RAG- and Chain-of-Thought-based NL2SQL systems, where inaccurate decomposition and imprecise keyword extraction propagate into retrieval errors, logical errors, and incorrect field identification during SQL generation (Chen et al., 18 Sep 2025).
1. Definition and problem setting
DeKeyNLU is defined around the front-end interpretation stage of NL2SQL rather than around SQL decoding alone. In the paper’s formulation, recent systems such as CHASE-SQL, CHESS, PURPLE, DTS-SQL, and MAC-SQL improve NL2SQL by modular reasoning, but still suffer from “inadequate task decomposition and imprecise keyword extraction from user queries,” especially when questions involve complex, multi-table relationships (Chen et al., 18 Sep 2025).
The dataset therefore targets a specific systems bottleneck. Instead of treating user-question understanding as an incidental prompt-engineering step, it provides explicit supervision for how a question should be decomposed and which terms should drive retrieval. This makes DeKeyNLU distinct from datasets that focus only on question–SQL pairs: each item begins with a BIRD natural-language question and its gold SQL, but adds a structured interpretation layer intended for pre-generation reasoning and evidence retrieval.
A common misconception is to read DeKeyNLU as a generic decomposition corpus. The paper argues otherwise. It explicitly criticizes resources such as BREAK, QDecomp, QPL, and DARA for not being sufficiently tailored to Database QA, either because they “often lack data directly pertinent to the primary task of complex SQL generation” or because they do not provide “comprehensive domain-specific annotations for Database QA evaluation” (Chen et al., 18 Sep 2025). In that sense, DeKeyNLU is database-grounded rather than decomposition-centric in the abstract.
2. Annotation schema and representational design
Each DeKeyNLU instance is annotated along two axes: task decomposition and keyword extraction. The decomposition side consists of a main task and sub-tasks. The keyword side consists of object and implementation fields.
| Component | Subfield | Function |
|---|---|---|
| Task decomposition | Main task; sub-tasks | Clarifies query structure and dependencies |
| Keyword extraction | Object; implementation | Supports retrieval of schema and filtering evidence |
The paper defines the keyword types narrowly. Object refers to terms related to table or column names. Implementation refers to filtering criteria and is represented “as a dictionary of actions and conditions” (Chen et al., 18 Sep 2025). This is not generic keyword extraction in the information-retrieval sense. It is a schema- and condition-oriented representation designed to support entity retrieval in a database setting.
The decomposition annotations are also explicitly SQL-relevant. The paper states that the main task often corresponds to the main SELECT component, while sub-tasks often map to operations such as INNER JOIN and WHERE (Chen et al., 18 Sep 2025). The representation is therefore closer to SQL-oriented question analysis than to free-form reasoning traces.
The appendix-level error taxonomy described in the paper makes the intended granularity concrete. Task-decomposition errors include “Main Task Belongs to Sub Task,” “Main Task Is Incomplete,” “Sub Task Is Incomplete,” “Sub Task Number Is Wrong,” and “Sub Task Is Ambiguous.” Keyword errors include “Miss Keyword,” “Wrong Keyword,” and “Useless Keyword” (Chen et al., 18 Sep 2025). These categories show that DeKeyNLU is meant to encode operationally useful structure: not only whether a decomposition exists, but whether its hierarchy, completeness, and schema relevance are correct.
A plausible implication is that DeKeyNLU is intended less as a benchmark for abstract semantic decomposition than as supervision for a retrieval-conditioned, SQL-oriented interpretation interface.
3. Dataset construction and quality control
DeKeyNLU is built from the BIRD dataset. The paper describes BIRD as containing 12,751 text-to-SQL pairs across 95 databases, totaling 33.4 GB and spanning 37 professional domains; BIRD itself integrates 80 open-source relational databases and 15 hidden-test databases (Chen et al., 18 Sep 2025). From the BIRD training dataset, the authors randomly selected 1,500 instances.
The annotation workflow has three stages. First, GPT-4o produces pre-annotations for tasks, objects, and implementations. The paper notes that even with CoT and few-shot prompting, GPT-4o frequently generated “redundant/incomplete tasks or incorrect keywords” (Chen et al., 18 Sep 2025). Second, three expert annotators reviewed and corrected the outputs in a three-phase cyclic process. Annotators began from different subsets, exchanged subsets, and ensured that each instance was evaluated by all three. Before the main pass, the team conducted initial training on 50 data points to calibrate annotation standards. Third, after rotational review, annotators rated the revised outputs on a 5-point Likert scale, where 1 is unsatisfactory and 5 is excellent; any case with an average score below 4 was discussed collaboratively and revised.
The main reported agreement statistic is Krippendorff’s Alpha , which the paper interprets as “a high level of consistency” (Chen et al., 18 Sep 2025). The final dataset is split into 70% training, 20% validation, and 10% testing.
The dataset statistics indicate nontrivial compositional structure. For main tasks, 68.2% of questions have one main task, 24.9% have two, and 6.9% have three or more. For sub-tasks, 31.7% have 1–2, 60.0% have 3–4, and 8.3% have over 5. For keywords, 20.3% have 1–2, 60.6% have 3–4, and 19.2% have 5 or more (Chen et al., 18 Sep 2025). These figures support the paper’s claim that the dataset captures substantial compositional complexity rather than only trivial single-step questions.
4. Role in the DeKeySQL architecture
DeKeyNLU is operationalized through DeKeySQL, a RAG-based NL2SQL framework with three main modules: User Question Understanding, Entity Retrieval, and Generation; a separate Revision phase is part of Generation (Chen et al., 18 Sep 2025). The dataset is used directly for fine-tuning the User Question Understanding module.
In UQU, the user question is inserted into a prompt template and passed to an LLM fine-tuned on DeKeyNLU. The model outputs task decomposition and keyword extraction. The decomposition is described as a “two-step CoT approach” that breaks the question into a main task and sub-tasks. The keyword output preserves DeKeyNLU’s object/implementation distinction rather than collapsing all terms into a flat keyword list.
The Entity Retrieval stage uses those outputs to retrieve table names, column names, table values, and textual descriptions. The module contains an embedder, retriever, and re-ranker, and all table data are encoded and stored in a Chroma database. The pipeline retrieves the top five most similar entities, re-ranks them, and keeps the top two (Chen et al., 18 Sep 2025). For database retrieval, the paper uses MinHash + Jaccard Score or BM25. It also specifies task-specific rules: for column names, select the top five entities with score ; for purely numeric keywords, only exact matches for table values are considered; for mixed text/numeric keywords, the top five scoring entities are selected without a threshold. Textual description retrieval uses an embedding model with cosine similarity to obtain top five candidates, followed by specialized re-ranking.
Generation then consumes the original question, the task decomposition from UQU, the retrieved entity information, schema data, hints from the dataset, and prompt-side constraints and incentives. The paper describes the prompt as being organized into four parts: data schema, user question reasoning, constraints, and incentives (Chen et al., 18 Sep 2025). If the resulting SQL fails or contains errors, a revision LLM receives the erroneous SQL and the corresponding error messages. The appendix notes that the revision module is activated only once during standard processing, although the authors also report a threshold sweep from 1 to 5 over 50 queries and state that a threshold of 3 gave the best balance.
A common misunderstanding is to treat DeKeyNLU as an isolated benchmark disconnected from downstream generation. In DeKeySQL, its outputs are structural inputs to both retrieval and generation, so the dataset is embedded into the pipeline rather than appended as an auxiliary evaluation layer.
5. Evaluation, model behavior, and benchmark results
The paper evaluates DeKeyNLU at both the NLU level and the end-to-end NL2SQL level. For task decomposition, the reported metrics are BLEU, ROUGE, and GPT-4o Score. For keyword extraction, the reported metrics are Precision, Recall, and F1 Score. For SQL generation, the metric is Execution Accuracy (EX), defined by comparing executed predicted query results with reference query results on specific database instances (Chen et al., 18 Sep 2025).
The paper does not provide an explicit training-objective equation for UQU fine-tuning, does not formalize task decomposition with a symbolic grammar, and does not print explicit formulas for BM25, Jaccard, cosine retrieval, or re-ranking. The main explicit equation is a calibration formula used to align GPT-4o-based decomposition evaluation with human judgments:
The paper reports that this calibration reduces the average difference between GPT-4o scores and human evaluations from 0.152 to 0.046 (Chen et al., 18 Sep 2025).
On the DeKeyNLU test set, GPT-4 is the strongest decomposition model among those listed, with BLEU 0.722, ROUGE 0.816, and GPT-4o Score 4.286. For keyword extraction, Mistral-7B is best, with F1 . GPT-4o-mini obtains 0.672 and GPT-4 obtains 0.665 on the same keyword-extraction metric (Chen et al., 18 Sep 2025). The paper interprets this as evidence that larger models are stronger on the more semantically complex decomposition task, whereas smaller models can be more effective for keyword extraction.
The dataset-size and epoch analysis is centered on keyword extraction F1. For Mistral-7B, the reported scores are 0.648 at 20% of the data, 0.640 at 40%, 0.634 at 60%, 0.694 at 80%, and 0.696 at 100%. With 1, 2, and 3 epochs, the model reaches 0.696, 0.755, and 0.769, respectively; without fine-tuning, it scores 0.502 (Chen et al., 18 Sep 2025). The paper notes that, except for Mistral-7B, performance first rises and then falls with more data, whereas increasing epochs consistently helps.
The main end-to-end result is that fine-tuning with DeKeyNLU improves SQL generation accuracy. The abstract reports BIRD dev improvement from 62.31% to 69.10% and Spider dev improvement from 84.2% to 88.7% (Chen et al., 18 Sep 2025). In the BIRD ablation table, the best UQU configuration is GPT-4o-mini fine-tuned on DeKeyNLU, combined with MinHash, Stella-400M, and GPT-4o generation, yielding 69.10 Dev EX. The comparison configuration without DeKeyNLU fine-tuning—GPT-4o, no DeKeyNLU fine-tuning, plus the same retrieval and generation backbone—reaches 62.31 Dev EX.
Against the listed baselines, DeKeySQL achieves 69.10 on BIRD dev, compared with 68.31 for CHESS, 67.21 for Distillery, 57.56 for MAC-SQL, and 46.35 for GPT-4. On Spider dev, it reaches 88.7, compared with 87.2 for CHESS, 86.7 for MAC-SQL, and 84.4 for Dail-SQL; on Spider test, it reaches 87.1, compared with 86.6 for Dail-SQL and 82.8 for MAC-SQL (Chen et al., 18 Sep 2025).
The module ablation table also matters for interpreting DeKeyNLU’s role. With GPT-4 as backbone, Generation Only yields 46.35 EX; Entity Retrieval + Generation yields 51.25; Entity Retrieval + Revision + Generation yields 55.28; and UQU + Entity Retrieval + Revision + Generation yields 60.36 (Chen et al., 18 Sep 2025). The source itself contains a mild inconsistency: the abstract states that user question understanding emerged as the most significant factor, whereas the conclusion states that entity retrieval was the most critical component, followed by user question understanding and revision. The ablation table supports a particularly strong role for UQU, while also showing that retrieval and revision materially contribute.
6. Significance, limitations, and relation to adjacent NLU traditions
DeKeyNLU is significant because it re-centers NL2SQL around question understanding before generation. The paper’s central claim is that strong SQL generation is insufficient if upstream decomposition and keyword extraction are unreliable. By annotating questions with SQL-relevant decomposition and database-grounded keyword categories, the dataset provides supervision for a stage that many pipelines previously handled only implicitly or via prompting (Chen et al., 18 Sep 2025).
Its practical value is clearest in modular settings that rely on retrieval over schema, values, and textual descriptions. The paper explicitly positions the approach for cross-domain, schema-rich NL2SQL applications such as enterprise data access assistants and database copilots, where complex multi-table questions make upstream interpretation errors costly (Chen et al., 18 Sep 2025).
The limitations are also explicit. DeKeyNLU contains only 1,500 samples. It is derived from BIRD, so transfer to completely unseen schemas and question types requires further study. The authors also note resource constraints, the inability to benchmark the largest LLMs or more advanced RAG modules, and potential annotation or data-availability constraints for expansion. They suggest future work on semi-automated annotation, data augmentation for structured NLU, adaptive decomposition granularity, and evaluation on unseen schemas (Chen et al., 18 Sep 2025).
The name “DeKeyNLU” also sits near several adjacent research traditions, but it does not denote the same problem as any of them. Delexicalized paraphrase generation focuses on intent–slot utterance expansion through delexicalized templates, slot-aware representations, and pointer-based slot placement (Yu et al., 2020). Iterative delexicalization treats placeholder insertion as an inference-time search problem guided by parser confidence in joint intent classification and slot tagging (Ray et al., 2019). DKPLM decomposes knowledge injection across pre-training, fine-tuning, and inference for knowledge-enhanced LLMs (Zhang et al., 2021). Knowledge-driven semantic representation work based on VerbNet and Answer Set Programming emphasizes explicit meaning representation, commonsense reasoning, and justification (Basu et al., 2021). DeKeyNLU differs from all four by locating its contribution in database-grounded task decomposition and keyword extraction for NL2SQL. This suggests that the term occupies a distinct niche: not delexicalized NLU, not knowledge-enhanced pre-training, and not symbolic text understanding in general, but structured front-end supervision for retrieval-conditioned SQL generation.
In that narrower sense, DeKeyNLU’s contribution is dataset-centric rather than architectural. Its importance lies in defining question understanding as a first-class, annotated interface between natural-language questions and the retrieval/generation machinery that produces executable SQL.