CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases (1909.05378v1)

Abstract: We present CoSQL, a corpus for building cross-domain, general-purpose database (DB) querying dialogue systems. It consists of 30k+ turns plus 10k+ annotated SQL queries, obtained from a Wizard-of-Oz (WOZ) collection of 3k dialogues querying 200 complex DBs spanning 138 domains. Each dialogue simulates a real-world DB query scenario with a crowd worker as a user exploring the DB and a SQL expert retrieving answers with SQL, clarifying ambiguous questions, or otherwise informing of unanswerable questions. When user questions are answerable by SQL, the expert describes the SQL and execution results to the user, hence maintaining a natural interaction flow. CoSQL introduces new challenges compared to existing task-oriented dialogue datasets:(1) the dialogue states are grounded in SQL, a domain-independent executable representation, instead of domain-specific slot-value pairs, and (2) because testing is done on unseen databases, success requires generalizing to new domains. CoSQL includes three tasks: SQL-grounded dialogue state tracking, response generation from query results, and user dialogue act prediction. We evaluate a set of strong baselines for each task and show that CoSQL presents significant challenges for future research. The dataset, baselines, and leaderboard will be released at https://yale-lily.github.io/cosql.

Overview of CoSQL: A Conversational Text-to-SQL Challenge

The paper presents CoSQL, a comprehensive dataset intended to advance the development of conversational database querying systems. By simulating real-world database interactions through dialogues, CoSQL introduces challenges that demand both a deep understanding of user intent and the ability to generate SQL queries that interface seamlessly with databases. This corpus enables research into building systems that interpret and execute natural language queries across multiple domains, thereby contributing significantly to the natural language interfaces to databases (NLIDBs).

Dataset Composition and Objectives

CoSQL is composed of over 3,000 dialogues with more than 30,000 conversational turns and 10,000 annotated SQL queries. This dataset spans over 200 intricate databases encompassing 138 domains. It emerges from a Wizard-of-Oz setting where crowd workers and SQL experts engage in dialogue, with the former exploring the database and the latter retrieving and clarifying responses. CoSQL stands out by grounding dialogue states in SQL rather than in domain-specific slot-value pairs, thus pushing the boundaries of NLIDBs by requiring these systems to generalize across unseen databases.

Challenges and Innovation

CoSQL introduces three critical tasks:

1. SQL-Grounded Dialogue State Tracking (DST): This task involves translating user questions into SQL queries considering the dialogue history and database schema. Unlike typical DST challenges, CoSQL's DST task includes resolving ambiguous queries by leveraging system clarifications given during dialogues.
2. Response Generation from SQL and Query Results: Following the execution of a SQL query, systems must generate responses that clearly describe both the SQL logic and the execution results. This task demands generating natural language descriptions that accurately capture SQL semantics, making SQL results interpretable by lay users.
3. User Dialogue Act Prediction: Systems must categorize user inputs into predefined dialogue action types to decide further actions, such as whether a SQL query should be generated or clarification is needed.

Baseline Evaluation

The paper evaluates a series of baseline models across the introduced tasks, highlighting the complexity of CoSQL:

• For DST, models based on seq2seq architectures and specialized SQL generation approaches achieve limited success, pointing to the intricate dependencies introduced by conversational contexts in CoSQL.
• Response generation models, both template-based and neural, show difficulty in preserving logical correctness, thus indicating a significant gap in current technologies for generating SQL-executed responses.
• Dialogue act prediction baselines demonstrate reasonable performance for frequent actions, though more nuanced actions remain challenging, emphasizing the need for more robust understanding models.

Implications and Future Directions

The research facilitated by CoSQL is poised to significantly impact both the practical and theoretical realms of AI. Practically, advancing NLIDBs can democratize data access, allowing users from various backgrounds to interface directly with complex databases using natural language. Theoretically, CoSQL challenges the limits of current models and encourages the exploration of dialogue systems that handle more comprehensive interaction phenomena, such as ambiguity resolution, multi-turn reasoning, and user clarification.

Future work might focus on improved semantical parsing strategies that better capture conversational context and user intent. Moreover, research could explore integrating external knowledge bases to enhance system responses, addressing contextually rich, ambiguous, or inferential queries more effectively. CoSQL thus lays a fertile ground for innovations aimed at realizing robust conversational database query systems adaptable across diverse domains.