StatBot.Swiss: Bilingual Open Data Exploration in Natural Language (2406.03170v2)

Abstract: The potential for improvements brought by LLMs in Text-to-SQL systems is mostly assessed on monolingual English datasets. However, LLMs' performance for other languages remains vastly unexplored. In this work, we release the StatBot.Swiss dataset, the first bilingual benchmark for evaluating Text-to-SQL systems based on real-world applications. The StatBot.Swiss dataset contains 455 natural language/SQL-pairs over 35 big databases with varying level of complexity for both English and German. We evaluate the performance of state-of-the-art LLMs such as GPT-3.5-Turbo and mixtral-8x7b-instruct for the Text-to-SQL translation task using an in-context learning approach. Our experimental analysis illustrates that current LLMs struggle to generalize well in generating SQL queries on our novel bilingual dataset.

An Insightful Overview of StatBot.Swiss: Bilingual Open Data Exploration in Natural Language

The paper "StatBot.Swiss: Bilingual Open Data Exploration in Natural Language" presents a novel and comprehensive dataset aimed at evaluating the performance of Text-to-SQL (T2SQL) systems in bilingual settings, specifically English and German. This dataset, named StatBot.Swiss, includes 455 natural language and SQL query pairs sourced from 35 large databases encompassing various complex and realistic domains.

Objectives and Contributions

The primary objectives of the paper are twofold:

1. To introduce a real-world bilingual benchmark dataset for T2SQL systems. This dataset not only includes a significant number of natural language/SQL pairs but also reflects the diverse and complex nature of queries encountered in real applications.
2. To evaluate the performance of state-of-the-art LLMs such as GPT-3.5-Turbo and Mixtral-8x7B using in-context learning approaches on this dataset.

The StatBot.Swiss dataset sets itself apart from existing T2SQL benchmarks by including databases curated by domain experts, ensuring linguistic and contextual accuracy. Furthermore, it is the first dataset to offer a T2SQL benchmark in both English and German, thus addressing the multilingual aspect that has been largely unexplored in previous studies.

Experimental Setup and Evaluation Metrics

The authors conduct an extensive experimental assessment using two LLMs, GPT-3.5-Turbo and Mixtral-8x7B, by applying in-context learning (ICL) strategies. They explore both zero-shot and few-shot learning scenarios, comparing random and similarity-based example selection methods. A variety of evaluation metrics are employed, including strict execution accuracy (EA), soft EA, and partial EA, to provide a nuanced evaluation of model performance in generating correct SQL queries.

Key Findings

1. Performance of LLMs:
   • GPT-3.5-Turbo consistently outperforms Mixtral-8x7B across various prompting strategies, particularly excelling in few-shot settings where a small number of labeled examples are provided.
   • The best performance for GPT-3.5-Turbo is achieved with a five-shot similarity-based selection approach, yielding a strict execution accuracy of 41.68%.
2. Zero-shot vs Few-shot Learning:
   • Both models show substantial improvement when transitioning from zero-shot to few-shot learning scenarios, with GPT-3.5-Turbo increasing its performance by approximately 20 percentage points.
3. Language-specific Performance:
   • Models generally perform better on German datasets despite the higher linguistic complexity as indicated by metrics such as the Type-Token Ratio.
   • GPT-3.5-Turbo achieves 46.83% strict execution accuracy for German, compared to 34.43% for English in few-shot settings.
4. Complex Query Handling:
   • The LLMs exhibit difficulty with complex SQL structures involving multi-column GROUP BYs, built-in functions, and queries requiring domain-specific knowledge.
   • Soft and partial EA metrics reveal that models often generate queries that are useful and semantically correct, though not exact matches to the ground truth.

Implications and Future Directions

The StatBot.Swiss dataset is a significant contribution to the field, providing a robust benchmark for evaluating T2SQL systems in a bilingual context. The findings indicate that while state-of-the-art LLMs show promise, there is considerable room for improvement, particularly in handling complex queries and domain-specific nuances.

Future Research Directions:

• Cross-lingual T2SQL tasks: Extending the dataset to include other languages such as French and Italian to further explore the capabilities of multilingual models.
• Enhanced In-context Learning: Developing more sophisticated strategies for in-context learning that include better alignment of examples with target queries.
• Handling Complexity: Improving model architectures to better handle the complexity of SQL queries and domain-specific knowledge.

In conclusion, the StatBot.Swiss dataset provides a valuable resource for advancing the development of T2SQL systems. It highlights the importance of multilingual capabilities and the challenges associated with complex query generation, setting the stage for future research in this critical area of natural language processing and database interaction.