Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources (2409.08239v1)

Abstract: LLMs still struggle in challenging scenarios that leverage structured data, complex reasoning, or tool usage. In this paper, we propose Source2Synth: a new method that can be used for teaching LLMs new skills without relying on costly human annotations. Source2Synth takes as input a custom data source and produces synthetic data points with intermediate reasoning steps grounded in real-world sources. Source2Synth improves the dataset quality by discarding low-quality generations based on their answerability. We demonstrate the generality of this approach by applying it to two challenging domains: we test reasoning abilities in multi-hop question answering (MHQA), and tool usage in tabular question answering (TQA). Our method improves performance by 25.51% for TQA on WikiSQL and 22.57% for MHQA on HotPotQA compared to the fine-tuned baselines.

• The paper presents a novel framework that generates and curates synthetic data grounded in real sources to enhance LLM reasoning.
• It employs a three-stage methodology involving data generation, model-guided curation, and fine-tuning for improved task performance.
• Experimental results demonstrate significant gains, with over 22% improvement in multi-hop and 25% in tabular question answering.

Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources

"Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources" is a paper that proposes a novel method for generating synthetic datasets tailored for enhancing the capabilities of LLMs. The method leverages existing external data sources and aims to produce high-quality synthetic data points that include intermediate reasoning steps grounded in real-world data. This is particularly significant in overcoming the limitations of LLMs when dealing with tasks that require structured data manipulation, complex reasoning, or tool usage.

Overview

The paper presents Source2Synth, a general framework for synthetic data generation and curation. This method follows three main stages:

1. Dataset Generation: This stage involves selecting a real data source, generating a seed topic to guide the creation of data examples, and constructing these examples step-by-step.
2. Dataset Curation: Here, the initially generated dataset is split into two slices. The first slice is used to fine-tune a model, which is then employed to curate and filter the second slice, enhancing the overall data quality.
3. Model Fine-tuning: The final stage involves fine-tuning an LLM on the curated synthetic dataset to improve performance on specific tasks.

Methodology

Dataset Generation

1. Data Source Selection: Real-world data sources such as Wikipedia articles or structured databases are selected. Unlike traditional approaches, this method does not require human-annotated data, thereby reducing cost and time.
2. Seed Generation: A seed topic is generated from the selected data source. This seed, derived from entities or factual statements, serves as the backbone for creating detailed and context-rich examples.
3. Constructing Examples: The seed is used to generate comprehensive data examples, including intermediate reasoning steps for challenging tasks such as multi-hop question answering (MHQA) or tool-based question answering (TQA).

Dataset Curation

1. Data Filtering: Generated examples are first used to fine-tune a model. This intermediate model (LLMsynth) is then applied to filter and curate the remaining data. Examples that fail to produce the correct answers within a defined number of trials are discarded.
2. Data Imputation: The model is tasked with re-constructing parts of the data, ensuring the final dataset is more coherent and natural.

Model Fine-tuning

Fine-tuning is performed using the curated dataset. This final model (LLMCurated) shows an enhanced ability to perform on the target task compared to the original LLM or models fine-tuned on non-curated synthetic data.

Applications

Two primary applications are explored:

1. Multi-hop Question Answering (MHQA): Using Wikipedia as the data source, Source2Synth generates multi-hop questions by leveraging interlinked articles. The effectiveness is validated on the HotPotQA dataset, showing significant performance improvements.
2. Tabular Question Answering (TQA): Using WikiSQL tables, the method generates SQL queries and their natural language counterparts. The resulting model, fine-tuned with Source2Synth generated data, demonstrates remarkable improvements on the WikiSQL benchmark.

Experimental Results

The paper reports strong numerical results:

• MHQA on HotPotQA: Source2Synth exhibits a performance improvement of 22.57% over fine-tuned baselines, with notable gains in handling complex bridge questions.
• TQA on WikiSQL: The model achieves a 25.51% improvement over fine-tuned baselines, highlighting the effective use of SQL for tabular data manipulation.

These substantial gains affirm the paper’s claims about Source2Synth's efficacy in generating high-quality synthetic data for complex reasoning and tool-based tasks.

Implications and Future Directions

Practically, the Source2Synth method reduces dependency on expensive and time-consuming human annotations, presenting a scalable solution for advancing LLM capabilities in nuanced tasks. Theoretically, it opens avenues for further research into automated data generation and curation methodologies.

Future developments could involve extending Source2Synth to other domains requiring intricate data manipulations, such as healthcare, finance, and scientific research. Exploring more sophisticated sampling techniques and handling larger-scale datasets could further refine the methodology and expand its applicability.

Conclusion

The Source2Synth framework introduces a robust method for synthetic data generation and curation grounded in real-world data sources, significantly enhancing the performance of LLMs in complex reasoning and data manipulation tasks. By addressing both practical and theoretical challenges, this approach contributes meaningfully to the field of artificial intelligence, offering a promising direction for future research and application.