Why LLMs Are Bad at Synthetic Table Generation (and what to do about it) (2406.14541v3)

Abstract: Synthetic data generation is integral to ML pipelines, e.g., to augment training data, replace sensitive information, and even to power advanced platforms like DeepSeek. While LLMs fine-tuned for synthetic data generation are gaining traction, synthetic table generation -- a critical data type in business and science -- remains under-explored compared to text and image synthesis. This paper shows that LLMs, whether used as-is or after traditional fine-tuning, are inadequate for generating synthetic tables. Their autoregressive nature, combined with random order permutation during fine-tuning, hampers the modeling of functional dependencies and prevents capturing conditional mixtures of distributions essential for real-world constraints. We demonstrate that making LLMs permutation-aware can mitigate these issues.

• The paper identifies limitations in LLMs for capturing functional dependencies in tabular data, showing that traditional fine-tuning is inadequate.
• It introduces Permutation-Aided Fine-Tuning (PAFT) to incorporate domain-specific constraints, significantly reducing dependency violations.
• Empirical results demonstrate that PAFT-generated data closely match original distributions and enhance performance in downstream tasks.

Are LLMs Naturally Good at Synthetic Tabular Data Generation?

The paper "Are LLMs Naturally Good at Synthetic Tabular Data Generation?" investigates the potential of LLMs to generate synthetic tabular data, a largely underexplored application in the domain of AI. The research demonstrates that conventional LLMs, even after typical fine-tuning, perform suboptimally when tasked with generating synthetic tabular data due to the specific requirements of this data type, such as the need to model mixtures of distributions and functional dependencies between columns.

Summary of Contributions

1. Identification of Deficiencies in LLMs for Tabular Data Generation: The authors first highlight the inherent limitations of autoregressive LLMs in generating tabular data. They critically examine the performance of existing state-of-the-art models, showing that these models often fail to capture the dependencies between columns accurately. Consequently, these models produce data that violate inherent relationships present in the original dataset.
2. Introduction of Permutation-Aided Fine-Tuning (PAFT): To address these deficiencies, the authors propose Permutation-aided Fine-Tuning (PAFT). PAFT injects knowledge of pre-existing functional dependencies among columns into the LLM fine-tuning process. This method involves the discovery and distillation of functional dependencies, organizing them into a column dependence graph. The graph informs an optimal permutation function that governs the auto-regressive generation process, ensuring the generated data respects real-world constraints.
3. Empirical Evaluation: The paper conducts rigorous experiments on multiple datasets, including both real-world and synthetic data, to evaluate the performance of PAFT. The proposed method demonstrates superior performance in maintaining functional dependencies and reducing violations compared to existing models, such as CTGAN, CopulaGAN, and GReaT.

Key Findings

• Functional Dependency Retention: PAFT significantly reduces the violation rates of real-world constraints in generated data. For instance, in a complex dataset involving US state locations, PAFT achieves considerably lower violation rates compared to the baseline methods. This demonstrates PAFT's efficacy in capturing both statistical correlations and real-world constraints.
• Distributional Fidelity: The generated data using PAFT more closely matches the distribution of original data. The evaluation metrics, such as Kolmogorov-Smirnov Test (KST) for numerical columns and Total Variation Distance (TVD) for categorical columns, indicate that PAFT-generated data are more faithful to the real data distributions.
• Utility in Downstream Tasks: Experiments reveal that models trained on PAFT-generated data perform comparably to those trained on real data in downstream machine learning tasks. This indicates that PAFT holds promise for applications where synthetic data can be used to augment training datasets while preserving predictive performance.

Implications and Future Directions

The research underscores the importance of considering functional dependencies in the generative modeling of tabular data, highlighting that merely relying on joint distribution learning can be insufficient. PAFT can be seen as a step forward in making LLMs more versatile by enabling them to handle tabular data effectively.

1. Practical Implications: The ability to generate high-quality synthetic tabular data has significant practical applications in various domains such as privacy-preserving data analysis, synthetic data augmentation for machine learning, and testing of data-driven systems.
2. Theoretical Implications: The paper sheds light on a fundamental aspect of LLMs—their alignment with auto-regressive generation and feature ordering. This alignment can be further explored to develop more sophisticated models and fine-tuning techniques that inherently understand and leverage data constraints.
3. Future Research: Several avenues for future work are suggested:
   • Scalability: Scaling PAFT to handle larger datasets and more complex dependencies remains an open challenge. Future research could aim at optimizing the computational efficiency of the PAFT algorithm.
   • Privacy Preservation: Integrating privacy-preserving mechanisms into the fine-tuning process could make PAFT applicable in sensitive domains where data privacy is paramount.
   • Enhanced Constraints: Exploring other types of constraints beyond functional dependencies, such as temporal dependencies and relational constraints, could further enhance the quality and applicability of synthetic data generated by LLMs.

Conclusion

The paper makes a compelling case for the need to adapt LLMs to better capture the nuances of tabular data. By introducing Permutation-aided Fine-Tuning (PAFT), the authors provide a robust method for improving the fidelity of synthetic tabular data generation. The research opens up new possibilities for leveraging LLMs in yet another domain, offering insights that could drive future innovations in AI-driven data generation techniques.