StructLM: Towards Building Generalist Models for Structured Knowledge Grounding (2402.16671v7)

Abstract: Structured data sources, such as tables, graphs, and databases, are ubiquitous knowledge sources. Despite the demonstrated capabilities of LLMs on plain text, their proficiency in interpreting and utilizing structured data remains limited. Our investigation reveals a notable deficiency in LLMs' ability to process structured data, e.g., ChatGPT lags behind state-of-the-art (SoTA) model by an average of 35%. To augment the Structured Knowledge Grounding (SKG) capabilities in LLMs, we have developed a comprehensive instruction tuning dataset comprising 1.1 million examples. Utilizing this dataset, we train a series of models, referred to as StructLM, based on the Mistral and the CodeLlama model family, ranging from 7B to 34B parameters. Our StructLM series surpasses task-specific models on 16 out of 18 evaluated datasets and establishes new SoTA performance on 8 SKG tasks. Furthermore, StructLM demonstrates strong generalization across 6 novel held-out SKG tasks, outperforming TableLlama by an average of 35\% and Flan-UL2 20B by an average of 10\%. Contrary to expectations, we observe that scaling model size offers marginal benefits, with StructLM-34B showing only slight improvements over StructLM-7B. This suggests that structured knowledge grounding is still a challenging task and requires more innovative design to push to a new level.

StructLM: Building Generalist Models for Structured Knowledge Grounding

The paper "StructLM: Towards Building Generalist Models for Structured Knowledge Grounding" presents a novel approach to enhancing LLMs to effectively process structured data sources such as tables, graphs, and databases. Despite the proficiency of LLMs with unstructured text, their capabilities with structured data have shown significant limitations. The researchers identified a marked deficiency in LLMs to handle structured inputs, with an example analysis demonstrating that ChatGPT underperforms against state-of-the-art (SoTA) models by 35% on average.

Main Contributions

The authors aimed to improve LLMs' Structured Knowledge Grounding (SKG) abilities by designing an extensive instruction tuning dataset encompassing 1.1 million examples. Utilizing this dataset, various models, collectively named StructLM, were trained based on the CodeLlama architecture with parameters ranging from 7B to 34B. Remarkably, StructLM models surpassed task-specific models across 14 of 18 evaluated datasets, achieving new SoTA results on 7 SKG tasks, and displaying superior generalization across novel tasks. Notably, the results revealed that mere scaling of model size offered marginal gains, as StructLM-34B showed only slight improvements over StructLM-7B, suggesting that structured knowledge grounding remains a challenging domain requiring innovative approaches.

Evaluation and Results

The StructLM models were meticulously evaluated against prominent baselines like GPT-3.5-Turbo and task-specific models. The findings demonstrated that the StructLM series not only exceeded SoTA results on several tasks but also offered a parameter-efficient solution. While the inferior capabilities of LLMs like ChatGPT on these tasks were made evident, StructLM's performance highlights the benefit of focused instruction tuning on structured tasks. The findings also showed improved cross-task generalization when utilizing a mixed dataset, compared to single-task models.

Ablation Studies

Further analysis was conducted to examine the effects of pretraining data types and the role of general instruction data. Code-pretrained models showed an edge in performance across diverse SKG tasks. The inclusion of general instruction data was found to significantly enhance zero-shot performance on held-out tasks, reducing overfitting to specific training formats.

Implications and Future Directions

The implications of this research stretch across both practical and theoretical domains. Practically, StructLM can enhance automation capabilities in applications involving databases and knowledge graphs, potentially streamlining question-answering, summarization, and fact verification. Theoretically, the findings suggest that specialized pretraining, such as on structured data formats, could prove worthwhile.

The paper identifies critical areas for further exploration, such as developing more diverse structured data representations during pretraining and employing constrained LLM evaluation methods. These directions point toward broadening the capabilities of LLMs in processing structured data and establishing SKG as a foundational capability.

The research represents a significant stride in addressing the structured knowledge grounding challenges, establishing a robust baseline for future advancements in LLM capabilities.