DeepStruct: Pretraining of Language Models for Structure Prediction (2205.10475v2)

Abstract: We introduce a method for improving the structural understanding abilities of LLMs. Unlike previous approaches that finetune the models with task-specific augmentation, we pretrain LLMs on a collection of task-agnostic corpora to generate structures from text. Our structure pretraining enables zero-shot transfer of the learned knowledge that models have about the structure tasks. We study the performance of this approach on 28 datasets, spanning 10 structure prediction tasks including open information extraction, joint entity and relation extraction, named entity recognition, relation classification, semantic role labeling, event extraction, coreference resolution, factual probe, intent detection, and dialogue state tracking. We further enhance the pretraining with the task-specific training sets. We show that a 10B parameter LLM transfers non-trivially to most tasks and obtains state-of-the-art performance on 21 of 28 datasets that we evaluate.

DeepStruct: Pretraining of LLMs for Structure Prediction

The paper, "DeepStruct: Pretraining of LLMs for Structure Prediction," presents a methodology for enhancing the structural comprehension capabilities of LLMs (LMs), specifically targeting their application in structure prediction tasks. The research delineates a divergence from traditional approaches that focus on downstream task-specific fine-tuning, by pretraining LMs on a comprehensive set of task-agnostic corpora to generate structures in text.

Overview and Approach

The paper underscores the growing proficiency of pretrained LMs in executing diverse NLP tasks. However, it highlights a notable gap in their performance on structure prediction tasks, which require a nuanced understanding of structural details in text. Structure prediction tasks, such as open information extraction and named entity recognition, demand integration of multiple contextual aspects into a cohesive structure. DeepStruct introduces structure pretraining, designed specifically to teach LMs to comprehend and generate text structures during the pretraining phase, thus facilitating zero-shot transfer to downstream structure prediction tasks.

DeepStruct employs a methodology that reformulates structure prediction as a sequence of unit tasks focused on generating triples—comprising a head entity, relationship, and tail entity—from text. By pretraining LMs with task-agnostic structural corpora, the methodology achieves unification of multiple structure prediction tasks under a consistent task format. The efficiency of this approach is validated across 28 datasets encompassing 10 distinct structure prediction tasks, highlighting its efficiency in zero-shot and multi-task scenarios.

Empirical Findings

In empirical evaluations, DeepStruct was tested on 28 datasets covering tasks such as open information extraction, entity and relation extraction, and dialogue state tracking. Significant findings include:

• DeepStruct achieved state-of-the-art performance on 21 out of the 28 datasets.
• A 10B parameter LM demonstrated substantial improvements over smaller models, indicating a scaling law benefit where larger models translate learned knowledge more effectively.
• In direct comparisons to 175B parameter GPT-3, DeepStruct's approach outperformed GPT-3 in select zero-shot structure prediction benchmarks, emphasizing the effectiveness of structure pretraining.

Practical and Theoretical Implications

The implications of this paper are twofold. Practically, the success of DeepStruct in achieving competitive results in zero-shot structure prediction opens pathways for applying pretrained LMs to a broader spectrum of tasks without necessitating task-specific architecture designs or extensive labeled datasets. Theoretically, the research advocates for the exploration of pretraining strategies that can endow LMs with higher-level cognitive abilities, specifically structural understanding. It suggests this as a more sophisticated measure of LM competence beyond traditional token prediction tasks.

Furthermore, the paper posits that unifying structure prediction tasks via structure pretraining could enhance the adaptability and efficiency of LMs in practical applications, as these tasks often converge across different domains and languages.

Future Directions

The research points towards several future directions, including the exploration of larger and more diverse pretraining datasets, the development of improved methods for schema alignment between pretraining and task-specific datasets, and the refinement of model architectures that can better leverage the structural knowledge acquired during pretraining. The paper also suggests the potential for integrating structure pretraining approaches with generative models like T5 and BART to further advance the field of NLP.

In summary, DeepStruct provides a comprehensive blueprint for revolutionizing the application of pretrained LMs in structure prediction tasks by leveraging structure pretraining. The approach not only enhances LM capabilities but also bridges the gap between structural understanding and traditional NLP tasks, paving the way for sophisticated AI systems that are responsive to structural nuances in language data.