StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding (1908.04577v3)

Abstract: Recently, the pre-trained LLM, BERT (and its robustly optimized version RoBERTa), has attracted a lot of attention in natural language understanding (NLU), and achieved state-of-the-art accuracy in various NLU tasks, such as sentiment classification, natural language inference, semantic textual similarity and question answering. Inspired by the linearization exploration work of Elman [8], we extend BERT to a new model, StructBERT, by incorporating language structures into pre-training. Specifically, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential order of words and sentences, which leverage language structures at the word and sentence levels, respectively. As a result, the new model is adapted to different levels of language understanding required by downstream tasks. The StructBERT with structural pre-training gives surprisingly good empirical results on a variety of downstream tasks, including pushing the state-of-the-art on the GLUE benchmark to 89.0 (outperforming all published models), the F1 score on SQuAD v1.1 question answering to 93.0, the accuracy on SNLI to 91.7.

• The paper introduces auxiliary pre-training tasks—word-order recovery and three-option sentence prediction—that enhance BERT's structural language comprehension.
• The word structural objective refines token order understanding, contributing to a 5% improvement in grammatical error detection on GLUE's CoLA task.
• The sentence structural objective bolsters inter-sentence coherence, achieving 91.7% accuracy on SNLI and improved performance on SQuAD.

StructBERT: Incorporating Language Structures into Pre-training

The paper "StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding" introduces a modification to the widely implemented BERT model, termed as StructBERT. The goal is to enhance the language representation abilities of BERT by incorporating more explicit language structural information during the pre-training phase.

Overview of StructBERT

StructBERT extends the base architecture of BERT, which exploits a multi-layer Transformer network for bidirectional context integration. The innovation primarily lies in the introduction of two additional auxiliary pre-training tasks—word structural and sentence structural objectives. These tasks are designed to enrich the model's understanding of internal sentence structures and inter-sentence relationships respectively.

Word Structural Objective: StructBERT supplements the traditional masked LLM (MLM) task with a word-order recovery component. Tokens are shuffled, and the model must predict their correct sequential order, enhancing the model's capability to understand word dependencies.

Sentence Structural Objective: Inspired by the perceived simplicity of BERT’s Next Sentence Prediction (NSP) task, StructBERT introduces a three-option sentence prediction task that requires the model to identify whether the second sentence follows the first, precedes it, or is randomly selected from another document. This approach more robustly conditions the model to understand sentence coherence and order.

Empirical Performance

Empirical results illustrate StructBERT's efficacy across multiple natural language understanding benchmarks. On the GLUE benchmark, StructBERT achieved an average score of 83.9, with significant improvements in tasks such as CoLA, where it demonstrated a 5% enhancement over BERT in grammatical error detection contexts. This improvement can be attributed to the word structural objective, which finely tunes the model's sensitivity to word order.

Particularly notable are the improvements on the SNLI dataset, where StructBERT outperformed prior models, achieving an accuracy of 91.7%. The sentence structural objective likely plays a critical role here, facilitating better sentence relation modeling.

Moreover, for the SQuAD v1.1 dataset, StructBERT improved fine-grained answer extraction capabilities, demonstrating an F1 score of 93.0, attesting to the model's enhanced understanding of nuanced textual relationships.

Implications and Future Directions

The explicit modeling of language structures as proposed by StructBERT offers significant potential for not only improving downstream task performance but also providing a more interpretable learning approach for NLP models. This method exemplifies the benefit of integrating linguistic cues into deep learning architectures for improved performance.

Looking forward, augmenting the StructBERT methodology with more sophisticated language and discourse level objectives could further enhance its utility in broader NLP applications. Additionally, leveraging similar structural tasks in conjunction with recently proposed models like RoBERTa and XLNet shows promise for setting new state-of-the-art results across diverse datasets.

In conclusion, StructBERT's integration of syntactic and sequential structures into pre-training represents a thoughtful evolution in contextual LLMing, achieving substantial performance gains while also enriching the semantic underpinnings of natural language processing tasks.