Tree Transformer: Integrating Tree Structures into Self-Attention (1909.06639v2)

Abstract: Pre-training Transformer from large-scale raw texts and fine-tuning on the desired task have achieved state-of-the-art results on diverse NLP tasks. However, it is unclear what the learned attention captures. The attention computed by attention heads seems not to match human intuitions about hierarchical structures. This paper proposes Tree Transformer, which adds an extra constraint to attention heads of the bidirectional Transformer encoder in order to encourage the attention heads to follow tree structures. The tree structures can be automatically induced from raw texts by our proposed "Constituent Attention" module, which is simply implemented by self-attention between two adjacent words. With the same training procedure identical to BERT, the experiments demonstrate the effectiveness of Tree Transformer in terms of inducing tree structures, better LLMing, and further learning more explainable attention scores.

Insights into the Tree Transformer Model for NLP

The paper presents the Tree Transformer, an innovative model designed to incorporate hierarchical tree structures into traditional self-attention mechanisms, as originally conceptualized in the Transformer architecture. This work aims to address the gap where self-attention, while effective in language representation learning, does not inherently capture hierarchical linguistic structures that align with human intuition.

Motivation and Approach

The rationale behind integrating tree structures into self-attention is grounded in the complexity and hierarchical nature of human languages, which are not fully leveraged by existing Transformer models. Existing methods like Tree-RNNs or Tree-LSTMs have explored hierarchical LLMing but faced limitations, especially when supervised syntactic parsers are unavailable. The Tree Transformer is an unsupervised approach that addresses these challenges by embedding tree induction directly within the Transformer framework.

The proposed architecture, termed the Tree Transformer, introduces an additional "Constituent Attention" module to the self-attention layers. This module uses a novel method to infer tree structures from raw text, promoting the development of explainable and interpretable attention scores. Unlike traditional approaches that require pre-annotated data, the Tree Transformer includes mechanisms like Neighboring Attention and Hierarchical Constraint to automatically induce constituent structures.

Key Contributions

1. Constituent Attention Module: The addition of a self-attention-based module that calculates a "Constituent Prior" to encourage attention heads to focus on linguistically plausible tree structures. This module operates with minimal changes to the standard Transformer architecture, simplifying integration.
2. Unsupervised Tree Induction: The Tree Transformer's ability to infer syntactic structures without explicit annotations is demonstrated through superior performance in unsupervised parsing tasks. Results indicate significant improvements in recognizing noun phrases and adverbial phrases.
3. Improved Masked LLMing: Empirical evaluations reveal that the Tree Transformer achieves lower perplexity on masked LLMing tasks compared to conventional Transformer models. This underscores the utility of tree structures in better understanding sentence semantics.

Experimental Evaluation

The evaluation is thorough, leveraging the WSJ test sets and comparing the model's performance with established baselines such as PRPN and On-LSTM. Tree Transformers with layers ranging from six to twelve demonstrate resilience in parsing tasks and are shown to benefit from deeper layers but plateau in performance beyond a certain depth.

Furthermore, the analysis of attention heads across layers indicates that the Tree Transformer interprets text more effectively and hierarchically, which aligns with the syntactic structures humans expect.

Implications and Future Work

The integration of tree structures into Transformer models not only enhances interpretability but also paves the way for more linguistically informed neural network design. This work opens multiple avenues for future exploration: refining the efficiency of tree induction, optimizing constituent attention mechanisms for various languages, and applying these enhancements to other domains within NLP where hierarchy might play a critical role.

The Tree Transformer represents a noteworthy advancement in natural language processing, challenging the notion of purely sequential language representation learning by reinforcing the significance of linguistic hierarchies.