Language Models as Hierarchy Encoders (2401.11374v4)

Abstract: Interpreting hierarchical structures latent in language is a key limitation of current LMs. While previous research has implicitly leveraged these hierarchies to enhance LMs, approaches for their explicit encoding are yet to be explored. To address this, we introduce a novel approach to re-train transformer encoder-based LMs as Hierarchy Transformer encoders (HiTs), harnessing the expansive nature of hyperbolic space. Our method situates the output embedding space of pre-trained LMs within a Poincar\'e ball with a curvature that adapts to the embedding dimension, followed by training on hyperbolic clustering and centripetal losses. These losses are designed to effectively cluster related entities (input as texts) and organise them hierarchically. We evaluate HiTs against pre-trained LMs, standard fine-tuned LMs, and several hyperbolic embedding baselines, focusing on their capabilities in simulating transitive inference, predicting subsumptions, and transferring knowledge across hierarchies. The results demonstrate that HiTs consistently outperform all baselines in these tasks, underscoring the effectiveness and transferability of our re-trained hierarchy encoders.

• The paper presents a novel approach by re-training language models as Hierarchy Transformer encoders (HiTs) to better capture hierarchical structures.
• It employs hyperbolic clustering and centripetal losses, achieving superior performance in transitive inference and subsumption prediction compared to traditional models.
• Evaluations on datasets like WordNet demonstrate that HiTs more accurately position entities within semantic hierarchies, indicating strong potential for hierarchy-based applications.

Introduction

LMs have marked a significant progression in NLP, with transformer-based models such as BERT, GPT, and other LLMs like GPT-4 and Llama 2 achieving remarkable success. Despite their advancements, encoding and interpreting hierarchical structures latent in language remains a challenge for current LMs. Hanna & Mareček (2021) and He et al. (2023b) demonstrated the limited hierarchical knowledge in pre-trained LMs, while Lin & Ng (2022) showed these models' struggles with transitivity of hierarchical relationships. Various methods to incorporate hierarchical information into LMs have been explored, yet explicit encoding of hierarchies warrants further attention.

Novel Approach: Hierarchy Transformer encoders (HiTs)

This paper bridges the gap with a novel approach for re-training transformer encoder-based LMs as Hierarchy Transformer encoders (HiTs), utilizing hyperbolic geometry's effectiveness in representing hierarchical structures. The method re-situates the output embedding space of LMs within a Poincaré ball with a curvature adapting to the embedding dimension. Hyperbolic clustering and centripetal losses are introduced to cluster entities and organize them hierarchically. HiTs are evaluated against pre-trained and fine-tuned LMs, demonstrating superior performance in simulating transitive inference, predicting subsumptions, and knowledge transfer across hierarchies.

Implementation Insights and Evaluation

The paper provides a comprehensive overview of key concepts including transformer encoder-based LMs and hyperbolic geometry, also providing a formal definition of hierarchy. The method harnesses the output embedding space of the transformer encoder-based LMs, typically within a d-dimensional hyper-cube due to the tanh activation function, and constructs a Poincaré ball with a boundary that encircles the hyper-cube, using hyperbolic space for re-training. HiTs’ capabilities are showcased in the Multi-hop Inference and Mixed-hop Prediction tasks. The evaluation leverages datasets derived from WordNet’s noun hierarchy and other ontologies, illustrating that HiTs significantly outperform the pre-trained and fine-tuned LMs.

Analysis and Future Work

The analysis of HiT embeddings presents a clear distribution of WordNet entity embeddings concerning their hyperbolic norms, indicating an effective capture of hierarchical expansions. Selected cases provide evidence of HiT’s effectiveness, with more specific entities located further from the manifold’s origin, suggesting a clear hierarchical positioning. This underlines HiTs’ potential in hierarchy-oriented semantic search, earmarked as a direction for future exploration.

In conclusion, the paper presents an innovative approach to extend the capabilities of LMs for better encoding hierarchical structures. The introduction of HiTs represents a promising direction in the deployment of LMs for tasks demanding an understanding of complex semantic hierarchies, demonstrating the potential for significant advancements in hierarchy-oriented applications within NLP.