LexGLUE: A Benchmark Dataset for Legal Language Understanding in English (2110.00976v4)

Abstract: Laws and their interpretations, legal arguments and agreements\ are typically expressed in writing, leading to the production of vast corpora of legal text. Their analysis, which is at the center of legal practice, becomes increasingly elaborate as these collections grow in size. Natural language understanding (NLU) technologies can be a valuable tool to support legal practitioners in these endeavors. Their usefulness, however, largely depends on whether current state-of-the-art models can generalize across various tasks in the legal domain. To answer this currently open question, we introduce the Legal General Language Understanding Evaluation (LexGLUE) benchmark, a collection of datasets for evaluating model performance across a diverse set of legal NLU tasks in a standardized way. We also provide an evaluation and analysis of several generic and legal-oriented models demonstrating that the latter consistently offer performance improvements across multiple tasks.

An Overview of the LexGLUE Benchmark for Legal Language Understanding

The paper "LexGLUE: A Benchmark Dataset for Legal Language Understanding in English" addresses the niche yet significant challenge of legal natural language understanding (NLU) by introducing a standardized benchmark, LexGLUE, for evaluating the performance of NLU models on legal text. The research identifies the unique characteristics inherent in legal language and tasks, which differentiate them from more generic NLU challenges, and aims to assess model capabilities in handling these distinctions.

Motivation and Context

Legal texts, including legislation, judicial decisions, and contracts, are dense, complex, and specific. They also encompass unique linguistic constructs and terminologies distinct from those seen in everyday language. Current state-of-the-art NLU technologies primarily trained on general-domain data may falter when extended to legal documents due to this specificity. The introduction of LexGLUE seeks to create a clear performance benchmark for legal-focused NLU models and to explore whether domain-specific pre-training can enhance their effectiveness.

The LexGLUE Benchmark

LexGLUE is built upon seven existing legal datasets, selected based on certain desiderata, including language (English), substance, difficulty, and availability. These datasets involve varied tasks central to legal practice, such as judgment prediction, legal topic classification, and legal QA, among others.

Datasets and Tasks

1. ECtHR Tasks A and B: These tasks focus on the European Court of Human Rights cases, aiming to predict articles of the European Convention on Human Rights that were allegedly or actually violated based on factual paragraphs from case descriptions.
2. SCOTUS: Based on decisions from the US Supreme Court, the task involves classifying court opinions into issue areas, providing a real-world assessment of the model's ability to process legal arguments.
3. EUR-LEX: This involves classifying European Union legislation into various EuroVoc concepts, focusing on the applicability of legislative acts.
4. LEDGAR: Extracted from US Securities and Exchange Commission filings, this dataset is centered around contract provision classification.
5. UNFAIR-ToS: This dataset involves determining the presence of unfair terms in online Terms of Service, assessing adaptability in consumer protection law contexts.
6. CaseHOLD: Focused on hold summary extraction from court cases, it challenges models to choose the correct summary holding for a legal excerpt.

Evaluated Models

The benchmark evaluates both generic pre-trained Transformer models (e.g., BERT, RoBERTa, DeBERTa) and legal-specific models (e.g., Legal-BERT, CaseLaw-BERT). Results demonstrate that legal-specific models generally outperform generic ones on legal tasks, especially those heavily reliant on domain knowledge, such as legal text from the US legal system seen in SCOTUS and CaseHOLD datasets.

Implications and Future Directions

The paper concludes that domain adaptation significantly enhances the performance of NLU models on legal tasks. However, the results also indicate substantial room for improvement across tasks, motivating the further investigation into enhanced pre-training strategies, hierarchical model structures for handling lengthy documents, and the eventual development of larger and more diverse legal corpora for pre-training. It also highlights the need for developing methodologies to allow for structured text processing that takes into account the hierarchical nature of legal documents.

Conclusion

LexGLUE offers a robust framework for advancing legal NLU by facilitating transparency and benchmarking. The paper's findings underscore the necessity of developing domain-specific machine learning models to meet the evolving complexities of legal language processing. Researchers are encouraged to utilize this benchmark to continue pushing the boundaries of what is achievable in the domain of legal language understanding, potentially leading to more precise and efficient tools that assist legal practitioners in managing the intricacies of legal texts.