A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Prediction (2206.05224v2)

Abstract: The recent advances of deep learning have dramatically changed how machine learning, especially in the domain of natural language processing, can be applied to legal domain. However, this shift to the data-driven approaches calls for larger and more diverse datasets, which are nevertheless still small in number, especially in non-English languages. Here we present the first large-scale benchmark of Korean legal AI datasets, LBOX OPEN, that consists of one legal corpus, two classification tasks, two legal judgement prediction (LJP) tasks, and one summarization task. The legal corpus consists of 147k Korean precedents (259M tokens), of which 63k are sentenced in last 4 years and 96k are from the first and the second level courts in which factual issues are reviewed. The two classification tasks are case names (11.3k) and statutes (2.8k) prediction from the factual description of individual cases. The LJP tasks consist of (1) 10.5k criminal examples where the model is asked to predict fine amount, imprisonment with labor, and imprisonment without labor ranges for the given facts, and (2) 4.7k civil examples where the inputs are facts and claim for relief and outputs are the degrees of claim acceptance. The summarization task consists of the Supreme Court precedents and the corresponding summaries (20k). We also release realistic variants of the datasets by extending the domain (1) to infrequent case categories in case name (31k examples) and statute (17.7k) classification tasks, and (2) to long input sequences in the summarization task (51k). Finally, we release LCUBE, the first Korean legal LLM trained on the legal corpus from this study. Given the uniqueness of the Law of South Korea and the diversity of the legal tasks covered in this work, we believe that LBOX OPEN contributes to the multilinguality of global legal research. LBOX OPEN and LCUBE will be publicly available.

• The paper introduces LBox Open, a multi-task benchmark integrating six legal datasets and LCube, a domain-specific language model for Korean legal texts.
• It employs tailored deep learning methodologies, achieving near mt5-large performance in classification tasks with lower computational requirements.
• The research addresses challenges in data heterogeneity and fairness, paving the way for more equitable and effective legal AI systems.

An Overview of "A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Prediction"

This paper introduces "LBox Open," a pioneering benchmark for Korean legal language understanding and judgment prediction tasks in AI. Historically, the application of AI in the legal domain has progressed substantially with advances in deep learning. However, most of these advancements have centered on English-language resources, leading to an imbalance in accessible datasets for non-English languages like Korean.

Key Components of LBox Open

LBox Open is particularly notable for its comprehensive compilation of datasets, which consists of six distinct legal data resources:

1. Precedent Corpus: A collection of 147k Korean legal precedents amounting to 259 million tokens. This corpus is unique for including recent judgments and cases from first and second-tier district courts, capturing vital factual arguments and legal decision-making processes.
2. Classification Tasks:
   • Case Name Classification: Comprising 11.3k instances, this task involves predicting case names from factual descriptions.
   • Statute Classification: Including 2.8k instances, this task involves predicting applicable statutes from case facts.
3. Legal Judgment Prediction (LJP) Tasks:
   • LJP-Criminal: Featuring 10.5k criminal cases with tasks to predict fine amounts and imprisonment durations based on legal facts.
   • LJP-Civil: Covering 4.7k civil cases, this task predicts the degree of claim acceptance, analyzing how much of the claimed amount is granted.
4. Summarization Task: Featuring 20k precedent summaries from the Korean Supreme Court, aimed at generating concise legal document summaries.

Legal LLM Development

A significant addition to this work is "LCube," the first large-scale legal LLM pre-trained using the Precedent Corpus. This model is based on GPT-2 and is specifically fine-tuned to surpass the limitations faced by general domain models in handling legal text.

Empirical Results

The authors highlight the necessity for domain-specific training data. Experiments using LCube show improved performance on legal domain tasks compared to models trained on generic datasets. For instance, in classification tasks, LCube demonstrates performance close to larger models like mt5-large but with significantly reduced computational requirements. However, for tasks like summarization, encoder-decoder architectures such as mt5 still perform better due to their ability to handle long context inputs better.

Challenges and Implications

The research underscores several challenges intrinsic to legal AI tasks:

• Data Heterogeneity: Legal documents contain complex language, sensitive topics, and anonymized information, requiring sophisticated handling.
• Bias and Fairness: The authors address potential biases, such as gender inconsistencies in judgment predictions, advocating for transparency and caution in ML model applications in legal settings.
• Generalizability: The uniqueness of Korean law necessitates culture-specific adaptations, which might not directly transfer to legal systems elsewhere.

Future Prospects

The paper sets a foundation for future research in multilingual legal AI, aiming to bridge gaps between legal and computational paradigms. The release of LBox Open and LCube is anticipated to stimulate development and fairness in AI applications across different legal systems, encouraging further augmentation of this dataset with additional legal understanding tasks.

In summary, this paper presents a foundational step towards building equitable and functional legal AI systems for non-English languages. It emphasizes the need for contextual understanding through advanced, domain-specific datasets and models that reflect the nuances of complex legal documentation. With these resources made publicly available, they promise to enhance the scope of legal AI research worldwide.