The CLC-UKET Dataset: Benchmarking Case Outcome Prediction for the UK Employment Tribunal (2409.08098v3)

Abstract: This paper explores the intersection of technological innovation and access to justice by developing a benchmark for predicting case outcomes in the UK Employment Tribunal (UKET). To address the challenge of extensive manual annotation, the study employs a LLM for automatic annotation, resulting in the creation of the CLC-UKET dataset. The dataset consists of approximately 19,000 UKET cases and their metadata. Comprehensive legal annotations cover facts, claims, precedent references, statutory references, case outcomes, reasons and jurisdiction codes. Facilitated by the CLC-UKET data, we examine a multi-class case outcome prediction task in the UKET. Human predictions are collected to establish a performance reference for model comparison. Empirical results from baseline models indicate that finetuned transformer models outperform zero-shot and few-shot LLMs on the UKET prediction task. The performance of zero-shot LLMs can be enhanced by integrating task-related information into few-shot examples. We hope that the CLC-UKET dataset, along with human annotations and empirical findings, can serve as a valuable benchmark for employment-related dispute resolution.

• The paper presents the CLC-UKET dataset of approximately 19,000 annotated cases to enable precise benchmarking of legal outcome predictions.
• It benchmarks various models, showing that fine-tuned transformer architectures like T5 outperform advanced LLMs in multi-class classification tasks.
• The study underscores the benefit of combining human annotations with zero-shot and few-shot learning to enhance the accuracy and reliability of legal predictions.

The CLC-UKET Dataset: Benchmarking Case Outcome Prediction for the UK Employment Tribunal

This paper presents a significant dataset, CLC-UKET, engineered for benchmarking case outcome predictions specific to the UK Employment Tribunal (UKET). The research encompasses the development of a dataset that comprises approximately 19,000 UKET cases, complete with comprehensive annotations including facts, claims, case outcomes, and statutory references. The paper leverages a combination of human annotations and LLMs for creating legal annotations, illustrating the integration of automation and expert knowledge in legal data processing.

Empirical Evaluation and Findings

A prominent aspect of the investigation involves a multi-class classification task where various baseline models were evaluated. Among these models, transformer-based architectures such as BERT and T5, and advanced LLMs like GPT-3.5 and GPT-4, were employed. It was observed that fine-tuned transformer models notably surpassed LLMs in terms of classification performance. Specifically, fine-tuned T5 achieved superior results compared to other methodologies.

Moreover, the research illustrates the resourcefulness of zero-shot and few-shot learning paradigms when supplemented with task-specific information. Few-shot prediction employing similar jurisdiction codes slightly ameliorated the predictive capability of LLMs, an observation that may guide future adaptations in legal AI applications. Furthermore, human predictions in this paper underscored the potential of leveraging expert annotations to calibrate and potentially enhance AI models.

Implications and Future Directions

The implications of the CLC-UKET dataset are twofold—practically, it serves as an invaluable resource for legal professionals and policymakers to anticipate case outcomes and streamline the adjudication process in employment-related disputes. Theoretically, it furnishes a corpus that facilitates the exploration of NLP techniques, legal reasoning frameworks, and domain-specific LLMing.

The deployment of LLMs to automate annotation highlights a significant stride towards increased efficiency and scalability in legal dataset augmentation. However, it reveals inherent limitations such as the occasional loss of nuanced legal context and sentiment that might impact the outcomes of AI predictions.

Future research endeavors could focus on refining LLM architectures to better capture legal discourse intricacies, employing semi-supervised learning to enhance data representation, and integrating retrieval-augmented generation techniques to address context lack. Additionally, combining human inputs with AI predictions iteratively could further refine the accuracy and reliability of legal predictions, enhancing access to justice through faster, fair, and informed legal decision-making processes.

Overall, the CLC-UKET dataset sets a precedent for data-driven legal technology applications and represents an essential step forward in merging computational techniques with jurisprudence to address real-world challenges in the legal domain.