JEC-QA: A Legal-Domain Question Answering Dataset

Abstract: We present JEC-QA, the largest question answering dataset in the legal domain, collected from the National Judicial Examination of China. The examination is a comprehensive evaluation of professional skills for legal practitioners. College students are required to pass the examination to be certified as a lawyer or a judge. The dataset is challenging for existing question answering methods, because both retrieving relevant materials and answering questions require the ability of logic reasoning. Due to the high demand of multiple reasoning abilities to answer legal questions, the state-of-the-art models can only achieve about 28% accuracy on JEC-QA, while skilled humans and unskilled humans can reach 81% and 64% accuracy respectively, which indicates a huge gap between humans and machines on this task. We will release JEC-QA and our baselines to help improve the reasoning ability of machine comprehension models. You can access the dataset from http://jecqa.thunlp.org/.

• The paper introduces JEC-QA, the largest legal-domain QA dataset built from China's National Judicial Examination that challenges modern QA systems.
• It reveals a significant performance gap with machines at 28% accuracy compared to humans at 64-81%, stressing the need for enhanced reasoning.
• Baseline evaluations demonstrate the complexity of legal reasoning, establishing JEC-QA as a crucial benchmark for advancing QA technology.

"JEC-QA: A Legal-Domain Question Answering Dataset" introduces the JEC-QA dataset, which is tailored for the challenging task of question answering (QA) within the legal domain. This dataset is derived from the National Judicial Examination of China, an essential test for legal professionals in the country. The examination evaluates various professional skills necessary for becoming lawyers or judges, making the dataset comprehensive and demanding.

Key Contributions

1. Dataset and Scope:
   • JEC-QA: The dataset is presented as the largest of its kind in the legal domain, containing a significant number of QA pairs extracted from the National Judicial Examination.
   • Size and Complexity: The questions involve complex legal documents and require sophisticated reasoning abilities, making the dataset particularly difficult for standard QA systems.
2. Importance of Reasoning:
   • Logic Reasoning: Answering the questions in JEC-QA requires diverse reasoning skills, including logical deductions and the ability to synthesize information from multiple documents.
   • Performance Metrics: The authors highlight the discrepancy between machine learning models and human performance. State-of-the-art models attain approximately 28% accuracy on the dataset, whereas skilled and unskilled humans achieve 81% and 64% accuracy respectively. This significant gap underscores the complexity of the task and the need for improved reasoning in machine comprehension models.
3. Baselines and Challenges:
   • The paper provides baseline results using current state-of-the-art QA methods, emphasizing the difficulty posed by the dataset.
   • The authors suggest that the dataset will serve as a critical benchmark for enhancing the reasoning capabilities of future QA models.
4. Dataset Availability:
   • The JEC-QA dataset and the baselines mentioned in the paper are made publicly available to foster community engagement and progress in the domain of legal QA.

Implications and Future Work

The introduction of JEC-QA aims to spur advancements in machine understanding and reasoning, particularly within specialized domains like law. By releasing this dataset, the authors encourage further research into enhancing the reasoning abilities of QA systems, which currently lag significantly behind human performance. The paper implies that overcoming these challenges could have substantial implications for fields requiring intricate comprehension and reasoning of domain-specific texts, potentially leading to more sophisticated and capable AI systems.

In summary, "JEC-QA: A Legal-Domain Question Answering Dataset" presents a significant resource and benchmark that addresses a critical gap in the current capabilities of QA systems, particularly emphasizing the need for advanced reasoning skills in the legal domain. The dataset's release is pivotal for future research aimed at bridging the performance gap between machines and humans.