2000 character limit reached
NLP at UC Santa Cruz at SemEval-2024 Task 5: Legal Answer Validation using Few-Shot Multi-Choice QA (2404.03150v1)
Published 4 Apr 2024 in cs.CL and cs.AI
Abstract: This paper presents our submission to the SemEval 2024 Task 5: The Legal Argument Reasoning Task in Civil Procedure. We present two approaches to solving the task of legal answer validation, given an introduction to the case, a question and an answer candidate. Firstly, we fine-tuned pre-trained BERT-based models and found that models trained on domain knowledge perform better. Secondly, we performed few-shot prompting on GPT models and found that reformulating the answer validation task to be a multiple-choice QA task remarkably improves the performance of the model. Our best submission is a BERT-based model that achieved the 7th place out of 20.
- Longformer: The Long-Document Transformer.
- The Legal Argument Reasoning Task in Civil Procedure.
- Language Models are Few-Shot Learners.
- Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4.
- LEGAL-BERT: The Muppets straight out of Law School. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898–2904, Online. Association for Computational Linguistics.
- LeXFfiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15513–15535, Toronto, Canada. Association for Computational Linguistics.
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
- Glannon, J. W. (2019). The Glannon Guide to Civil Procedure. Wolters Kluwer, New York, NY, 4 edition.
- A Free Format Legal Question Answering System. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 107–113, Punta Cana, Dominican Republic. Association for Computational Linguistics.
- OpenAI (2023a). GPT-3.5 Model Documentation. https://platform.openai.com/docs/models/gpt-3-5-turbo. Accessed: 2024-02-05.
- OpenAI (2023b). GPT-4 Model Documentation. https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo. Accessed: 2024-02-05.
- Boosting methods for multi-class imbalanced data classification: an experimental review. Journal of Big Data, 7:1–47.
- A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT.
- When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset.