Generating Clarification Questions for Disambiguating Contracts (2403.08053v1)
Abstract: Enterprises frequently enter into commercial contracts that can serve as vital sources of project-specific requirements. Contractual clauses are obligatory, and the requirements derived from contracts can detail the downstream implementation activities that non-legal stakeholders, including requirement analysts, engineers, and delivery personnel, need to conduct. However, comprehending contracts is cognitively demanding and error-prone for such stakeholders due to the extensive use of Legalese and the inherent complexity of contract language. Furthermore, contracts often contain ambiguously worded clauses to ensure comprehensive coverage. In contrast, non-legal stakeholders require a detailed and unambiguous comprehension of contractual clauses to craft actionable requirements. In this work, we introduce a novel legal NLP task that involves generating clarification questions for contracts. These questions aim to identify contract ambiguities on a document level, thereby assisting non-legal stakeholders in obtaining the necessary details for eliciting requirements. This task is challenged by three core issues: (1) data availability, (2) the length and unstructured nature of contracts, and (3) the complexity of legal text. To address these issues, we propose ConRAP, a retrieval-augmented prompting framework for generating clarification questions to disambiguate contractual text. Experiments conducted on contracts sourced from the publicly available CUAD dataset show that ConRAP with ChatGPT can detect ambiguities with an F2 score of 0.87. 70% of the generated clarification questions are deemed useful by human evaluators.
- Jaspreet Bhatia and Travis D. Breaux. 2018. Semantic incompleteness in privacy policy goals. 2018 IEEE 26th International Requirements Engineering Conference (RE), pages 159–169.
- Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality.
- Free Dolly: Introducing the world’s first truly open instruction-tuned LLM.
- Brian A. Garner. 2014. Black’s Law Dictionary, chapter 9th Edition. St. Paul, MN :Thomson Reuters.
- Cuad: An expert-annotated nlp dataset for legal contract review.
- Lora: Low-Rank Adaptation of Large Language Models.
- A transformer-based approach for abstractive summarization of requirements from obligations in software engineering contracts. In 2023 IEEE 31st International Requirements Engineering Conference (RE), pages 169–179.
- Large language models are zero-shot reasoners.
- ViCLOUD: Measuring Vagueness in Cloud Service Privacy Policies and Terms of Services. In IEEE International Conference on Cloud Computing (CLOUD), 2020. IEEE.
- Clam: Selective clarification for ambiguous questions with generative language models.
- Ranking clarification questions via natural language inference. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management. ACM.
- Logan Lebanoff and Fei Liu. 2018. Automatic detection of vague words and sentences in privacy policies. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3508–3517, Brussels, Belgium. Association for Computational Linguistics.
- Shuangling Li. 2017. A corpus-based study of vague language in legislative texts: Strategic use of vague terms. English for Specific Purposes, 45:98–109.
- Identifying and classifying ambiguity for regulatory requirements. In 2014 IEEE 22nd International Requirements Engineering Conference (RE), pages 83–92.
- A survey on asking clarification questions datasets in conversational systems. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2698–2716, Toronto, Canada. Association for Computational Linguistics.
- Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2737–2746, Melbourne, Australia. Association for Computational Linguistics.
- Extracting and classifying requirements from software engineering contracts. In 2020 IEEE 28th International Requirements Engineering Conference (RE), pages 147–157.
- Towards mitigating perceived unfairness in contracts from a non-legal stakeholder’s perspective. In Proceedings of the Natural Legal Language Processing Workshop 2023, pages 99–112, Singapore. Association for Computational Linguistics.
- From legal contracts to formal specifications: A systematic literature review. SN Comput. Sci., 3(5).
- Pearl: Prompting large language models to plan and execute actions over long documents.
- Stanford Alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/stanford_alpaca.
- Zero-shot clarifying question generation for conversational search. In Proceedings of the ACM Web Conference 2023, WWW ’23, page 3288–3298, New York, NY, USA. Association for Computing Machinery.
- Chain-of-thought prompting elicits reasoning in large language models.
- ConReader: Exploring implicit relations in contracts for contract clause extraction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2581–2594, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Zhiling Zhang and Kenny Zhu. 2021. Diverse and specific clarification question generation with keywords. In Proceedings of the Web Conference 2021. ACM.
- Dan Hendrycks and Collin Burns and Anya Chen and Spencer Ball. 2021. Contract Atticus Dataset (CUAD). NeurIPS 2021. PID https://www.atticusprojectai.org/cuad.