2000 character limit reached
Evaluating AI for Law: Bridging the Gap with Open-Source Solutions (2404.12349v1)
Published 18 Apr 2024 in cs.AI and cs.HC
Abstract: This study evaluates the performance of general-purpose AI, like ChatGPT, in legal question-answering tasks, highlighting significant risks to legal professionals and clients. It suggests leveraging foundational models enhanced by domain-specific knowledge to overcome these issues. The paper advocates for creating open-source legal AI systems to improve accuracy, transparency, and narrative diversity, addressing general AI's shortcomings in legal contexts.
- M. Shur-Ofry, “Multiplicity as an AI Governance Principle.” Rochester, NY, May 10, 2023. doi: 10.2139/ssrn.4444354.
- D. M. Katz, M. J. Bommarito, S. Gao, and P. Arredondo, “GPT-4 Passes the Bar Exam.” Rochester, NY, Mar. 15, 2023. doi: 10.2139/ssrn.4389233.
- Jed Stiglitz, “Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement,” Working Paper, 2023.
- M. C. Cohen, S. Dahan, W. Khern-Am-Nuai, H. Shimao, and J. Touboul, “The use of AI in legal systems: determining independent contractor vs. employee status,” Artificial intelligence and law, pp. 1–30, 2023.
- J. Kleinberg, J. Ludwig, S. Mullainathan, and C. R. Sunstein, “Discrimination in the Age of Algorithms,” Journal of Legal Analysis, vol. 10, pp. 113–174, Dec. 2018, doi: 10.1093/jla/laz001.
- Y. Chen, M. Andiappan, T. Jenkin, and A. Ovchinnikov, “A Manager and an AI Walk into a Bar: Does ChatGPT Make Biased Decisions Like We Do?” Rochester, NY, May 20, 2023. doi: 10.2139/ssrn.4380365.
- Medianik, Katherine. ”Artificially intelligent lawyers: updating the model rules of professional conduct in accordance with the new technological era.” Cardozo L. Rev. 39 (2017): 1497.
- Martínez, Eric. ”Re-Evaluating GPT-4’s Bar Exam Performance.” Available at SSRN 4441311 (2023).
- Geex, Law. ”Comparing the performance of artificial intelligence to human lawyers in the review of standard business contracts. Law Geex.” (2018).
- Rudin, Cynthia. ”Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.” Nature machine intelligence 1.5 (2019): 206-215.
- Samuel Dahan, Jonathan Touboul, Jason Lam, and Dan Sfedj, “Predicting Employment Notice Period with Machine Learning: Promises and Limitations,” McGill Law Journal, 2020.
- C. Markou and S. Deakin, “Ex Machina Lex: Exploring the Limits of Legal Computability.” Rochester, NY, Jun. 21, 2019. doi: 10.2139/ssrn.3407856.
- D. Ha and J. Schmidhuber “World Models.” Advances in Neural Information Processing Systems 31 (NeurIPS 2018). https://arxiv.org/abs/1803.10122
- R. Bhambhoria, S. Dahan, and X. Zhu, “Investigating the State-of-the-Art Performance and Explainability of Legal Judgment Prediction.,” in Canadian Conference on AI, 2021.
- C. F. Luo, R. Bhambhoria, S. Dahan, and X. Zhu, “Evaluating Explanation Correctness in Legal Decision Making,” in Proceedings of the Canadian Conference on Artificial Intelligence (5 2022). https://doi. org/10.21428/594757db. 8718dc8b, 2022.
- C. F. Luo, R. Bhambhoria, S. Dahan, and X. Zhu, “Prototype-Based Interpretability for Legal Citation Prediction.” arXiv, May 25, 2023. doi: 10.48550/arXiv.2305.16490.
- J. Hilton, R. Nakano, S. Balaji, and J. Schulman, “WebGPT: Improving the factual accuracy of language models through web browsing,” OpenAI Blog, December, vol. 16, 2021.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.