Papers
Topics
Authors
Recent
Search
2000 character limit reached

Justifiable Artificial Intelligence: Engineering Large Language Models for Legal Applications

Published 27 Nov 2023 in cs.CL, cs.HC, and cs.IR | (2311.15716v1)

Abstract: In this work, I discuss how LLMs can be applied in the legal domain, circumventing their current drawbacks. Despite their large success and acceptance, their lack of explainability hinders legal experts to trust in their output, and this happens rightfully so. However, in this paper, I argue in favor of a new view, Justifiable Artificial Intelligence, instead of focusing on Explainable Artificial Intelligence. I discuss in this paper how gaining evidence for and against a LLM's output may make their generated texts more trustworthy - or hold them accountable for misinformation.

Authors (1)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (70)
  1. Copyright Violations and Large Language Models. arXiv preprint arXiv:231013771. 2023.
  2. Sag M. Copyright safety for generative AI. Forthcoming in the Houston Law Review. 2023.
  3. Attention is All you Need. In: Guyon I, von Luxburg U, Bengio S, Wallach HM, Fergus R, Vishwanathan SVN, et al., editors. Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA; 2017. p. 5998-6008. Available from: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
  4. How to Protect Copyright Data in Optimization of Large Language Models? arXiv preprint arXiv:230812247. 2023.
  5. Better language models and their implications. OpenAI blog. 2019;1(2).
  6. OpenAI. GPT-4 Technical Report. CoRR. 2023;abs/2303.08774. Available from: https://doi.org/10.48550/arXiv.2303.08774.
  7. LaMDA: Language Models for Dialog Applications. CoRR. 2022;abs/2201.08239. Available from: https://arxiv.org/abs/2201.08239.
  8. LLaMA: Open and Efficient Foundation Language Models. CoRR. 2023;abs/2302.13971. Available from: https://doi.org/10.48550/arXiv.2302.13971.
  9. Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model. CoRR. 2022;abs/2201.11990. Available from: https://arxiv.org/abs/2201.11990.
  10. PaLM 2 Technical Report. CoRR. 2023;abs/2305.10403. Available from: https://doi.org/10.48550/arXiv.2305.10403.
  11. OPT: Open Pre-trained Transformer Language Models. CoRR. 2022;abs/2205.01068. Available from: https://doi.org/10.48550/arXiv.2205.01068.
  12. Improving alignment of dialogue agents via targeted human judgements. CoRR. 2022;abs/2209.14375. Available from: https://doi.org/10.48550/arXiv.2209.14375.
  13. Language Models are Few-Shot Learners. In: Larochelle H, Ranzato M, Hadsell R, Balcan M, Lin H, editors. Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual; 2020. Available from: https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html.
  14. Fine-Tuning Language Models from Human Preferences. CoRR. 2019;abs/1909.08593. Available from: http://arxiv.org/abs/1909.08593.
  15. Deep Reinforcement Learning from Human Preferences. In: Guyon I, von Luxburg U, Bengio S, Wallach HM, Fergus R, Vishwanathan SVN, et al., editors. Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA; 2017. p. 4299-307. Available from: https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html.
  16. Training language models to follow instructions with human feedback. In: NeurIPS; 2022. Available from: http://papers.nips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html.
  17. Performance of ChatGPT on USMLE: Unlocking the Potential of Large Language Models for AI-Assisted Medical Education. CoRR. 2023;abs/2307.00112. Available from: https://doi.org/10.48550/arXiv.2307.00112.
  18. Trialling a large language model (ChatGPT) in general practice with the Applied Knowledge Test: observational study demonstrating opportunities and limitations in primary care. JMIR Medical Education. 2023;9(1):e46599.
  19. Capabilities of GPT-4 on Medical Challenge Problems. CoRR. 2023;abs/2303.13375. Available from: https://doi.org/10.48550/arXiv.2303.13375.
  20. Are Emergent Abilities of Large Language Models a Mirage? CoRR. 2023;abs/2304.15004. Available from: https://doi.org/10.48550/arXiv.2304.15004.
  21. Emergent Abilities of Large Language Models. Trans Mach Learn Res. 2022;2022. Available from: https://openreview.net/forum?id=yzkSU5zdwD.
  22. Large language models in medicine. Nature medicine. 2023;29(8):1930-40.
  23. Day T. A preliminary investigation of fake peer-reviewed citations and references generated by ChatGPT. The Professional Geographer. 2023:1-4.
  24. Giray L. ChatGPT References Unveiled: Distinguishing the Reliable from the Fake. Internet Reference Services Quarterly. 2023:1-10.
  25. Persistent Anti-Muslim Bias in Large Language Models. In: Fourcade M, Kuipers B, Lazar S, Mulligan DK, editors. AIES ’21: AAAI/ACM Conference on AI, Ethics, and Society, Virtual Event, USA, May 19-21, 2021. ACM; 2021. p. 298-306. Available from: https://doi.org/10.1145/3461702.3462624.
  26. Risks and Benefits of Large Language Models for the Environment. Environmental Science & Technology. 2023;57(9):3464-6. PMID: 36821477. Available from: https://doi.org/10.1021/acs.est.3c01106.
  27. BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage. CoRR. 2022;abs/2208.03188. Available from: https://doi.org/10.48550/arXiv.2208.03188.
  28. Mehdi Y. Reinventing search with a new AI-powered Microsoft Bing and Edge, your copilot for the web; 2023. Accessed on: 02 November 2023. Available from: https://tinyurl.com/yckmtuss.
  29. Nay JJ. Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humans. CoRR. 2022;abs/2209.13020. Available from: https://doi.org/10.48550/arXiv.2209.13020.
  30. The Capacity for Moral Self-Correction in Large Language Models. CoRR. 2023;abs/2302.07459. Available from: https://doi.org/10.48550/arXiv.2302.07459.
  31. Right to be Forgotten in the Era of Large Language Models: Implications, Challenges, and Solutions. CoRR. 2023;abs/2307.03941. Available from: https://doi.org/10.48550/arXiv.2307.03941.
  32. Bowman SR. Eight Things to Know about Large Language Models. CoRR. 2023;abs/2304.00612. Available from: https://doi.org/10.48550/arXiv.2304.00612.
  33. End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models. SIGIR ’23. New York, NY, USA: Association for Computing Machinery; 2023. p. 2733–2743. Available from: https://doi.org/10.1145/3539618.3591879.
  34. Interpreting black-box models: a review on explainable artificial intelligence. Cognitive Computation. 2023:1-30.
  35. Explainable Artificial Intelligence: Evaluating the Objective and Subjective Impacts of xAI on Human-Agent Interaction. Int J Hum Comput Interact. 2023;39(7):1390-404. Available from: https://doi.org/10.1080/10447318.2022.2101698.
  36. The future of human-centric eXplainable Artificial Intelligence (XAI) is not post-hoc explanations. CoRR. 2023;abs/2307.00364. Available from: https://doi.org/10.48550/arXiv.2307.00364.
  37. LMExplainer: a Knowledge-Enhanced Explainer for Language Models. CoRR. 2023;abs/2303.16537. Available from: https://doi.org/10.48550/arXiv.2303.16537.
  38. Greco CM, Tagarelli A. Bringing order into the realm of Transformer-based language models for artificial intelligence and law. CoRR. 2023;abs/2308.05502. Available from: https://doi.org/10.48550/arXiv.2308.05502.
  39. Can GPT-3 Perform Statutory Reasoning? In: Grabmair M, Andrade F, Novais P, editors. Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, ICAIL 2023, Braga, Portugal, June 19-23, 2023. ACM; 2023. p. 22-31. Available from: https://doi.org/10.1145/3594536.3595163.
  40. Emergent Analogical Reasoning in Large Language Models. CoRR. 2022;abs/2212.09196. Available from: https://doi.org/10.48550/arXiv.2212.09196.
  41. Hodel D, West J. Response: Emergent analogical reasoning in large language models. CoRR. 2023;abs/2308.16118. Available from: https://doi.org/10.48550/arXiv.2308.16118.
  42. LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. CoRR. 2023;abs/2308.11462. Available from: https://doi.org/10.48550/arXiv.2308.11462.
  43. Legal Prompting: Teaching a Language Model to Think Like a Lawyer. CoRR. 2022;abs/2212.01326. Available from: https://doi.org/10.48550/arXiv.2212.01326.
  44. Large Language Models are Zero-Shot Reasoners. In: NeurIPS; 2022. Available from: http://papers.nips.cc/paper_files/paper/2022/hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html.
  45. Can Large Language Models Truly Understand Prompts? A Case Study with Negated Prompts. In: Albalak A, Zhou C, Raffel C, Ramachandran D, Ruder S, Ma X, editors. Transfer Learning for Natural Language Processing Workshop, 03 December 2022, New Orleans, Louisiana, USA. vol. 203 of Proceedings of Machine Learning Research. PMLR; 2022. p. 52-62. Available from: https://proceedings.mlr.press/v203/jang23a.html.
  46. Jiang C, Yang X. Legal Syllogism Prompting: Teaching Large Language Models for Legal Judgment Prediction. In: Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law. ICAIL ’23. New York, NY, USA: Association for Computing Machinery; 2023. p. 417–421. Available from: https://doi.org/10.1145/3594536.3595170.
  47. Exploring the Effectiveness of Prompt Engineering for Legal Reasoning Tasks. In: Rogers A, Boyd-Graber JL, Okazaki N, editors. Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023. Association for Computational Linguistics; 2023. p. 13582-96. Available from: https://doi.org/10.18653/v1/2023.findings-acl.858.
  48. Chain Of Reference prompting helps LLM to think like a lawyer. In: Generative AI+ Law Workshop; 2023. .
  49. Prompt-based Effective Input Reformulation for Legal Case Retrieval. CoRR. 2023;abs/2309.02962. Available from: https://doi.org/10.48550/arXiv.2309.02962.
  50. Billions of Parameters Are Worth More Than In-domain Training Data: A case study in the Legal Case Entailment Task. CoRR. 2022;abs/2205.15172. Available from: https://doi.org/10.48550/arXiv.2205.15172.
  51. IITDLI: Legal Case Retrieval Based on Lexical Models. In: Workshop of the Tenth Competition on Legal Information Extraction/Entailment (COLIEE’2023) in the 19th International Conference on Artificial Intelligence and Law (ICAIL); 2023. .
  52. Summary of the Competition on Legal Information, Extraction/Entailment (COLIEE) 2023. In: Grabmair M, Andrade F, Novais P, editors. Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, ICAIL 2023, Braga, Portugal, June 19-23, 2023. ACM; 2023. p. 472-80. Available from: https://doi.org/10.1145/3594536.3595176.
  53. Nay JJ. Large Language Models as Fiduciaries: A Case Study Toward Robustly Communicating With Artificial Intelligence Through Legal Standards. CoRR. 2023;abs/2301.10095. Available from: https://doi.org/10.48550/arXiv.2301.10095.
  54. Legal Prompt Engineering for Multilingual Legal Judgement Prediction. CoRR. 2022;abs/2212.02199. Available from: https://doi.org/10.48550/arXiv.2212.02199.
  55. How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization? In: Conrad JG, Jr DWL, Baron JR, Henseler H, Bhattacharya P, Nielsen A, et al., editors. Proceedings of the Third International Workshop on Artificial Intelligence and Intelligent Assistance for Legal Professionals in the Digital Workplace (LegalAIIA 2023) co-located with the 19th International Conference on Artificial Intelligence and Law (ICAIL 2023), Braga, Portugal, June 19, 2023. vol. 3423 of CEUR Workshop Proceedings. CEUR-WS.org; 2023. p. 8-19. Available from: https://ceur-ws.org/Vol-3423/paper2.pdf.
  56. Iu KY, Wong VMY. ChatGPT by OpenAI: The End of Litigation Lawyers? Available at SSRN. 2023.
  57. Pettinato Oltz T. ChatGPT, Professor of Law. Available at SSRN. 2023. Available from: https://ssrn.com/abstract=4347630.
  58. Hargreaves S. ‘Words Are Flowing Out Like Endless Rain Into a Paper Cup’: ChatGPT & Law School Assessments. The Chinese University of Hong Kong Faculty of Law Research Paper. 2023;(2023-03).
  59. II MJB, Katz DM. GPT Takes the Bar Exam. CoRR. 2022;abs/2212.14402. Available from: https://doi.org/10.48550/arXiv.2212.14402.
  60. Chatgpt goes to law school. Available at SSRN. 2023.
  61. Gpt-4 passes the bar exam. Available at SSRN 4389233. 2023.
  62. ChatLaw: Open-Source Legal Large Language Model with Integrated External Knowledge Bases. CoRR. 2023;abs/2306.16092. Available from: https://doi.org/10.48550/arXiv.2306.16092.
  63. DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services. CoRR. 2023;abs/2309.11325. Available from: https://doi.org/10.48550/arXiv.2309.11325.
  64. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models. CoRR. 2023;abs/2309.01219. Available from: https://doi.org/10.48550/arXiv.2309.01219.
  65. Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models. CoRR. 2023;abs/2309.17050. Available from: https://doi.org/10.48550/arXiv.2309.17050.
  66. FacTeR-Check: Semi-automated fact-checking through semantic similarity and natural language inference. Knowl Based Syst. 2022;251:109265. Available from: https://doi.org/10.1016/j.knosys.2022.109265.
  67. Trokhymovych M, Sáez-Trumper D. WikiCheck: An End-to-end Open Source Automatic Fact-Checking API based on Wikipedia. In: Demartini G, Zuccon G, Culpepper JS, Huang Z, Tong H, editors. CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021. ACM; 2021. p. 4155-64. Available from: https://doi.org/10.1145/3459637.3481961.
  68. Leslie D. Understanding artificial intelligence ethics and safety. CoRR. 2019;abs/1906.05684. Available from: http://arxiv.org/abs/1906.05684.
  69. Available from: https://srinstitute.utoronto.ca/news/hadfield-justifiable-ai.
  70. Jones S. Explainable AI is dead, long live justifiable decisions. Medium; 2023. Available from: https://tinyurl.com/mrytfuzs.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up