It Cannot Be Right If It Was Written by AI: On Lawyers' Preferences of Documents Perceived as Authored by an LLM vs a Human (2407.06798v2)

Abstract: LLMs enable a future in which certain types of legal documents may be generated automatically. This has a great potential to streamline legal processes, lower the cost of legal services, and dramatically increase access to justice. While many researchers focus on proposing and evaluating LLM-based applications supporting tasks in the legal domain, there is a notable lack of investigations into how legal professionals perceive content if they believe an LLM has generated it. Yet, this is a critical point as over-reliance or unfounded scepticism may influence whether such documents bring about appropriate legal consequences. This study is the necessary analysis of the ongoing transition towards mature generative AI systems. Specifically, we examined whether the perception of legal documents' by lawyers and law students (n=75) varies based on their assumed origin (human-crafted vs AI-generated). The participants evaluated the documents, focusing on their correctness and language quality. Our analysis revealed a clear preference for documents perceived as crafted by a human over those believed to be generated by AI. At the same time, most participants expect the future in which documents will be generated automatically. These findings could be leveraged by legal practitioners, policymakers, and legislators to implement and adopt legal document generation technology responsibly and to fuel the necessary discussions on how legal processes should be updated to reflect recent technological developments.

The paper "It Cannot Be Right If It Was Written by AI: On Lawyers' Preferences of Documents Perceived as Authored by an LLM vs a Human" explores the perception of legal professionals towards documents that they believe are either human-crafted or AI-generated. The context for this paper is the growing potential for LLMs to automate the generation of legal documents. Such automation could streamline legal processes, reduce costs, and enhance access to justice.

The paper specifically addresses a gap in research by focusing on how legal professionals perceive documents when they suspect these have been generated by AI. This perception is critical since it could influence legal outcomes, with professionals possibly exhibiting over-reliance on AI or unjust skepticism towards it.

Methodology:

• The paper evaluated 75 lawyers who were asked to assess legal documents for correctness and language quality.
• Documents were presented with indications of their supposed origin, whether human or AI.

Findings:

• There was a notable preference among the legal professionals for documents perceived as created by humans.
• Despite this preference, the participants also anticipated a future where the generation of legal documents would be automated by AI.

Implications:

• These insights could inform legal practitioners, policymakers, and legislators on responsibly adopting AI for legal documentation.
• The findings highlight the need for discussions on updating legal processes in light of emerging AI technologies.

The paper thus contributes to understanding the acceptance and potential integration challenges of AI technologies in the legal domain, suggesting that while there is a preference for human-attributed work, there is also an openness to future AI integration.