Using Large Language Models for Generating Smart Contracts for Health Insurance from Textual Policies (2407.07019v1)

Abstract: We explore using LLMs to generate application code that automates health insurance processes from text-based policies. We target blockchain-based smart contracts as they offer immutability, verifiability, scalability, and a trustless setting: any number of parties can use the smart contracts, and they need not have previously established trust relationships with each other. Our methodology generates outputs at increasing levels of technical detail: (1) textual summaries, (2) declarative decision logic, and (3) smart contract code with unit tests. We ascertain LLMs are good at the task (1), and the structured output is useful to validate tasks (2) and (3). Declarative languages (task 2) are often used to formalize healthcare policies, but their execution on blockchain is non-trivial. Hence, task (3) attempts to directly automate the process using smart contracts. To assess the LLM output, we propose completeness, soundness, clarity, syntax, and functioning code as metrics. Our evaluation employs three health insurance policies (scenarios) with increasing difficulty from Medicare's official booklet. Our evaluation uses GPT-3.5 Turbo, GPT-3.5 Turbo 16K, GPT-4, GPT-4 Turbo and CodeLLaMA. Our findings confirm that LLMs perform quite well in generating textual summaries. Although outputs from tasks (2)-(3) are useful starting points, they require human oversight: in multiple cases, even "runnable" code will not yield sound results; the popularity of the target language affects the output quality; and more complex scenarios still seem a bridge too far. Nevertheless, our experiments demonstrate the promise of LLMs for translating textual process descriptions into smart contracts.

The paper explores the innovative use of LLMs to generate smart contracts for health insurance by translating textual policies into executable blockchain code. This work focuses on healthcare processes, aiming to leverage the advantages of blockchain technology—such as immutability, verifiability, scalability, and operating in a trustless environment—where parties can utilize smart contracts without pre-established trust.

Methodology

The authors implemented a three-step methodology to generate outputs with increasing technical sophistication:

1. Textual Summaries: LLMs are utilized to produce concise and accurate summaries of health insurance policies.
2. Declarative Decision Logic: The paper explores the conversion of these summaries into declarative languages, which are preferred for formalizing healthcare policies. However, executing this step on a blockchain presents challenges due to the complex nature of healthcare regulations.
3. Smart Contract Code with Unit Tests: The final step involves transforming the structured outputs into smart contract code, complete with unit tests to ensure functionality.

Evaluation and Findings

The paper employs various LLMs, including GPT-3.5 Turbo, GPT-3.5 Turbo 16K, GPT-4, GPT-4 Turbo, and CodeLLaMA, to evaluate the methodologies on three health insurance policies of increasing complexity derived from Medicare's official materials.

• Performance on Textual Summaries: LLMs, particularly the ones evaluated, demonstrate strong performance in creating coherent and concise textual summaries of policy documents.
• Challenges in Decision Logic and Code Generation: While the structured outputs are useful, tasks (2) and (3) require significant human oversight. Some key challenges include:
  • Complexity and Quality: The quality and reliability of the generated outputs can vary greatly, especially for more complex scenarios. Even "runnable" code often fails to produce sound results.
  • Impact of Language Popularity: The popularity and the inherent characteristics of the target programming language influence the performance and correctness of the generated smart contract code.

Metrics for Assessment

The paper proposes a set of evaluation metrics to gauge the output generated by LLMs:

• Completeness: Whether the generated output covers all relevant aspects of the policy.
• Soundness: The logical correctness of the generated outputs.
• Clarity: How understandable and clear the generated descriptions and code are.
• Syntax: Adherence to the syntactical rules of the chosen programming languages.
• Functioning Code: The ability of the generated code to function accurately and effectively when executed.

Conclusion

The paper reaffirms the potential of LLMs in translating textual descriptions of insurance processes into smart contracts, albeit with needed human intervention, particularly for complex scenarios. While LLMs show promise, the research points out that more refined models and techniques are necessary to address existing challenges in smart contract code generation, taking into account the intricacies of healthcare regulations and blockchain execution.