PropertyGPT: LLM-driven Formal Verification of Smart Contracts through Retrieval-Augmented Property Generation (2405.02580v2)

Abstract: With recent advances in LLMs, this paper explores the potential of leveraging state-of-the-art LLMs,such as GPT-4, to transfer existing human-written properties (e.g.,those from Certora auditing reports) and automatically generate customized properties for unknown code. To this end, we embed existing properties into a vector database and retrieve a reference property for LLM-based in-context learning to generate a new property for a given code. While this basic process is relatively straightforward, ensuring that the generated properties are (i) compilable, (ii) appropriate, and (iii) verifiable presents challenges. To address (i), we use the compilation and static analysis feedback as an external oracle to guide LLMs in iteratively revising the generated properties. For (ii), we consider multiple dimensions of similarity to rank the properties and employ a weighted algorithm to identify the top-K properties as the final result. For (iii), we design a dedicated prover to formally verify the correctness of the generated properties. We have implemented these strategies into a novel LLM-based property generation tool called PropertyGPT. Our experiments show that PropertyGPT can generate comprehensive and high-quality properties, achieving an 80% recall compared to the ground truth. It successfully detected 26 CVEs/attack incidents out of 37 tested and also uncovered 12 zero-day vulnerabilities, leading to $8,256 in bug bounty rewards.

• The paper introduces a novel system that automates the generation of formal verification properties for smart contracts.
• It employs a multi-step process combining vector-based retrieval, in-context LLM property generation, compiler feedback, and ranking.
• The approach achieves an 80% recall versus human benchmarks and effectively detects both known and zero-day vulnerabilities.

Understanding PropertyGPT: Enhancing Smart Contract Verification with LLMs

Overview of PropertyGPT

PropertyGPT introduces a novel approach to enhancing the security and reliability of smart contracts through automated property generation facilitated by the latest advancements in LLMs. This system primarily addresses the challenge of automatically generating diverse formal properties, such as invariants and pre-/post-conditions, which are essential for verifying the correctness of smart contracts.

How PropertyGPT Works

The core functionality of PropertyGPT can be broken down into a clear, multi-step process:

1. Property Embedding and Retrieval: Existing properties are encoded into a vector database, allowing swift retrieval of similar properties when given new, unknown code snippets.
2. LLM-based Property Generation: With retrieved properties as references, PropertyGPT employs LLMs to generate new properties tailored to the input code through in-context learning.
3. Iterative Refinement: Leveraging external compiler feedback, the system iteratively refines any syntactical errors in the generated properties to ensure they are compilable and logically sound.
4. Ranking and Verification: A weighted mechanism is utilized to rank the generated properties. The top-ranked properties undergo formal verification to ensure their correctness concerning the smart contract code.

The entire system not only automates the generation of valuable properties but also ensures these properties are immediately applicable for practical verification use.

Results and Impact on the Smart Contract Ecosystem

PropertyGPT has displayed robust performance metrics in terms of property generation and vulnerability detection in smart contracts. It was able to achieve an 80% recall in comparison with human-generated ground truths, and notably, it detected several historical Common Vulnerabilities and Exposures (CVEs) and zero-day vulnerabilities, showcasing its practical relevance in real-world scenarios.

The detection capabilities of PropertyGPT extend to a broad range of vulnerability types thanks to the in-depth analysis and understanding capabilities of LLMs, coupled with a rigorous formal verification process driven by the generated properties.

Future Directions and Considerations

While PropertyGPT marks a significant step forward, the landscape of smart contract development and verification is continuously evolving. Future enhancements might integrate a deeper contextual understanding of smart contracts, leveraging supplementary materials like developer documentation and comments to enrich property generation. Additionally, expanding the database of properties with broader types of smart contracts can improve the model's versatility and the ability to handle more complex contract structures and logic.

Conclusion

The introduction of systems like PropertyGPT showcases the potential of combining AI, particularly LLMs, with traditional software verification techniques to innovate and improve smart contract security. As the field advances, such systems could become integral in the development pipelines of blockchain applications, ensuring more reliable and secure digital agreements in an increasingly digital-first world.