Formal specification synthesis by LLMs

Determine how to reliably synthesize high-quality formal program specifications with large language models, addressing the unresolved challenge that current LLM-based specification synthesis remains difficult.

Background

The paper compares LLM-generated Rocq specifications with human-written specifications using formal equivalence proofs. Among 29 problems for which both a human and an LLM specification passed the paper’s test-based validation, only 9 were provably equivalent in both directions; several additional cases were judged semantically equivalent despite proof failures attributed to prover limitations. The limited overlap between human and LLM successes indicates that current models do not consistently reproduce the intended formal behavior across the benchmark.

The authors therefore characterize LLM-based formal specification synthesis as an unresolved research challenge. This problem concerns improving the reliability and semantic precision of generated specifications, rather than merely producing syntactically valid Rocq code or specifications that pass a finite test suite.