KBSpec: LLM-driven Formal Specification Generation with Evolving Domain Knowledge Base

Abstract: Automated formal specification generation is a key step towards program understanding and formal verification. Recently, due to the success of LLMs in code generation, researchers have made early attempts to adopt LLMs for generating formal specifications. However, the lack of formal specification language corpora in the wild often makes LLMs fail to generate syntactically correct and semantically verifiable specifications. To mitigate this gap, we propose KBSpec, which augments LLMs with dual-source knowledge of formal specification language: external knowledge from official documentation, and internal knowledge distilled from verifier feedback on LLM-generated specifications. KBSpec maintains a self-evolving knowledge base that is continuously updated from successful generation and repair trajectories, without any LLM parameter tuning or labeled training data. We evaluate KBSpec on Java Modeling Language (JML) specification generation with three LLM backends, and results show that KBSpec improves verification pass rates by 10-25% over state-of-the-art LLM-based approaches, while producing the largest number of high-completeness specifications.

• The paper demonstrates that augmenting LLMs with an evolving internal knowledge base and external documentation substantially improves formal specification verification rates.
• It introduces a three-stage pipeline incorporating knowledge initialization, self-evolving internal repair data, and iterative knowledge-augmented generation steps.
• Experimental results show significant increases in pass rates and reduced errors across different LLM backends, validating the framework's robust design.

KBSpec: Formal Specification Generation via Knowledge-Augmented LLMs

Motivation and Context

Formal specification authoring is a high-leverage process for enhancing software correctness and maintainability, yet it remains prohibitively labor-intensive because specification languages such as JML are low-resource and complex. Recent advances in code-generating LLMs highlight their utility in program synthesis, but empirical analyses reveal their limitations: they frequently produce specifications with syntactic and semantic errors, primarily due to the absence of formal language domain knowledge in their training corpora. Notably, nearly half of LLM-generated JML specifications contain syntax errors, severely impacting practical verification (2606.21339).

To mitigate this domain-specific knowledge deficit, prior work has explored retrieval augmentation using external documentation. However, official resources are incomplete and often fail to cover edge cases or verification patterns encountered in real-world programs, limiting their efficacy. A core insight of KBSpec is that augmenting LLMs with both external documentation and a growing internal cache of verifier feedback enables robust specification generation.

KBSpec Framework

KBSpec introduces a three-stage pipeline for specification generation, grounded in dual-source knowledge integration:

1. Knowledge Base Initialization: External knowledge is harvested from JML official documents, curated examples, specification exercises, and repair guidelines. Each item is structured as {content, summary, keywords} to support multi-level relevance embedding.
2. Self-Evolving Internal Knowledge: KBSpec deploys a generation-and-repair loop over a corpus of Java programs, orchestrated by a formal verifier. Successful trajectories and repair operations are mined as internal knowledge items. Helpfulness scores are continuously tracked via verifier outcomes, ensuring only impactful knowledge is retained. Crucially, this evolution occurs without LLM parameter tuning, keeping the framework lightweight and model-agnostic.
3. Knowledge-Augmented Generation and Repair: During inference, KBSpec retrieves the top-N relevant knowledge items for each program, leveraging dense BERT-based embeddings on structured fields. The LLM autonomously selects the most helpful items to guide specification generation and repair, with iterative refinement up to three rounds if initial outputs fail verification.

Empirical Evaluation

Experimental Setup

KBSpec is evaluated using the FormalBench dataset, the largest available benchmark for JML specification generation. Three LLM backends are considered: GPT-5.2, GPT-5-mini, and DeepSeek-v3.2. Metrics include verification pass rate (PR), fail rate (FR), and completeness (C), with mutation-based completeness calculations reflecting specification strength.

Numerical Results

• Verification Pass Rate: KBSpec achieves a PR gain of 10–25% over state-of-the-art LLM-only baselines and exceeds prior methods such as SpecGen across all models.
  • On GPT-5.2, PR increases to 58.94% (SpecGen: 43.35%), representing a 15.59% absolute improvement.
  • For GPT-5-mini, KBSpec more than doubles SpecGen's PR.
  • DeepSeek-v3.2 shows smaller, yet substantial gains.
• Completeness: While average completeness decreases for some LLMs, KBSpec consistently yields the largest count of medium-to-high completeness specifications. Among mutually verified programs, KBSpec achieves higher or comparable completeness relative to SpecGen.
• Fail Rate and Timeout Reduction: KBSpec substantially lowers fail rates and timeout incidents, especially in lower-capacity models, supporting its robustness across LLM scales.

Ablation Study

Ablation reveals that internal knowledge evolution and iterative repair are the most impactful components, each contributing over 25% to pass rate increases. External documentation boosts foundational performance but is rapidly subsumed by internally learned repair trajectories, confirming the dominance of experiential knowledge in specification generation.

Knowledge Base Dynamics

Analysis of the knowledge base composition and retrieval frequency demonstrates a clear temporal shift: external knowledge is crucial initially, but internal repair-derived items ultimately dominate. Two main repair strategies emerge:

• Removal or substitution of unsupported constructs (e.g., quantifiers such as $\max$ and $\sum$)
• Addition of numeric preconditions to address arithmetic proof obligations (e.g., preventing overflows)

The majority of repairs yield "weaker but meaningful" specifications, favoring pragmatic strengthenings or domain restriction rather than trivial regressions. Semantic preservation occurs in a limited subset, indicating future opportunities for advanced synthesis techniques.

Implications and Future Directions

Practically, KBSpec’s parameter-free knowledge evolution and repair-centric memory enable its deployment across any specification language with an automated verifier. The LLM-agnostic nature ensures that even lower-capacity models can attain significant improvements without intensive retraining or annotation efforts. The framework is extensible to specification languages beyond JML—such as ACSL for C or Dafny—suggesting a generalizable architecture for automated formal specification in program verification pipelines.

Theoretically, KBSpec validates the hypothesis that dual-source knowledge provision, when coupled with explicit verifier-driven memory updates, is essential for overcoming the low-resource bottleneck observed in formal language tasks. The success of repair trajectory mining signals the viability of agentic memory approaches for structured program reasoning and specification synthesis, paralleling trends in LLM-based autonomous agents (Zhang et al., 6 Oct 2025, Suzgun et al., 10 Apr 2025).

Looking forward, further research can pursue:

• Knowledge base densification via hierarchical or graph-structured organization
• Extension to real-world, multi-module software projects
• Integration with symbolic reasoning and static analysis for combined program verification and specification strengthening

Conclusion

KBSpec demonstrates that augmenting LLMs with a self-evolving, verifier-informed knowledge base significantly increases the effectiveness of formal specification generation on low-resource languages like JML. The architecture is both efficient and general, outperforming past approaches in both verification performance and specification completeness. Its knowledge evolution paradigm sets a trajectory for future advances in automated specification synthesis across programming languages and verification regimes.

(2606.21339)