Broken by Default: A Formal Verification Study of Security Vulnerabilities in AI-Generated Code

Abstract: AI coding assistants are now used to generate production code in security-sensitive domains, yet the exploitability of their outputs remains unquantified. We address this gap with Broken by Default: a formal verification study of 3,500 code artifacts generated by seven frontier LLMs across 500 security-critical prompts (five CWE categories, 100 prompts each). Each artifact is subjected to the Z3 SMT solver via the COBALT analysis pipeline, producing mathematical satisfiability witnesses rather than pattern-based heuristics. Across all models, 55.8% of artifacts contain at least one COBALT-identified vulnerability; of these, 1,055 are formally proven via Z3 satisfiability witnesses. GPT-4o leads at 62.4% (grade F); Gemini 2.5 Flash performs best at 48.4% (grade D). No model achieves a grade better than D. Six of seven representative findings are confirmed with runtime crashes under GCC AddressSanitizer. Three auxiliary experiments show: (1) explicit security instructions reduce the mean rate by only 4 points; (2) six industry tools combined miss 97.8% of Z3-proven findings; and (3) models identify their own vulnerable outputs 78.7% of the time in review mode yet generate them at 55.8% by default.

• The paper demonstrates that seven LLMs generate exploitable code with a mean vulnerability rate of 55.8%, as confirmed by Z3 SMT-based proofs.
• The authors deploy the COBALT engine to structurally extract vulnerability patterns and encode them as SMT constraints, linking formal proofs with real-world exploits.
• The study reveals that static analysis tools detect only 7.6% of vulnerabilities, highlighting the need for robust formal verification in AI-generated code.

Formal Verification Analysis of AI-Generated Code Security Vulnerabilities

Introduction

"Broken by Default: A Formal Verification Study of Security Vulnerabilities in AI-Generated Code" (2604.05292) delivers a comprehensive empirical evaluation of code security in outputs from seven frontier LLMs. This study leverages the rigor of formal verification via the Z3 SMT solver to establish concrete exploitability of vulnerabilities, surpassing prior pattern-based or heuristic approaches. The work covers 3,500 code artifacts generated from 500 security-sensitive prompts, mapped to critical Common Weakness Enumeration (CWE) classes, and quantifies model susceptibility, detection tool efficacy, model self-awareness, and the ultimate limits of current mitigation strategies.

Methodological Rigor and Experiment Design

The analysis pipeline hinges on the COBALT engine, a multi-stage static tool which (1) identifies candidate vulnerability patterns using structural extraction, (2) encodes those patterns as SMT constraints for Z3, and (3) extracts concrete witness inputs for all Z3-proven SAT cases, yielding mathematical proof of exploitability. Experiments encompass coverage across five high-impact CWE classes: buffer (CWE-131/190), integer (CWE-190/195), authentication (CWE-916), cryptography (CWE-327/330), and input handling (CWE-89/22/78). The prompt set is constructed to elicit practical, non-adversarial code representations relevant to real-world developer workflows.

Seven LLMs—including GPT-4o, Claude Haiku 4.5, Gemini 2.5 Flash, and Llama 3.3 70B—were evaluated under strictly controlled API configurations, with all outputs generated at deterministic zero-temperature settings to maximize reproducibility and analysis tractability.

Quantitative Findings: Code Generation Vulnerability Landscape

Aggregate Vulnerability Rates

Across the 3,500 samples, the mean vulnerability rate is 55.8%. GPT-4o produces vulnerable code at 62.4% (grade F), while Gemini 2.5 Flash achieves the lowest (yet unacceptably high) rate at 48.4% (grade D). No model attains a vulnerability rate better than grade D. Severity analysis indicates dominant proportions of CRITICAL vulnerabilities, with memory allocation (67%) and integer arithmetic (87%) categories exhibiting the highest model-agnostic failure rates.

Notably, 1,055 vulnerabilities are not merely pattern matches but are formally proved exploitable via concrete Z3 witnesses. Runtime validation using GCC AddressSanitizer confirmed exploitable memory faults in 6 of 7 real-world code harnesses, empirically linking formal findings to practical exploit outcomes.

CWE Distribution and Dominant Failure Modes

Memory management and integer vulnerabilities are the modal failure modes, with all LLMs consistently generating unsafe malloc size calculation patterns:

int* buf = malloc(n * sizeof(int)); // unsafe: no overflow guard

Safe handling, requiring bounds checks as per:

if (n > SIZE_MAX / sizeof(int)) return NULL; int* buf = malloc(n * sizeof(int)); // safe

is exceedingly rare, indicating the presence of strong negative inductive biases from internet-scale code corpora and insufficient correction via post-pretraining alignment.

Evaluation of Mitigation: Security Prompting and Static Analysis Tools

Prompt Conditioning Fails to Mitigate Vulnerabilities

Adding explicit security instructions to system prompts provides, on average, a paltry 4% reduction in vulnerability rates (from 64.8% to 60.8%) on a subset of 50 prompts. Four of five models remain at grade F, and some models (e.g., Llama 3.3 70B) perform worse with security-targeted instructions. Category-level improvement is marginal, especially for memory and integer classes. This supports the hypothesis that the probability of unsafe code emission is anchored in the learned prior and is not substantially modifiable by instruction tweaking alone.

Structural Limitations of Industry Static Analysis

Six widely used commercial and open-source static analysis tools (Semgrep, Bandit, Cppcheck, Clang SA, FlawFinder, CodeQL) exhibit catastrophic blind spots: collectively detecting only 7.6% of the vulnerabilities, and missing 97.8% of formally proven (Z3 SAT) bugs. No tool—regardless of configuration—found a single instance of integer overflow in allocation arithmetic, and CodeQL, despite advanced dataflow reasoning, failed to detect any confirmed vulnerabilities (0/90). This exposes a foundational limitation: pattern and path-sensitive analyzers cannot reason about symbolic arithmetic over unconstrained attacker-controlled inputs, a gap only SMT-based approaches can bridge at scale.

Generation–Review Asymmetry and Model Self-Knowledge

A notable experimental result is the revealed generation–review asymmetry. When models were tasked to review code they had themselves generated (and which had been proven vulnerable via Z3), they successfully identified vulnerabilities in 78.7% of cases. However, those vulnerabilities were present in 55.8% of their direct generations by default. This demonstrates that models encode latent knowledge of these vulnerability patterns but fail to activate corresponding safety constraints during generation. The finding indicates that RLHF or filter-based instruction fine-tuning are insufficient for bridging the gap between security assessment and reliable code emission.

Implications and Recommendations

This study's findings have direct implications for AI-powered software engineering pipelines, especially in security- and safety-critical contexts. AI-generated code, especially in C/C++, must be assumed insecure by default and subject to the same or greater scrutiny as unaudited legacy code. Security prompt prefixes are ineffective as meaningful mitigations. Industry-standard static analyzers are grossly inadequate in detecting the most prevalent, dangerous bug classes emitted by LLMs. Integrating formal SMT-based verification or comprehensive manual audit into deployment pipelines is mandatory, particularly for dynamic memory and boundary-sensitive logic.

Limitations and Future Directions

While the prompt set (500 prompts) is broad and the model selection state-of-the-art, results are constrained to five major CWE categories and deterministic decoding at temperature zero. Further work extending secure prompt ablations to the full prompt suite, covering additional programming languages (Go, Rust, JavaScript), and evaluating model fine-tuning specifically for memory arithmetic and integer overflow mitigation is necessary. Multiturn and retrieval-augmented generation, as well as adversarial prompt slicing, represent unexplored dimensions.

Conclusion

This formal verification study (2604.05292) systematically demonstrates that current-leading LLMs reliably generate code with provably exploitable vulnerabilities across a majority of reasonable, real-world prompt scenarios—even under explicit security prompting. Standard static analysis tools are inadequate, detecting only the most superficial patterns, and fail to flag structurally hard vulnerabilities confirmed by mathematical proofs and runtime exploits. LLMs can recognize vulnerabilities in review but fail to prevent them during generation, underscoring the limits of present-day alignment approaches. Formal SMT-based analysis with tools like Z3 stands as the only scalable solution for ground-truth vulnerability certification in LLM-generated code. Rigorous security auditing and methodological advances are essential as AI coding assistants are further integrated into production pipelines for systems code.