SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code (2506.05692v3)

Abstract: The code generation capabilities of LLMs(LLMs) have emerged as a critical dimension in evaluating their overall performance. However, prior research has largely overlooked the security risks inherent in the generated code. In this work, we introduce SafeGenBench, a benchmark specifically designed to assess the security of LLM-generated code. The dataset encompasses a wide range of common software development scenarios and vulnerability types. Building upon this benchmark, we develop an automatic evaluation framework that leverages both static application security testing(SAST) and LLM-based judging to assess the presence of security vulnerabilities in model-generated code. Through the empirical evaluation of state-of-the-art LLMs on SafeGenBench, we reveal notable deficiencies in their ability to produce vulnerability-free code. Our findings highlight pressing challenges and offer actionable insights for future advancements in the secure code generation performance of LLMs. The data and code will be released soon.

SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code

The prevalence of LLMs in the field of software development has accelerated code generation capabilities significantly. As LLMs generate code snippets, concerns about the security of such automated code production have emerged, particularly in contexts where these models are predisposed to introducing vulnerabilities in the resultant code. The paper in question introduces SafeGenBench as a benchmark designed specifically to evaluate the security robustness of code produced by state-of-the-art LLMs, addressing an underexplored dimension in the evaluation of these models.

Overview of SafeGenBench and Its Methodology

SafeGenBench serves as a comprehensive benchmark that scrutinizes the security susceptibility of LLM-generated code against widespread vulnerability types recognized by standards such as the OWASP Top-10 and Common Weakness Enumeration (CWE). The framework is innovative in its dual-judge evaluation approach, which operates by employing Static Application Security Testing (SAST) tools in conjunction with LLM-based judgment mechanisms to assess vulnerabilities thoroughly.

The dataset supporting SafeGenBench is meticulously crafted through a multi-stage process: vulnerability extraction, test question generation, and expert-led annotation. Reflecting a spectrum of common vulnerabilities across eight main categories, the dataset encompasses 558 test cases spanning languages from Python to Swift. This ensures a well-rounded assessment of LLM's capabilities in producing secure code across various languages and scenarios.

The benchmark evaluates LLMs' outputs by scoring them on binary metrics, where a score of zero indicates the presence of vulnerabilities and a score of one denotes secure, vulnerability-free code. This clear scoring system provides an objective view of how well LLMs manage to generate secure code under different prompt settings.

Experimental Findings and Model Performance

The evaluations conducted using SafeGenBench revealed stark security risks inherent in the code generated by prominent LLMs. With average scores indicating a propensity for LLMs to produce vulnerable code, the empirical analysis highlighted deficiencies across all tested models. Notably, even with added guardrails—such as safety instructions and vulnerability exemplars—models demonstrated only marginal improvements in generating secure code snippets.

Moreover, reasoning models like those from OpenAI (e.g., GPT-4o and its variants) showed a relatively higher ability to generate secure code compared to non-reasoning models. Yet, even among these, the need for improved mechanisms to incorporate secure development practices was apparent. The dual-judge evaluation approach highlighted the complementary nature of SAST and LLM-based assessments, with each method uncovering different aspects of potential vulnerabilities.

Theoretical and Practical Implications

The findings from SafeGenBench hold significant implications for both AI model developers and users in software engineering domains. The recurrent vulnerabilities identified across LLM outputs call for enhancements in training datasets to better represent secure coding principles. Furthermore, the research underscores the pressing requirement for integrating robust security measures directly into LLM development frameworks.

For practitioners, these insights translate into a wariness about over-reliance on automatically generated code without extensive review and testing, especially in security-critical applications such as cryptographic or web-based systems. SafeGenBench also lays the groundwork for future advancements in automated security testing of LLM outputs, advocating for a harmonious integration of AI-driven tools with traditional manual and automated security review processes.

Future Directions

Moving forward, expanding SafeGenBench to cover even broader contexts and more complex coding environments would increase its robustness and applicability. Future developments might include project-level evaluations, considering intricate interdependencies between code modules. Additionally, evolving the evaluation framework to incorporate functionality assessments alongside security would provide a more comprehensive understanding of the safe and correct functioning of LLM-generated code.

In conclusion, SafeGenBench presents a critical step toward understanding and mitigating the security risks of LLM-generated code, fostering a path to more secure automatic code generation processes and enhancing the safety and trust in deploying AI-driven development tools.