Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks (2512.03262v1)

Abstract: Vibe coding is a new programming paradigm in which human engineers instruct LLM agents to complete complex coding tasks with little supervision. Although it is increasingly adopted, are vibe coding outputs really safe to deploy in production? To answer this question, we propose SU S VI B E S, a benchmark consisting of 200 feature-request software engineering tasks from real-world open-source projects, which, when given to human programmers, led to vulnerable implementations. We evaluate multiple widely used coding agents with frontier models on this benchmark. Disturbingly, all agents perform poorly in terms of software security. Although 61% of the solutions from SWE-Agent with Claude 4 Sonnet are functionally correct, only 10.5% are secure. Further experiments demonstrate that preliminary security strategies, such as augmenting the feature request with vulnerability hints, cannot mitigate these security issues. Our findings raise serious concerns about the widespread adoption of vibe-coding, particularly in security-sensitive applications.

• The paper introduces the SusVibes benchmark to evaluate both functional correctness and security of agent-generated code in real-world open-source projects.
• It analyzes 200 feature tasks from 108 Python repositories, revealing that over 80% of functionally correct outputs still harbor exploitable vulnerabilities.
• Security-focused prompting strategies failed to improve vulnerability rates, underscoring the need for robust agent architectures and integrated external security tools.

Benchmarking Security in Vibe Coding: An Analysis of SusVibes

Introduction

The proliferation of LLM-powered coding agents has led to a new paradigm termed "vibe coding," wherein high-level, natural language feature requests are directly translated by AI agents into substantial code contributions across complex codebases. The paper "Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks" (2512.03262) raises critical questions concerning the security risks inherent in this workflow. To address the current lack of benchmarks targeting agent-driven, repository-level feature implementations with real security impact, the authors introduce SusVibes—a benchmark designed to rigorously evaluate not only the functional correctness but also the security posture of agent-generated code in challenging, real-world settings.

Benchmark Design and Construction

SusVibes systematically curates 200 feature-request tasks spanning 108 open-source Python software repositories, chosen for their historical association with real security vulnerabilities. Each task mandates the addition of a feature whose prior, human-written implementation introduced a security bug since remediated in the project's version control history. The benchmark therefore targets the highest-risk mode of LLM agent deployment: unconstrained synthesis over production code, with ground truth drawn from real security incidents.

The curation pipeline involves mining vulnerability-fixing commits, masking the relevant feature across multiple files, and synthesizing both a functional feature request and corresponding unit/security tests to mirror realistic requirements elicitation and acceptance conditions.

Figure 2: Overview of the data curation process—commits fixing vulnerabilities are mined, feature masks and task descriptions are generated, and functional/security tests are harnessed for evaluation.

Functional test cases are taken from pre-fix artifacts, while security test cases are derived from post-fix additions. The pipeline automatically constructs both the runtime environment (including Python version selection and Docker-based containerization) and test log parsers, ensuring that code agents are evaluated under practical constraints matching CI/CD deployment scenarios.

Task complexity is notably higher than in prior LLM security benchmarks. The average repository context is 162K lines, with solution patches requiring an average of 172 lines of cross-file edits and extensive test coverage for both correctness and vulnerability detection.

Figure 1: Domain diversity in SusVibes's 108 target projects, illustrating breadth of application settings.

Experimental Evaluation

Three agent frameworks (SWE-agent, OpenHands, Claude Code) were paired with three leading LLMs (Claude 4 Sonnet, Kimi K2, Gemini 2.5 Pro) to rigorously assess functional and security performance across all resulting system configurations. Evaluation metrics include FuncPass (pass@1 for functional correctness) and SecPass (pass@1 for both correctness and security). Only functionally-correct outputs are considered when assessing security rates, reflecting realistic acceptance workflows.

Despite advances in LLM coding capabilities, results indicate that all systems exhibit a pronounced deficit in secure code generation. The best-performing configuration (SWE-agent + Claude 4 Sonnet) achieves 61% functional correctness, yet only 10.5% of these are secure—implying that upwards of 80% of successfully implemented features harbor known vulnerabilities.

Security Failures and CWE Analysis

SusVibes's coverage includes 77 distinct Common Weakness Enumeration (CWE) categories—a significant expansion over prior repositories. Stratified analysis reveals that both LLM choice and agentic scaffold influence which classes of vulnerabilities are avoided. Importantly, the overlap in CWE types successfully mitigated by different LLMs is minimal, indicating complementary but narrow "security strengths" and unaddressed blind spots. For example, only 25% of the CWEs avoided by functions produced via OpenHands+Claude 4 Sonnet overlap with those avoided by SWE-agent+Claude or SWE-agent+Gemini.

Figure 5: Venn diagram shows distribution over CWEs where at least 25% of instances are successfully mitigated by each system, illustrating limited overlap and differentiated blind spots.

Qualitative error analyses reinforce these quantitative findings. Case studies highlight vulnerabilities ranging from timing side-channels in user authentication flows and unsafe URL handling to session fixation and unbounded session lifetimes—all surfacing in apparently functional agent outputs.

Figure 7: Example of an agent-generated code patch for a security-critical feature, showing functional correctness but a critical vulnerability due to missing constant-time password comparison.

Figure 3: Agent solution to a session-management task fails to enforce session expiration, enabling session replay and privilege escalation.

Preliminary Security Mitigation Strategies

The paper investigates whether security can be improved via agent-prompting strategies alone. Two settings are explored: (1) requesting that agents self-select relevant CWE risks prior to code generation, and (2) providing the task's ground-truth CWE category as explicit guidance (oracle). The data demonstrate that neither approach improves the joint rate of secure+correct solutions—in fact, both result in a decrease in functionally correct implementations, indicating a trade-off rather than a net improvement (e.g., a 6-9 percentage point drop in functional passing rates with no statistically significant gain in security).

Figure 4: Secure and functional pass ratios for baseline and security-guided strategies; performance deteriorates or remains stagnant with additional security prompting.

Analysis of high-level agent behaviors reveals that increased attention to security often coincides with omission of necessary functional requirements, underscoring the inadequacy of naive prompt-based mitigation in multi-step agentic workflows.

Implications and Future Directions

The findings of SusVibes have direct implications for both academic research and industrial deployment of LLM-based code generation tools:

• Code review and security analysis remain essential: Reliance on functional test suites is insufficient to prevent introduction of exploitable vulnerabilities by agentic LLMs.
• Prompt engineering alone is ineffective for security: Security-aware prompts and CWE reminders do not yield meaningful improvements in secure-by-design solutions in open-ended, multi-step agent workflows.
• Benchmark-driven research is needed: SusVibes exposes diverse, real-world failure modes inaccessible to prior benchmarks, motivating the integration of property-based testing, semantic/static analysis tools, and security-focused fine-tuning rewards within agentic pipelines.
• Model specialization and agent design impact bias: Security strengths and weaknesses are highly contingent on both LLM pretraining and agent system architecture; ensemble or component-based approaches may partially mitigate individual blind spots.

Looking forward, comprehensive secure coding agents will likely require joint advances in multi-modal vulnerability detection, dynamic/adversarial evaluation, and explicit integration of external security analysis tools (e.g., fuzzers, taint analysis, property checkers) into the agent decision loop.

Conclusion

SusVibes establishes that current state-of-the-art LLM agents, even while achieving strong pass rates on complex feature requests, are systematically prone to generating insecure code patches. The lack of correlation between functional success and security pass rates, coupled with the ineffectiveness of prompt-based security interventions, demonstrates that security cannot be left as a post hoc concern. Rather, robust agent architectures and enhanced training signals incorporating security outcomes are necessary for safe adoption of vibe coding in production, especially in security-critical domains.