SecCodePLT: Unified Code Security Assessment

• SecCodePLT is a comprehensive platform that quantitatively evaluates AI-generated insecure code and cyberattack facilitation using dynamic, real-world simulations.
• It integrates expert annotations with automated LLM-driven mutations to produce scalable benchmarks covering 27 Python CWEs and realistic attack scenarios.
• The platform employs a hybrid dynamic evaluation, executing both functionality and security tests along MITRE ATT&CK phases for robust risk measurement.

SecCodePLT is a unified platform for comprehensive quantitative evaluation of code-generation AI security risks, emphasizing both the generation of insecure code and facilitation of executable cyberattacks. Its architecture combines expert-driven and automated benchmark creation, dynamic test execution, and detailed multi-dimensional metrics. Notably, SecCodePLT advances prior work by integrating dynamic evaluation and real-world attack simulation, supporting rigorous and scalable risk measurement across state-of-the-art code LLMs and agents (Yang et al., 2024).

1. Architecture and Evaluation Dimensions

SecCodePLT consists of two primary evaluation modules targeting distinct security axes: “insecure coding” and “cyberattack helpfulness”. The insecure coding module generates security-relevant code tasks spanning 27 Python-critical CWEs and leverages dynamic test cases for both functionality and vulnerability detection. The cyberattack helpfulness module orchestrates a realistic multi-machine environment (web server, database, AD, internal user, attacker host) wherein candidate models are prompted to engineer and execute actual end-to-end attacks mapped to MITRE ATT&CK phases.

This dual structure enables coverage that simultaneously exposes the tendency of models to produce insecure code and their ability to generate actionable exploits, unified by extensible JSON + Python templates and a dynamic execution engine (Yang et al., 2024).

2. Benchmark Creation and Scaling Methodology

SecCodePLT’s insecure-code benchmark construction follows a two-stage methodology. First, expert annotators author 153 “seed” tasks, each embodying a specific CWE scenario with natural-language prompts, a vulnerable code variant, a corrected reference, and modular functional plus security-oriented test cases. These seeds provide high fidelity but limited scale.

To scale, each seed undergoes LLM-driven mutation: prompt mutators paraphrase the scenario and code mutators synthesize code-level refactorings. After each mutation, candidate variants are filtered using the defined test cases (regenerating those that fail), producing an aggregate benchmark of 1 345 unique samples, each with a structured prompt, code, and test suite. This hybrid pipeline yields both diversity and precision at scale, without sacrificing expert verification quality (Yang et al., 2024).

3. Dynamic Evaluation Metrics and Execution Framework

SecCodePLT eschews purely static or LLM-judged metrics, adopting a hybrid dynamic framework. For each benchmark instance $i$ and model $M$, code outputs are assessed by running both functional (correctness) and security (violation) tests:

• Functionality pass rate (pass@1):

$$\mathrm{pass@1} = \frac{1}{N} \sum_{i=1}^{N} f_i, \quad f_i = \mathbb{I}[\text{func\_tests\_passed}_i]$$

• Secure-code rate:

$$\mathrm{SecureCodeRate} = \frac{1}{N} \sum_{i=1}^{N} s_i, \quad s_i = \mathbb{I}[\text{no\_security\_fail}_i]$$

Prompted code is then dynamically executed, with test outcomes directly determining security rates—where functional plus security-triggering tests provide robust defense against false positives/negatives endemic to static analysis (Yang et al., 2024).

For cyberattack helpfulness, attack success and refusal rates are similarly quantified by executing model-generated commands in the live environment and analyzing outcome traces:

$$\mathrm{SuccessRate}_c = \frac{S_c}{T},\qquad \mathrm{RefusalRate}_c = \frac{R_c}{T}$$

where $S_c$ and $R_c$ denote success and refusal counts in $T$ trials per category $c$.

4. End-to-End Cyberattack Simulation Environment

SecCodePLT implements a full-stack simulated infrastructure reflective of typical enterprise and e-commerce deployments. Attack evaluations proceed via interactive shell-based sessions: models receive structured prompts per attack step and return executable shell or PowerShell commands, which are executed in sequence within the live network (comprising web, DB, AD, internal user, and attacker hosts). Outputs—including stdout, stderr, network artifacts—are captured and fed back iteratively to the model for further exploit crafting, up to 40 steps per scenario.

Attack progression is mapped to MITRE ATT&CK phases and scored according to real system impact (e.g., privilege escalation, persistent compromise). This dynamic approach distinguishes SecCodePLT from prior benchmarks lacking executable attack paths (Yang et al., 2024).

5. Empirical Benchmarks and SOTA Comparisons

SecCodePLT is empirically benchmarked against CyberSecEval, the prior state-of-the-art. Key metrics include “security relevance” (the fraction of prompts truly reflecting a security scenario), “instruction faithfulness” (alignment of description and code intent), CWE/task coverage, evaluation modality (static vs dynamic), and expert verification.

	SecCodePLT	CyberSecEval
Insecure Coding & Attack Coverage	27 Python CWEs + attacks, 1 345 samples; dynamic	8 CWEs, 300 samples; static
Security relevance	~100% (LLM judge)	67.8% (Python subset)
Instruction faithfulness	~100%	~42% (CWE-dependent)
Dynamic execution & expert review	Yes	No
End-to-end attack evaluation	Yes	No (suggestion only)

SecCodePLT achieves near-complete security relevance and instruction faithfulness; empirically, GPT-4o models achieve only ~55% secure-code rate, and refusal rates on attacks vary by platform (Claude’s refusal is higher than GPT-4o’s 8–10%). The platform uncovers nontrivial security failures (e.g., failures on CWE-79, CWE-95) even in advanced agents such as Cursor, which were previously undetected (Yang et al., 2024).

6. Analysis of SOTA Agents: Cursor Case Study

Application of SecCodePLT to Cursor reveals substantial and specific security gaps. Across 153 seed tasks, rule-based secure-code rates for Cursor rise from 62% to 86.7% when explicit security-policy prompts are added, but pass@1 dynamic security rates remain at 52.8% (no policy) and 67.4% (with policy). Certain critical CWEs (XSS, eval injection, broken crypto, incorrect authorization, data exposure) show complete security failures (0% secure code). Cursor also fails to reliably implement atomic checks (TOCTOU), input sanitization, and other context-relevant security measures despite alignment attempts.

This suggests that high-level code agents may pass superficial static checks but fail dynamic, scenario-relevant security validations. A plausible implication is the necessity for robust platform-level dynamic security evaluation and alignment, beyond surface prompt engineering (Yang et al., 2024).

7. Summary, Further Implications, and Future Research

SecCodePLT establishes the first rigorous, extensible, and empirically validated platform for assessing both insecure code generation and real-world attack facilitation in GenAI models. Its scalable expert + LLM-based benchmark, combined with dynamic, executable metrics, provides a foundation for reproducible security assessments. Comparative results show substantial advances over prior benchmarks in both breadth and depth.

Open directions include multi-language extension, broader attack scenario inclusion, code reasoning tasks, and leveraging SecCodePLT as a curriculum or guardrail generator for alignment and model improvement. The platform’s unified methodology and dynamic analysis are positioned to become central tools for security-aware GenAI development and assessment (Yang et al., 2024).