CWEval-bench: Secure Code Generation Benchmark
- CWEval-bench is a multilingual security-critical coding benchmark that jointly evaluates code functionality and secure outcomes using dynamic test oracles.
- It comprises 119 tasks covering 31 CWE types across 5 programming languages, including focused tests for C memory vulnerabilities.
- The benchmark employs func@k and func-sec@k metrics to quantify the prevalence of functionally correct yet insecure code generated by LLMs.
Searching arXiv for papers on CWEval-bench and closely related benchmarks. CWEval-bench is a multilingual security-critical coding benchmark used within the broader CWEval framework to evaluate whether LLMs generate code that is not only functionally correct but also secure for the same task (Peng et al., 14 Jan 2025). It consists of task specifications, functionality and security test oracles, secure and insecure reference implementations, and multilingual task variants. The benchmark contains 119 security-critical coding tasks covering 31 CWE types across 5 popular programming languages, including 11 C-specific tasks focused on memory-related vulnerabilities (Peng et al., 14 Jan 2025). Its central purpose is to measure the practically important class of outputs that are functionally correct yet insecure, a failure mode that earlier secure code generation benchmarks are argued to miss because of vague specifications, incomplete functionality evaluation, and static-analysis-driven security judgments (Peng et al., 14 Jan 2025).
1. Position within secure code evaluation
CWEval-bench should be distinguished from CWEval, which is the evaluation framework or methodology, whereas CWEval-bench is the benchmark suite that instantiates that framework in practice (Peng et al., 14 Jan 2025). The framework is centered on jointly evaluating functionality and security on the same tasks by using high-quality task specifications, outcome-driven test oracles, and secure and insecure reference implementations. CWEval-bench is therefore the concrete benchmark layer of an outcome-driven evaluation design rather than merely a collection of vulnerability examples (Peng et al., 14 Jan 2025).
The benchmark was introduced in response to what the paper identifies as three deficiencies in prior secure code generation evaluation. First, benchmarks such as CyberSecEval and SecurityEval are described as having unclear or impractical task specifications. Second, they do not rigorously assess functionality and security simultaneously. Third, their security judgments rely heavily on static analyzers, which can produce both false negatives and false positives (Peng et al., 14 Jan 2025). The paper uses concrete evidence for this critique: in CyberSecEval, fewer than one third of included vulnerable samples were reproducible, specifically 562 / 1916 (Peng et al., 14 Jan 2025).
This positioning makes CWEval-bench distinct from non-security reasoning benchmarks. CRUXEval and CRUXEval-X are execution-based benchmarks for code reasoning and understanding, but they are not organized around vulnerability classes or secure coding outcomes (Gu et al., 2024, Xu et al., 2024). STEPWISE-CODEX-Bench similarly targets multi-function dynamic execution reasoning rather than weakness identification, exploitability, or security semantics (Yan et al., 7 Aug 2025). By contrast, CWEval-bench is explicitly designed so that a task may admit both a functionally correct but insecure solution and a functionally correct and secure solution, making secure code generation the primary object of evaluation (Peng et al., 14 Jan 2025).
2. Benchmark scope and task design
CWEval-bench spans 119 tasks, 31 CWE types, and 5 programming languages, with a core multilingual testing set of tasks valid across supported languages and additional language-specific tasks (Peng et al., 14 Jan 2025). The paper explicitly highlights 11 C-specific memory vulnerability tasks, indicating that the benchmark is not restricted to language-agnostic weaknesses and instead incorporates vulnerabilities tied to specific language properties (Peng et al., 14 Jan 2025).
The benchmark focuses on security-critical programming behaviors rather than on generic algorithmic coding. Representative task categories mentioned in the paper include file operations, process creation, sensitive data processing, URL sanitization / redirect handling, SQL-related integrity/side-effect scenarios, denial-of-service related tasks where time complexity/runtime matters, and memory-safety tasks in C such as out-of-bounds access (Peng et al., 14 Jan 2025). This design choice matters because it separates CWEval-bench from benchmarks like HumanEval, which measure code generation ability but do not primarily test security-relevant semantics (Peng et al., 14 Jan 2025).
Task specifications are written in a style inspired by functionality benchmarks such as HumanEval. Each specification may include a function signature, a natural language docstring/description, and optional example input-output pairs (Peng et al., 14 Jan 2025). The paper states three explicit design requirements for specification writing. First, security-semantics existence: each task must contain semantics that make security relevant. Second, no security-awareness leakage: specifications avoid direct security hints such as using words like “safe” or “unsafe” in variable names or explicitly instructing the model to be secure. Third, expectation unambiguity: the specification must be clear enough that failure is not attributable to misunderstanding the intended functionality (Peng et al., 14 Jan 2025).
This suggests that CWEval-bench is designed to minimize a common confound in secure generation evaluation: a model should not be able to score well merely by refusing to implement the requested behavior, producing a no-op, or exploiting underspecified requirements. A plausible implication is that the benchmark aims to separate true secure coding competence from both alignment-driven refusal and task-specification ambiguity.
3. Construction methodology and quality control
The paper describes a multi-stage construction process grounded in CWE documentation and CodeQL documentation rather than repository mining (Peng et al., 14 Jan 2025). Tasks are therefore primarily hand-designed / manually crafted, not extracted wholesale from existing vulnerable projects. This is a deliberate contrast with CyberSecEval, which uses vulnerable code automatically mined from open-source repositories (Peng et al., 14 Jan 2025).
The first stage is coding task design. Each task is made self-contained, and the paper emphasizes security semantics isolation, meaning that the core vulnerability-relevant semantics are preserved while minimizing or eliminating third-party library dependencies (Peng et al., 14 Jan 2025). The stated rationale is improved reproducibility, easier evaluation, simpler multilingual translation, and cleaner testing of fundamental security understanding rather than of framework-specific API familiarity (Peng et al., 14 Jan 2025).
The second stage is specification writing, using the requirements described above. The paper adds a practical quality-control step: specifications are tested with one or more common LLMs and iteratively refined if needed (Peng et al., 14 Jan 2025). The third stage is test oracle and reference solution development. For each task, the authors develop one functional and secure implementation and at least one functional but insecure implementation (Peng et al., 14 Jan 2025). This is used to validate that the vulnerability is real and reproducible, that secure functionality is achievable, and that the test oracles actually distinguish secure from insecure behavior (Peng et al., 14 Jan 2025).
The fourth stage is multilingual evolution. Tasks are first translated using LLMs and then manually reviewed and refined (Peng et al., 14 Jan 2025). The paper does not enumerate all five languages in the visible excerpt, but it explicitly mentions C and discusses multilingual translation methodology across the supported set (Peng et al., 14 Jan 2025).
Compared with CASTLE, another CWE-focused benchmark, CWEval-bench is less about cross-paradigm comparison among static analyzers, formal verification tools, and LLMs, and more about outcome-driven evaluation of generated code under joint functional and security constraints (Dubniczky et al., 12 Mar 2025). CASTLE uses 250 manually crafted micro-benchmark programs in C, covering 25 common CWEs, with exactly 10 test cases per CWE: 6 vulnerable and 4 non-vulnerable (Dubniczky et al., 12 Mar 2025). CWEval-bench instead centers the generated-solution setting and multilingual task design (Peng et al., 14 Jan 2025).
4. Outcome-driven functionality and security oracles
The benchmark’s defining technical feature is its use of outcome-driven dynamic test oracles (Peng et al., 14 Jan 2025). Functionality is evaluated by running functionality test oracles, and security is evaluated by running security test oracles on the same generated implementation (Peng et al., 14 Jan 2025). This differs from static-analysis-driven evaluation, which asks whether code resembles a vulnerable pattern; CWEval instead asks whether execution produces a secure or insecure outcome (Peng et al., 14 Jan 2025).
The security oracles are not restricted to output checking. The paper explicitly states that they may inspect runtime time cost, memory access validity, and side effects / data integrity, in addition to output behavior (Peng et al., 14 Jan 2025). Several examples are given. For DoS detection, the implementation is run with a timeout, and secure behavior must finish gracefully within the time limit. For out-of-bounds access in C, the code is compiled with AddressSanitizer and checked for memory errors. For SQL injection, the oracle examines side effects / integrity of data, not merely returned values (Peng et al., 14 Jan 2025).
This design is intended to overcome the weaknesses of earlier static-analysis-based benchmarks. The paper argues that semantically equivalent insecure implementations may evade rules, while secure implementations may be spuriously flagged. It presents a Figure 1 case study in which two insecure implementations are caught by CodeQL, two semantically similar insecure implementations are missed, and one secure implementation is wrongly flagged (Peng et al., 14 Jan 2025). The claimed consequence is a reduction in both false negatives and false positives when evaluation is shifted from static pattern recognition to outcome-driven behavioral testing (Peng et al., 14 Jan 2025).
The benchmark conceptually distinguishes four classes of generated code: functional and secure, functional and insecure, non-functional and secure, and non-functional and insecure (Peng et al., 14 Jan 2025). The principal analytic target is the second class, because functionally correct yet insecure code is precisely the sort of output that developers may accept and deploy (Peng et al., 14 Jan 2025).
5. Metrics and experimental protocol
CWEval-bench uses two pass@k-style metrics. The first is , defined as the probability that at least one of the generated implementations is functionally correct, meaning that it passes all functionality test oracles (Peng et al., 14 Jan 2025). The second is , defined as the probability that at least one of the generated implementations is both functionally correct and secure, meaning that it passes all functionality and all security test oracles (Peng et al., 14 Jan 2025).
The paper states that both metrics are calculated in the same way as using the unbiased estimator (Peng et al., 14 Jan 2025). It also reports greedy decoding results as $\text{func@}1^\*$ and $\text{func-sec@}1^\*$, which are deterministic single-sample pass rates under greedy search (Peng et al., 14 Jan 2025). The difference between and is used to quantify the prevalence of functional-but-insecure outputs (Peng et al., 14 Jan 2025).
For the main evaluation setup, each model is sampled with program samples and temperatures in 0, and the best-performing func@1, func@10, func@50, func-sec@1, func-sec@10, and func-sec@50 are reported (Peng et al., 14 Jan 2025). Greedy decoding is also evaluated (Peng et al., 14 Jan 2025). For budget-constrained follow-up experiments, the paper uses random sampling with 1 and temperature 2 (Peng et al., 14 Jan 2025).
The model suite includes GPT-4o mini, Claude 3.5 Haiku, Gemini 1.5 Flash, and Llama 3.1 70B Instruct in the main comparison, with larger and smaller family variants including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3.1 8B Instruct, and Llama 3.1 405B Instruct (Peng et al., 14 Jan 2025). The paper also evaluates a fine-tuning baseline using CodeLlama-7b-hf and CodeLlama-7b-hf w/ SafeCoder (Peng et al., 14 Jan 2025).
6. Empirical findings and benchmark significance
The headline result is that functionally correct yet insecure code is common. The paper states that from func@10 to func-sec@10, performance drops by around 30% across all models, with the maximum drop being 35.79% for Gemini 1.5 Flash (Peng et al., 14 Jan 2025). This is the benchmark’s central empirical message: evaluating only functionality materially overestimates safe deployment readiness (Peng et al., 14 Jan 2025).
Model scaling within a family generally improves the joint functionality-and-security metric more clearly than it improves functionality alone. Reported examples include GPT-4o vs GPT-4o mini on func-sec@10: 65.33 vs 60.28, Claude 3.5 Sonnet vs Claude 3.5 Haiku: 59.2 vs 56.48, Gemini 1.5 Pro vs Flash: 53.08 vs 47.81, and Llama 3.1 405B vs 70B: 53.7 vs 52.07 (Peng et al., 14 Jan 2025). The paper emphasizes that functionality-only metrics can understate security-relevant differences between models (Peng et al., 14 Jan 2025).
The benchmark also supports prompting studies. Adding the instruction “Your code should be secure and should NOT contain any vulnerability” generally improves func-sec@k with only slight functionality loss (Peng et al., 14 Jan 2025). Reported func-sec@10 changes include GPT-4o mini: 60.28 → 64.12, Claude 3.5 Haiku: 56.48 → 66.28, Gemini 1.5 Flash: 47.81 → 48.97, and Llama 3.1 70B: 52.07 → 53.47 (Peng et al., 14 Jan 2025). The largest reported gain is 9.8% for Claude 3.5 Haiku on func-sec@10 (Peng et al., 14 Jan 2025).
A particularly important result concerns security fine-tuning. On CWEval-bench, CodeLlama-7b-hf w/ SafeCoder shows strong degradation relative to CodeLlama-7b-hf, with func@10: 68.82 → 33.75 and func-sec@10: 40.74 → 18.26 (Peng et al., 14 Jan 2025). The paper interprets this as evidence that separate functionality and security benchmarks may reward models that avoid generating useful security-sensitive code, whereas CWEval-bench exposes this as an alignment tax in security-critical programming (Peng et al., 14 Jan 2025).
Relative to CASTLE, CWEval-bench addresses a different evaluation problem. CASTLE evaluates vulnerability detection capabilities of 13 static analysis tools, 10 LLMs, and 2 formal verification tools on 250 micro-benchmark programs in C, using the CASTLE Score to compare heterogeneous detectors (Dubniczky et al., 12 Mar 2025). CWEval-bench instead evaluates secure code generation under joint outcome constraints (Peng et al., 14 Jan 2025). Relative to SX-Bench, CRUXEval, and CRUXEval-X, CWEval-bench is orthogonal in semantic domain: those benchmarks probe execution reasoning, dynamic control/data-flow understanding, or multilingual code reasoning, but not CWE taxonomy knowledge or secure code outcomes (Yan et al., 7 Aug 2025, Gu et al., 2024, Xu et al., 2024). This suggests that CWEval-bench measures a security-specific capability layer, while the execution-reasoning benchmarks measure substrate capabilities that may be complementary prerequisites for vulnerability reasoning.
7. Limitations, comparison landscape, and research use
The paper acknowledges that the current benchmark size is limited: CWEval-bench contains 119 tasks, and this scale is explicitly attributed to limited human resources (Peng et al., 14 Jan 2025). The construction process relies heavily on manual design, manual verification, and manual refinement, which improves quality but constrains scalability (Peng et al., 14 Jan 2025). The authors suggest future automation of benchmark creation and expansion (Peng et al., 14 Jan 2025).
Its methodological strengths and limitations become clearer when compared with neighboring benchmarks. CASTLE offers stronger support for cross-tool comparison, line-level localization, and false-positive analysis in a compact C-only setting (Dubniczky et al., 12 Mar 2025). CRUXEval and CRUXEval-X offer execution-based reasoning tasks with pass@1 evaluation and explicit code-understanding focus, but they do not evaluate security-critical semantics (Gu et al., 2024, Xu et al., 2024). STEPWISE-CODEX-Bench pushes further toward multi-function dynamic execution reasoning, counting atomic computation steps via an instrumented global counter run_steps, yet remains explicitly outside vulnerability analysis (Yan et al., 7 Aug 2025). ComplexCodeEval emphasizes repository-grounded, context-rich software engineering tasks such as code generation, completion, API recommendation, and test generation, but again not vulnerability detection or secure generation (Feng et al., 2024). In this landscape, CWEval-bench occupies the niche of multilingual, security-critical, outcome-driven evaluation of generated code (Peng et al., 14 Jan 2025).
For research and practice, the benchmark supports at least four use cases that are explicit in the paper. It can benchmark an LLM’s secure generation ability via func@k and func-sec@k; quantify the prevalence of functional-but-insecure outputs by comparing those metrics; compare prompting strategies such as explicit security reminders; and evaluate whether alignment or fine-tuning methods actually improve secure code generation rather than merely suppressing task completion (Peng et al., 14 Jan 2025). Because tasks are mapped to 31 CWE types, including language-specific classes such as C memory errors, it also supports per-vulnerability-class analysis in principle, although the excerpt does not provide a full task-to-CWE schema (Peng et al., 14 Jan 2025).
CWEval-bench is therefore best understood as a benchmark for joint secure functionality rather than for vulnerability detection alone. Its major conceptual contribution is to treat security evaluation as an outcome problem: the decisive question is not whether generated code superficially resembles a known secure pattern, but whether it behaves correctly and securely under executable test oracles on tasks where security semantics are intrinsic (Peng et al., 14 Jan 2025).
