---
title: 'CVEFixes Dataset: Real-World Vulnerability Fixes'
url: https://www.emergentmind.com/topics/cvefixes-dataset
type: topic
---

# CVEFixes Dataset: Real-World Vulnerability Fixes

CVEfixes, also written as CVEFixes or CVE_Fixes in later work, is a CVE-grounded dataset of real-world vulnerability fixes in open-source software. In its automated form, it is a relational collection built from National Vulnerability Database records that reference public fixing commits, and it stores vulnerable and patched code together with programming-language annotations, CWE and CVSS metadata, and code metrics at repository, CVE, commit, file, and method levels [2107.08760]. Subsequent studies have reused the same underlying resource to derive task-specific views, notably function-level C/C++ slices for binary vulnerability detection and larger cross-language snippet collections for language-model evaluation [2407.14372] [2412.15905].

## 1. Historical development and scope

The best-known form of CVEfixes originates in the 2021 automated collection pipeline that harvested all published CVEs up to 9 June 2021 when they contained public repository references with git commit hashes. That initial release covered 5,365 CVE records, 1,754 open-source repositories, and 5,495 unique fixing commits, with 27 detected programming languages and 180 CWE types [2107.08760]. The design goal was not only to enumerate fixes, but to preserve both pre-fix and post-fix source code and to connect them to security metadata such as CVSS scores and CWE classifications.

The name also sits within a broader lineage of CVE-linked fix datasets. A manually curated SAP Security Research release from 2019 mapped 624 publicly disclosed vulnerabilities affecting 205 distinct open-source Java projects onto 1,282 fixing commits, using a four-column schema centered on vulnerability identifiers, repository URLs, commit hashes, and class labels [1902.02595]. Later CVEfixes-based studies then shifted toward broader language coverage and task-oriented representations, including a cross-language snapshot with 277,948 labeled code snippets and a refined C/C++ function subset for LLM evaluation [2412.15905] [2407.14372].

## 2. Collection workflow and database organization

The automated collection process begins with yearly NVD JSON feeds. These feeds are flattened and parsed to extract CVE identifiers, publication and modification dates, CWE types, reference URLs, and CVSS v2 and v3 scores and subscores. CVEs without git-fix references in their `reference_json` entries are filtered out. For the remaining records, the referenced repositories are cloned from GitHub, GitLab, or Bitbucket, the fix commit is checked out, and its parent commit is retrieved so that the vulnerable and patched versions can be compared directly [2107.08760].

Code-level mining is then performed with PyDriller. At the file level, the pipeline stores the blobs before and after the fix, the raw git diff, a parsed JSON representation of added and deleted lines, the file language as detected via Guesslang, and size and complexity metrics. At the method level, it stores method name, signature, parameters, start and end lines, token count, lines of code, cyclomatic complexity, code, and a Boolean indicator specifying whether the method snapshot is taken before or after the change. Commit-level metadata include author, timestamp, commit message, merge status, added and deleted lines, and Delta Maintainability Model metrics [2107.08760].

The resulting schema is normalized around the relations `cve`, `cwe`, `cwe_classification`, `repository`, `fixes`, `commits`, `file_change`, and `method_change`. This organization makes the dataset queryable across several abstraction levels. In the maintainability layer, the overall Delta Maintainability Model score is defined as

$$
\text{DMM}_{\text{overall}}=\frac{\text{DMM}_{\text{size}}+\text{DMM}_{\text{complexity}}+\text{DMM}_{\text{interface}}}{3}.
$$

This multi-level structure is central to how CVEfixes differs from corpora that expose only commit messages or diffs, because it supports repository-level, CVE-level, commit-level, file-level, and method-level analyses within a single database [2107.08760].

## 3. Labels, granularity, and cross-language representations

Although the original database is relational and multi-level, later vulnerability-detection studies commonly use snippet-level views extracted from it. In a cross-language benchmark derived from CVEfixes and updated up to 23 July 2024, the collection contains 277,948 code snippets labeled as vulnerable or non-vulnerable, with 126,599 vulnerable and 151,349 non-vulnerable instances [2412.15905]. Each instance is linked to a CVE-ID from the NVD and to the fixing commit in an open-source repository.

The same study reports the following language-specific counts after trimming JavaScript to 100,000 samples stratified by commit timestamp [2412.15905]:

| Language | Vulnerable / Non-vulnerable | Total |
|---|---:|---:|
| JavaScript | 46,802 / 53,198 | 100,000 |
| PHP | 4,758 / 23,499 | 28,257 |
| C / C++ | 8,299 / 11,761 | 20,060 |
| Java | 3,102 / 5,285 | 8,387 |
| Python | 2,775 / 4,793 | 7,568 |
| Go | 1,880 / 4,403 | 6,283 |

In this representation, the per-instance schema includes `code_before_fix`, `code_after_fix` for vulnerable instances, `label ∈ {0,1}`, `CVE_ID`, `CWE_type`, `CVSS_score`, `project_name`, `repo_url`, `commit_hash`, `commit_date`, and a language tag, with optional fix descriptions and patch hunk coordinates [2412.15905]. The preprocessing used for this view includes byte-for-byte deduplication of identical code snippets, language-tag validation through Linguist and file extension checks, normalization of trailing whitespace and line endings, and a temporal integrity check that excludes fix commits later than the NVD publishing date of the CVE. Train and test partitions are temporal rather than purely random, so that the model sees only past vulnerabilities during training and future vulnerabilities during testing [2412.15905].

This use of CVEfixes as a snippet corpus suggests that the dataset functions less as a single immutable artifact than as a reusable source database from which different vulnerability-learning tasks extract different supervised views.

## 4. SCoPE refinement of the C/C++ function subset

A particularly detailed derivative is the C/C++ function-level subset studied in SCoPE. In that work, CVEFixes version 1.0.7 is treated as a collection of function-level code slices from real-world C/C++ projects. The reported subset contains 15,649 functions, of which 6,515 are vulnerable and 9,134 are non-vulnerable, with binary labels and metadata such as `method_change_id`, `before_change`, programming language, associated CVE and CWE identifiers, cyclomatic complexity, `file_change_id`, and project identifiers [2407.14372].

SCoPE, the Source Code Processing Engine, applies five orthogonal transformations implemented via `antlr4` parsing: programmer-defined identifier renaming into generic tokens such as `VAR_1` and `FUNC_1`, string literal replacement with `STR`, whitespace normalization, comment removal, and optional tokenization into space-separated tokens [2407.14372]. Deduplication is performed after transformation. If the normalized code strings for two samples satisfy \(C_i == C_j\), they are considered duplicates. Duplicate groups are then categorized into three cases: code differing only in comments, truly identical code before and after, and code identical modulo renamed identifiers. Exactly one record is retained from each duplicate group, and if both labels occur in the group, the clean sample with label \(0\) is preferentially kept [2407.14372].

The reported processing outcome is highly specific. Of the original subset, 12,035 samples were processed without parsing errors, 3,614 were excluded due to unsupported macros, 905 duplicates were detected, and the final retained set contained 11,573 functions, including 4,612 vulnerable and 6,961 clean functions [2407.14372]. For model training, this refined set was down-sampled to a balanced 9,224-sample subset with 4,612 vulnerable and 4,612 clean functions. Mean token length decreased from 374 to 300 after processing. The refined schema for this derivative contains `method_change_id`, `original_code`, `processed_code`, `label`, `programming_language`, `cve_id`, `cwe_id`, `cyclomatic_complexity`, and `parsing_error` [2407.14372].

This refinement is notable because it treats deduplication and normalization as first-class methodological issues rather than incidental cleaning. It also makes explicit that some of the apparent learning signal in raw vulnerability datasets can arise from duplicated or near-duplicated code.

## 5. Use in vulnerability-detection research

CVEfixes is widely used for binary vulnerability detection, but the exact evaluation setup depends on the derived view. In the refined C/C++ SCoPE subset, the train, validation, and test split is \(80\%\), \(10\%\), and \(10\%\), respectively, with stratification on the binary label to preserve the 50:50 class balance in each split [2407.14372]. Precision, recall, and \(F_1\) are defined in the standard way:

$$
\mathrm{Precision}=\frac{TP}{TP+FP}, \qquad
\mathrm{Recall}=\frac{TP}{TP+FN}, \qquad
F_1=2\times\frac{\mathrm{Precision}\times\mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}.
$$

On that subset, three pre-trained LLMs were fine-tuned for vulnerability detection: CodeBERT from the RoBERTa family, CodeGPT from the GPT-2 family, and NatGen from the Flan-T5 family. The reported hyperparameters are 10 epochs, batch size 16, learning rate \(2\times10^{-5}\), 200 warmup steps, and LoRA for PEFT. Best test-set results after SCoPE processing were approximately \(53\%\) precision, \(53\%\) recall, and \(53\%\) \(F_1\) for NatGen, about \(51\%\) \(F_1\) for CodeBERT, and about \(49\%\) \(F_1\) for CodeGPT. No significant improvement in \(F_1\) was observed before versus after processing, and the study states that current LLMs alone remain insufficient for reliable C/C++ vulnerability detection on real-world data [2407.14372].

Cross-language experiments derived from CVEfixes broaden that picture. A later study fine-tuned CodeT5, CodeBERT, UnixCoder, and DeepSeek-Coder on JavaScript, Java, Python, PHP, Go, and C/C++, using AdamW with learning rate \(2\times10^{-5}\), 10 epochs, gradient accumulation, linear warmup, and NVIDIA A100 GPUs [2412.15905]. That study reports that vulnerability-detection performance varies significantly by language and that JavaScript exhibits the best performance, with considerably better and more practical detection capabilities than C/C++. It also computes token length, Halstead volume, difficulty, effort, cyclomatic complexity, and NLOC, then correlates language-level \(F_1\) with these complexity measures; all reported \(|r|<0.6\) with \(p \gg 0.05\), indicating no statistically significant correlation [2412.15905]. A plausible implication is that benchmark difficulty in CVEfixes-derived evaluations is shaped at least as much by language ecology, label construction, and context granularity as by simple code-complexity measures.

## 6. Strengths, limitations, and benchmark status

Several strengths recur across the literature. CVEfixes offers cross-language coverage, direct linkage to real CVE records, rich metadata including CWE and CVSS information, commit dates and project names, and before/after code connected through a relational schema that supports arbitrary queries [2412.15905]. The collection process is repeatable, and the original release was explicitly designed to be rerun to incorporate newly published or newly linked CVEs [2107.08760]. These properties make it useful not only for vulnerability detection, but also for vulnerability classification, severity prediction, patch-pattern mining, and automated repair [2107.08760].

The limitations are equally well documented. Some CVE-referenced repositories are no longer available, which produces unavoidable coverage gaps; fix commits may be incomplete or may mix vulnerability remediation with unrelated edits; and extension to other version-control systems and issue trackers remains unfinished [2107.08760]. In cross-language LM usage, the sample sizes are strongly imbalanced by language, approximately \(20\%\) of NVD fixes on non-GitHub forges are not automatically collected, binary classification can be too coarse for the diversity of CWE patterns, and the common single-function framing lacks inter-procedural or data-flow context [2412.15905].

A recurring controversy is whether CVEfixes is heavily contaminated by public code-LLM training corpora. A 2026 perturbation-based study evaluates CVEFixes as a vulnerability-detection benchmark under consistent identifier-renaming perturbations, defines memorization advantage as \(M=P_{\text{seen}}-P_{\text{unseen}}\), and defines per-example sensitivity as \(s(X)=\max_{k=0\ldots4}[m^k-m^{k+1}]\) [2604.13997]. In that study, CVEFixes is the only vulnerability benchmark whose median sensitivity is below \(0.1\) for all eight evaluated code LLMs, with reported median values from \(0.02\) for DeepSeek-Coder to \(0.08\) for StarCoder [2604.13997]. The paper argues that this low perturbation sensitivity challenges the assumption that performance on CVEfixes is primarily driven by direct memorization. This suggests that, despite longstanding concerns about leakage, CVEfixes remains a viable benchmark for probing learned vulnerability patterns, provided that evaluation protocols explicitly control for duplication, temporal leakage, and granularity effects.

Source: https://www.emergentmind.com/topics/cvefixes-dataset