ARVO: Atlas of Reproducible Vulnerabilities for Open Source Software (2408.02153v1)

Abstract: High-quality datasets of real-world vulnerabilities are enormously valuable for downstream research in software security, but existing datasets are typically small, require extensive manual effort to update, and are missing crucial features that such research needs. In this paper, we introduce ARVO: an Atlas of Reproducible Vulnerabilities in Open-source software. By sourcing vulnerabilities from C/C++ projects that Google's OSS-Fuzz discovered and implementing a reliable re-compilation system, we successfully reproduce more than 5,000 memory vulnerabilities across over 250 projects, each with a triggering input, the canonical developer-written patch for fixing the vulnerability, and the ability to automatically rebuild the project from source and run it at its vulnerable and patched revisions. Moreover, our dataset can be automatically updated as OSS-Fuzz finds new vulnerabilities, allowing it to grow over time. We provide a thorough characterization of the ARVO dataset, show that it can locate fixes more accurately than Google's own OSV reproduction effort, and demonstrate its value for future research through two case studies: firstly evaluating real-world LLM-based vulnerability repair, and secondly identifying over 300 falsely patched (still-active) zero-day vulnerabilities from projects improperly labeled by OSS-Fuzz.

• The paper presents ARVO, a system that generates a reproducible dataset of 5,001 real-world open source vulnerabilities using OSS-Fuzz metadata.
• It details a robust reproducer and patch locator methodology that mitigates dependency issues and accurately identifies security patches.
• The dataset’s scalability and reliability offer significant advances for automated vulnerability repair and software security research.

ARVO: Atlas of Reproducible Vulnerabilities for Open Source Software

Introduction

The paper "ARVO: Atlas of Reproducible Vulnerabilities for Open Source Software" addresses the complexities involved in curating a robust and scalable dataset of real-world software vulnerabilities for research purposes. The authors pinpoint the shortcomings of existing datasets and introduce ARVO, a system designed to generate a comprehensive dataset of reproducible vulnerabilities sourced from Google’s OSS-Fuzz. This dataset, accommodating over 5,000 memory vulnerabilities from more than 250 C/C++ projects, ensures reproducibility and encompasses metadata crucial for software security research.

Motivation and Challenges

The primary motivation behind ARVO is the pivotal role that high-quality vulnerability datasets play in advancing research in software security. Many existing datasets, such as those from the National Vulnerability Database (NVD) and Common Vulnerabilities and Exposures (CVE), are primarily designed for alerting users rather than serving as research benchmarks. These datasets often miss critical features such as re-compilability with triggering inputs and precise patches. Additionally, continuously updating these datasets manually is labor-intensive and inefficient.

Methodology

The ARVO system is grounded in the goal of addressing these limitations by ensuring high reproducibility and scalability. The authors delineated several key strategies to tackle the challenges associated with dataset creation.

Reproducer

The reproducer is central to ARVO's approach, utilizing metadata from OSS-Fuzz to recreate the environment through precise dependency control and minimal build script alteration. By retaining the project's build environment from the initial target version, ARVO ensures that dependencies match the specific versions at which vulnerabilities were discovered. This approach effectively mitigates common compilation issues stemming from incorrect dependency versions, thus enabling successful reproduction of vulnerabilities.

Patch Locator

Identifying the exact patches that fix vulnerabilities is indispensable for dataset quality. ARVO employs a robust bisecting approach to locate precise patches across versions indicated by OSS-Fuzz. Leveraging the reproducible build environments, ARVO meticulously identifies and verifies the correct patches, thereby ensuring that detected vulnerabilities are appropriately cataloged and continuously updated.

Dataset Characteristics

The resulting ARVO dataset offers a substantial improvement over prior databases in several ways. Firstly, it comprises 5,001 validated vulnerabilities and their corresponding fixes, all within a precisely controlled build environment. Importantly, ARVO maintains continuous and scalable growth by incorporating new vulnerabilities as discovered by OSS-Fuzz.

The dataset's project distribution is diversified, with no single project overwhelming the data pool, supporting varied and representative research applications. Another noteworthy feature is the dataset's focus on ensuring the availability of precise patches, which has significant implications for the paper of security fixes.

Case Studies on Utility

To demonstrate ARVO's utility, the authors presented two case studies:

1. LLM-Based Vulnerability Repair: Leveraging ARVO, the authors evaluated the performance of GPT-3.5 and GPT-4 in automatically fixing vulnerabilities. Results highlighted GPT-4's superior ability (20% success rate) in generating patches that mitigate vulnerabilities. However, they also pointed out that some fixes generated were superficial, stopping the PoC from triggering but not addressing the underlying vulnerability.
2. Zero-day Vulnerabilities and False Positives in OSS-Fuzz: The analysis revealed a significant number of false positives in OSS-Fuzz's reported fixes, some of which resulted in publicly exposed zero-day vulnerabilities. ARVO identified instances where reported vulnerabilities were not actually fixed, thereby underscoring the importance of comprehensive and validated vulnerability datasets.

Implications and Future Directions

ARVO's contribution to the field of software security research is substantial, providing a reproducible and robust framework that complements and enhances existing datasets. By offering an open-source and continuously updating platform, ARVO supports a wide array of research activities, from vulnerability detection to automated program repair.

Moving forward, integrating additional sources of vulnerabilities beyond OSS-Fuzz, such as kernel vulnerabilities, could further broaden ARVO's utility. Enhancements in build reproducibility and dependency management will augment the dataset's robustness. Additionally, extending ARVO's framework to other programming languages and environments will diversify its applicability and research impact.

Conclusion

The ARVO framework represents a significant advancement in the creation and maintenance of reproducible vulnerability datasets. By leveraging automation and precise metadata from sources like OSS-Fuzz, ARVO addresses key limitations of existing datasets and provides an indispensable resource for ongoing research in software security. The dataset's thoroughness in capturing and verifying real-world vulnerabilities underscores its potential to drive future developments in vulnerability analysis and mitigation strategies.