An Empirical Study of Static Analysis Tools for Secure Code Review (2407.12241v1)

Abstract: Early identification of security issues in software development is vital to minimize their unanticipated impacts. Code review is a widely used manual analysis method that aims to uncover security issues along with other coding issues in software projects. While some studies suggest that automated static application security testing tools (SASTs) could enhance security issue identification, there is limited understanding of SAST's practical effectiveness in supporting secure code review. Moreover, most SAST studies rely on synthetic or fully vulnerable versions of the subject program, which may not accurately represent real-world code changes in the code review process. To address this gap, we study C/C++ SASTs using a dataset of actual code changes that contributed to exploitable vulnerabilities. Beyond SAST's effectiveness, we quantify potential benefits when changed functions are prioritized by SAST warnings. Our dataset comprises 319 real-world vulnerabilities from 815 vulnerability-contributing commits (VCCs) in 92 C and C++ projects. The result reveals that a single SAST can produce warnings in vulnerable functions of 52% of VCCs. Prioritizing changed functions with SAST warnings can improve accuracy (i.e., 12% of precision and 5.6% of recall) and reduce Initial False Alarm (lines of code in non-vulnerable functions inspected until the first vulnerable function) by 13%. Nevertheless, at least 76% of the warnings in vulnerable functions are irrelevant to the VCCs, and 22% of VCCs remain undetected due to limitations of SAST rules. Our findings highlight the benefits and the remaining gaps of SAST-supported secure code reviews and challenges that should be addressed in future work.

An Empirical Study of Static Analysis Tools for Secure Code Review

In the field of software security, identifying vulnerabilities early in the software development process is crucial to minimizing potential impacts. Secure code reviews are a traditional method for detecting security issues but can be resource-intensive and rely heavily on the expertise of the reviewers. Automated Static Application Security Testing Tools (SASTs) have the potential to augment this process by providing automated insights into security vulnerabilities. This paper provides a critical evaluation of SASTs, focusing on their application within the secure code review process for C and C++ projects.

Study Objectives and Methodology

The research presented in this paper examines the effectiveness of SASTs in identifying vulnerabilities through actual code changes that have contributed to exploitable vulnerabilities. The researchers used a dataset consisting of 319 real-world vulnerabilities from 815 vulnerability-contributing commits (VCCs) across 92 C and C++ projects. The paper explored three main aspects: the effectiveness of SAST tools in detecting vulnerabilities, the benefits of using SASTs to prioritize code changes during reviews, and the computation time required for SAST analysis.

Key Findings

1. Effectiveness of SASTs: The paper found that Flawfinder was capable of producing warnings for vulnerable functions in 52% of the VCCs, highlighting the potential of SASTs in early vulnerability detection. However, it was noted that SASTs often produced irrelevant warnings, with 76% of warnings in vulnerable functions being unrelated to the actual vulnerabilities. Moreover, 22% of VCCs received no warnings at all, pointing to limitations in existing SAST rules.
2. Tool Combination: The results indicated that employing multiple SASTs could improve detection rates significantly. By combining the tools analyzed, the detection rates increased by an additional 26 percentage points, illustrating the benefits of a multi-tool approach to security analysis.
3. Prioritization and Performance: The paper tested different strategies for prioritizing code reviews based on SAST warnings. It was found that using CodeQL warnings improved precision by up to 12% and reduced Initial False Alarms by 13% at equivalent code review efforts. This suggests that incorporating SASTs into the code review process can improve security-focused code inspection efficiency.
4. Computation Time: SAST computation times were found to be variable, ranging from 20 seconds to 45 minutes on average, depending on the tool and the size of the project. This is an important consideration for integrating these tools into real-world development environments where waiting times could affect workflow efficiency.

Implications and Future Directions

The findings of this paper present several implications for practitioners and researchers. For practitioners, the results suggest that while SASTs can assist in secure code reviews, care must be taken in choosing the right tools and strategies to maximize benefits. This includes selecting tools that align with specific security needs and project constraints, and considering tool combinations to optimize detection efficacy.

For SAST developers, there is an opportunity to improve the accuracy and relevance of tool warnings and enhance the prioritization algorithms to better focus reviewer efforts. Additionally, expanding tool coverage to reduce the number of undetected vulnerabilities remains a critical area for future work.

For researchers, this paper opens up several avenues for investigation. This includes developing improved strategies for mitigating false positives, exploring the integration of SASTs with other automated tools to enhance security review processes, and advancing the scalability and efficiency of SASTs in handling large codebases.

Overall, while SASTs provide valuable assistance in secure code reviews, their current limitations underline the need for ongoing refinement and research to fully realize their potential in safeguarding software systems.