Process-based Indicators of Vulnerability Re-Introducing Code Changes: An Exploratory Case Study

Abstract: Software vulnerabilities often persist or re-emerge even after being fixed, revealing the complex interplay between code evolution and socio-technical factors. While source code metrics provide useful indicators of vulnerabilities, software engineering process metrics can uncover patterns that lead to their introduction. Yet few studies have explored whether process metrics can reveal risky development activities over time -- insights that are essential for anticipating and mitigating software vulnerabilities. This work highlights the critical role of process metrics along with code changes in understanding and mitigating vulnerability reintroduction. We move beyond file-level prediction and instead analyze security fixes at the commit level, focusing not only on whether a single fix introduces a vulnerability but also on the longer sequences of changes through which vulnerabilities evolve and re-emerge. Our approach emphasizes that reintroduction is rarely the result of one isolated action, but emerges from cumulative development activities and socio-technical conditions. To support this analysis, we conducted a case study on the ImageMagick project by correlating longitudinal process metrics such as bus factor, issue density, and issue spoilage with vulnerability reintroduction activities, encompassing 76 instances of reintroduced vulnerabilities. Our findings show that reintroductions often align with increased issue spoilage and fluctuating issue density, reflecting short-term inefficiencies in issue management and team responsiveness. These observations provide a foundation for broader studies that combine process and code metrics to predict risky fixes and strengthen software security.

• The paper demonstrates a novel approach to identify vulnerability reintroducing commits by combining the SZZ algorithm with LLM-based validation in the ImageMagick project.
• The study employs longitudinal analysis of SEP metrics—including bus factor, issue density, and spoilage—to highlight temporal risk patterns in vulnerability management.
• The findings suggest that process-based indicators can alert maintainers to elevated risk periods, informing the development of predictive and preventive security tools.

Process-Based Indicators of Vulnerability Re-Introducing Code Changes: An Exploratory Case Study

Introduction

This paper investigates the relationship between software engineering process (SEP) metrics and the phenomenon of vulnerability reintroduction in open-source software, focusing on the ImageMagick project. The study moves beyond traditional file-level vulnerability prediction by analyzing commit-level security fixes and their subsequent evolution, emphasizing that vulnerability reintroduction is typically a cumulative result of ongoing development activities and socio-technical factors rather than isolated events. The authors introduce a methodology for identifying vulnerability reintroducing commits, leveraging both established algorithms and LLM-based validation, and correlate these events with longitudinal process metrics such as bus factor, issue density, and issue spoilage.

Figure 1: Overview of the vulnerability reintroduction process, illustrating the cyclical nature of vulnerability introduction, detection, remediation, and potential reintroduction in open-source projects.

Methodology

The study constructs a dataset of 76 vulnerability reintroducing commit pairs in ImageMagick by extending existing vulnerability datasets (BigVul, DiverseVul) and applying a combination of the SZZ algorithm and LLM-based (Anthropic Claude 3.5 Sonnet) validation to confirm true reintroduction events. The methodology involves:

• Extracting vulnerability-fixing commits from public datasets.
• Identifying subsequent commits that modify the same or adjacent code regions using SZZ.
• Employing LLMs to validate whether the subsequent commit indeed fixes a new vulnerability introduced by the previous fix.
• Computing longitudinal SEP metrics (bus factor, issue density, issue spoilage) using the PRIME tool, with granularity at both project and CVE-specific levels.

  Figure 2: Methodology for extracting and computing longitudinal SEP metrics for vulnerability reintroducing and fixing commits in ImageMagick.

Dataset and Case Selection

From an initial pool of 175 (BigVul) and 225 (DiverseVul) commits, 81 confirmed vulnerability reintroducing-fixing commit pairs were identified, with 76 associated with CVEs. Four representative CVEs (CVE-2016-4564, CVE-2017-16546, CVE-2018-11625, CVE-2019-13299) were selected for detailed longitudinal analysis based on severity (CVSS ≥ 8.8), non-overlapping timeframes, and unique annual occurrence.

Longitudinal Analysis of Process Metrics

Bus Factor

The bus factor, computed every six months, remained consistently between 4 and 7, indicating a distributed contributor base capable of handling vulnerability resolution and review. This distributed knowledge base reduces the risk of single points of failure but does not eliminate the possibility of vulnerability reintroduction, as any contributor may inadvertently introduce regressions.

Figure 3: ImageMagick's bus factor over time, showing sustained contributions from 4–7 unique maintainers per six-month period.

Issue Density

Issue density, defined as the ratio of open issues to project size, peaked at 28% in 2022 but declined to 16.4% subsequently. The analysis of the four selected CVEs revealed three distinct trajectories:

• Declining stability (CVE-2016-4564): Issue density decreased after the risky fix, indicating improved triage.
• Fluctuating recovery (CVE-2017-16546, CVE-2018-11625): Issue density oscillated due to concurrent code refactoring and issue handling.
• Persistent growth (CVE-2019-13299): Issue density continuously increased, reflecting sustained issue accumulation.

These patterns suggest that vulnerability reintroduction often coincides with temporary inefficiencies in issue management, with the project’s capacity to balance triage and development fluctuating during such periods.

Figure 4: Longitudinal issue density for ImageMagick and selected CVEs, highlighting periods of increased density during vulnerability reintroduction events.

Issue Spoilage

Issue spoilage, measuring the age of unresolved issues, peaked at 74,657 days in 2023 but dropped to 25 days at the time of writing. The four CVEs exhibited:

• Reactive escalation (CVE-2016-4564): Spoilage increased after the risky fix, then decreased post-remediation.
• Aggressive velocity (CVE-2017-16546, CVE-2018-11625): Spoilage declined after the risky fix, indicating active issue resolution.
• Sustained degradation (CVE-2019-13299): Spoilage continuously increased, reflecting inability to keep up with issue closure.

These findings indicate that periods of vulnerability reintroduction are associated with increased issue spoilage, suggesting that remediation efforts may temporarily divert resources from regular issue management.

Figure 5: Longitudinal issue spoilage for ImageMagick and selected CVEs, showing peaks in spoilage during vulnerability reintroduction and subsequent remediation.

Implications and Future Directions

The study demonstrates that longitudinal SEP metrics provide actionable insight into the socio-technical conditions under which vulnerabilities are reintroduced. The observed correlation between increased issue spoilage, fluctuating issue density, and vulnerability reintroduction highlights the need for process-aware tooling and decision support systems that can alert maintainers to periods of elevated risk.

Future research directions include:

• Generalization: Extending the methodology to a broader set of open-source projects across languages and domains to assess the universality of the observed correlations.
• Predictive Modeling: Integrating SEP metrics with code-level features and commit metadata to develop classifiers for predicting risky fixes at commit time.
• Pull Request Dynamics: Analyzing review process metrics (e.g., review latency, reviewer diversity) to understand their impact on vulnerability reintroduction risk.

Limitations

The study is limited by its focus on a single project (ImageMagick) and a relatively small sample of confirmed reintroduction events. The bus factor metric does not account for contributor specialization at the file or directory level. The reliance on public vulnerability disclosures and LLM-based validation introduces potential biases and may not generalize to projects with different disclosure practices or development cultures.

Conclusion

This work establishes the value of commit-level, process-based analysis for understanding and mitigating vulnerability reintroduction in open-source software. The findings underscore that vulnerability reintroduction is a systemic phenomenon, emerging from the interplay of development activity, team dynamics, and process bottlenecks. Longitudinal SEP metrics such as bus factor, issue density, and issue spoilage serve as early indicators of periods of elevated risk, providing a foundation for the development of predictive and preventive tools to enhance software security resilience.