A Systematically Empirical Evaluation of Vulnerability Discovery Models: a Study on Browsers' Vulnerabilities

Abstract: A precise vulnerability discovery model (VDM) will provide a useful insight to assess software security, and could be a good prediction instrument for both software vendors and users to understand security trends and plan ahead patching schedule accordingly. Thus far, several models have been proposed and validated. Yet, no systematically independent validation by somebody other than the author exists. Furthermore, there are a number of issues that might bias previous studies in the field. In this work, we fill in the gap by introducing an empirical methodology that systematically evaluates the performance of a VDM in two aspects: quality and predictability. We further apply this methodology to assess existing VDMs. The results show that some models should be rejected outright, while some others might be adequate to capture the discovery process of vulnerabilities. We also consider different usage scenarios of VDMs and find that the simplest linear model is the most appropriate choice in terms of both quality and predictability when browsers are young. Otherwise, logistics-based models are better choices.