Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Security Defect Detection via Code Review: A Study of the OpenStack and Qt Communities (2307.02326v1)

Published 5 Jul 2023 in cs.SE

Abstract: Background: Despite the widespread use of automated security defect detection tools, software projects still contain many security defects that could result in serious damage. Such tools are largely context-insensitive and may not cover all possible scenarios in testing potential issues, which makes them susceptible to missing complex security defects. Hence, thorough detection entails a synergistic cooperation between these tools and human-intensive detection techniques, including code review. Code review is widely recognized as a crucial and effective practice for identifying security defects. Aim: This work aims to empirically investigate security defect detection through code review. Method: To this end, we conducted an empirical study by analyzing code review comments derived from four projects in the OpenStack and Qt communities. Through manually checking 20,995 review comments obtained by keyword-based search, we identified 614 comments as security-related. Results: Our results show that (1) security defects are not prevalently discussed in code review, (2) more than half of the reviewers provided explicit fixing strategies/solutions to help developers fix security defects, (3) developers tend to follow reviewers' suggestions and action the changes, (4) Not worth fixing the defect now and Disagreement between the developer and the reviewer are the main causes for not resolving security defects. Conclusions: Our research results demonstrate that (1) software security practices should combine manual code review with automated detection tools, achieving a more comprehensive coverage to identifying and addressing security defects, and (2) promoting appropriate standardization of practitioners' behaviors during code review remains necessary for enhancing software security.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. R. Telang and S. Wattal, “An empirical analysis of the impact of software vulnerability announcements on firm stock price,” IEEE Transactions on Software Engineering, vol. 33, no. 8, pp. 544–557, 2007.
  2. H. Cavusoglu, B. Mishra, and S. Raghunathan, “The effect of internet security breach announcements on market value: Capital market reactions for breached firms and internet security developers,” International Journal of Electronic Commerce, vol. 9, no. 1, pp. 70–104, 2004.
  3. E. Iannone, R. Guadagni, F. Ferrucci, A. D. Lucia, and F. Palomba, “The secret life of software vulnerabilities: A large-scale empirical study,” IEEE Transactions on Software Engineering, vol. 49, no. 01, pp. 44–63, 2022.
  4. G. McGraw, J. H. Allen, N. Mead, R. J. Ellison, and S. Barnum, “Software security engineering: A guide for project managers,” tech. rep., CMU/SEI, 2013.
  5. S. Planning, “The economic impacts of inadequate infrastructure for software testing,” National Institute of Standards and Technology, vol. 1, 2002.
  6. M. Alfadel, D. E. Costa, and E. Shihab, “Empirical analysis of security vulnerabilities in python packages,” Empirical Software Engineering, vol. 28, no. 3, p. 59, 2023.
  7. https://about.gitlab.com/developer-survey/#operations.
  8. A. Bosu, J. C. Carver, M. Hafiz, P. Hilley, and D. Janni, “Identifying the characteristics of vulnerable code changes: An empirical study,” in Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering (FSE), pp. 257–268, ACM, 2014.
  9. C. Thompson and D. Wagner, “A large-scale study of modern code review and security in open source projects,” in Proceedings of the 13th International Conference on Predictive Models and Data Analytics in Software Engineering (PROMISE), pp. 83–92, ACM, 2017.
  10. A. Bosu, J. C. Carver, C. Bird, J. Orbeck, and C. Chockley, “Process aspects and social dynamics of contemporary code review: Insights from open source development and industrial practice at microsoft,” IEEE Transactions on Software Engineering, vol. 43, no. 1, pp. 56–75, 2016.
  11. S. McConnell, Code Complete. Pearson Education, 2004.
  12. A. Edmundson, B. Holtkamp, E. Rivera, M. Finifter, A. Mettler, and D. Wagner, “An empirical study on the effectiveness of security code review,” in Proceedings of the 5th International Symposium on Engineering Secure Software and Systems (ESSoS), pp. 197–212, Springer, 2013.
  13. R. Paul, “Improving the effectiveness of peer code review in identifying security defects,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE), pp. 1645–1649, ACM, 2021.
  14. M. di Biase, M. Bruntink, and A. Bacchelli, “A security perspective on code review: The case of chromium,” in Proceedings of the 16th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM), pp. 21–30, IEEE, 2016.
  15. M. Alfadel, N. Nagy, D. Costa, R. Abdalkareem, and E. Shihab, “Qualitative analysis of security-related code reviews in npm packages: An empirical study,” Available at SSRN 4161317, 2022.
  16. B. Chinthanet, R. G. Kula, S. McIntosh, T. Ishio, A. Ihara, and K. Matsumoto, “Lags in the release, adoption, and propagation of npm vulnerability fixes,” Empirical Software Engineering, vol. 26, pp. 1–28, 2021.
  17. J. Lin, H. Zhang, B. Adams, and A. E. Hassan, “Vulnerability management in linux distributions: An empirical study on debian and fedora,” Empirical Software Engineering, vol. 28, no. 2, p. 47, 2023.
  18. F. M. Tudela, J.-R. B. Higuera, J. B. Higuera, J.-A. S. Montalvo, and M. I. Argyros, “On combining static, dynamic and interactive analysis security testing tools to improve owasp top ten security vulnerability detection in web applications,” Applied Sciences, vol. 10, no. 24, p. 9119, 2020.
  19. N. Singh, V. Meherhomji, and B. Chandavarkar, “Automated versus manual approach of web application penetration testing,” in Proceedings of the 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT), pp. 1–6, IEEE, 2020.
  20. L. J. Osterweil, M. Bishop, H. M. Conboy, H. Phan, B. I. Simidchieva, G. S. Avrunin, L. A. Clarke, and S. Peisert, “A comprehensive framework for using iterative analysis to improve human-intensive process security: An election example,” ACM Transactions on Information and System Security, 2017.
  21. R. Paul, A. K. Turzo, and A. Bosu, “Why security defects go unnoticed during code reviews? a case-control study of the chromium os project,” in Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering (ICSE), pp. 1373–1385, IEEE, 2021.
  22. L. Braz, C. Aeberhard, G. Çalikli, and A. Bacchelli, “Less is more: supporting developers in vulnerability detection during code review,” in Proceedings of the 44th International Conference on Software Engineering (ICSE), pp. 1317–1329, ACM, 2022.
  23. A. Bosu and J. C. Carver, “Peer code review to prevent security vulnerabilities: An empirical evaluation,” in Proceedings of the 7th IEEE International Conference on Software Security and Reliability Companion (SERE-C), pp. 229–230, IEEE, 2013.
  24. C. Sadowski, E. Söderberg, L. Church, M. Sipko, and A. Bacchelli, “Modern code review: a case study at google,” in Proceedings of the 40th international conference on software engineering: Software engineering in practice (ICSE), pp. 181–190, ACM, 2018.
  25. S. McIntosh, Y. Kamei, B. Adams, and A. E. Hassan, “The impact of code review coverage and code review participation on software quality: A case study of the qt, vtk, and itk projects,” in Proceedings of the 11th Working Conference on Mining Software Repositories (MSR), pp. 192–201, ACM, 2014.
  26. P. Thongtanunam, S. McIntosh, A. E. Hassan, and H. Iida, “Review participation in modern code review: An empirical study of the android, qt, and openstack projects,” Empirical Software Engineering, vol. 22, pp. 768–817, 2017.
  27. D. Spadini, M. Aniche, M.-A. Storey, M. Bruntink, and A. Bacchelli, “When testing meets code review: Why and how developers review tests,” in Proceedings of the 40th International Conference on Software Engineering (ICSE), pp. 677–687, ACM, 2018.
  28. K. Hamasaki, R. G. Kula, N. Yoshida, A. C. Cruz, K. Fujiwara, and H. Iida, “Who does what during a code review? datasets of oss peer review repositories,” in Proceedings of the 10th Working Conference on Mining Software Repositories (MSR), pp. 49–52, IEEE, 2013.
  29. X. Han, A. Tahir, P. Liang, S. Counsell, K. Blincoe, B. Li, and Y. Luo, “Code smells detection via modern code review: A study of the openstack and qt communities,” Empirical Software Engineering, vol. 27, no. 6, p. 127, 2022.
  30. L. Fu, P. Liang, Z. Rasheed, Z. Li, A. Tahir, and X. Han, “Potential technical debt and its resolution in code reviews: An exploratory study of the openstack and qt communities,” in Proceedings of the 16th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pp. 216–226, ACM, 2022.
  31. T. Hirao, S. McIntosh, A. Ihara, and K. Matsumoto, “Code reviews with divergent review scores: An empirical study of the openstack and qt communities,” IEEE Transactions on Software Engineering, vol. 48, no. 1, pp. 69–81, 2020.
  32. https://cwe.mitre.org/.
  33. O’Reilly Media, 2009.
  34. Pearson Education India, 2016.
  35. J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement, vol. 20, no. 1, pp. 37–46, 1960.
  36. A. Tahir, A. Yamashita, S. Licorish, J. Dietrich, and S. Counsell, “Can you tell me if it smells? a study on how developers discuss code smells and anti-patterns in stack overflow,” in Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering (EASE), pp. 68–78, ACM, 2018.
  37. A. Tahir, J. Dietrich, S. Counsell, S. Licorish, and A. Yamashita, “A large scale study on how developers discuss code smells and anti-pattern in stack exchange sites,” Information and Software Technology, vol. 125, p. 106333, 2020.
  38. B. G. Glaser, “The constant comparative method of qualitative analysis,” Social Problems, vol. 12, no. 4, pp. 436–445, 1965.
  39. J. L. Campbell, C. Quincy, J. Osserman, and O. K. Pedersen, “Coding in-depth semistructured interviews: Problems of unitization and intercoder reliability and agreement,” Sociological Methods & Research, vol. 42, no. 3, pp. 294–320, 2013.
  40. J. Yu, L. Fu, P. Liang, A. Tahir, and M. Shahin, Dataset of the Paper “Security Issue Detection in Code Review: An Exploratory Study of OpenStack and Qt communities”, 2023. https://doi.org/10.5281/zenodo.7886148.
  41. https://owasp.org/www-project-top-ten/.
  42. T. Wu, J. Liu, X. Deng, J. Yan, and J. Zhang, “Relda2: an effective static analysis tool for resource leak detection in android apps,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 762–767, IEEE, 2016.
  43. L. Zhang and C. Wang, “Rclassify: classifying race conditions in web applications via deterministic replay,” in Proceedings of the 39th IEEE/ACM International Conference on Software Engineering (ICSE), pp. 278–288, IEEE, 2017.
  44. E. Shihab, A. Mockus, Y. Kamei, B. Adams, and A. E. Hassan, “High-impact defects: a study of breakage and surprise defects,” in Proceedings of the 19th ACM SIGSOFT Symposium on the Foundations of Software Engineering and the 13th European Software Engineering Conference (ESEC/FSE), pp. 300–310, ACM, 2011.
  45. S. McIntosh, Y. Kamei, B. Adams, and A. E. Hassan, “An empirical study of the impact of modern code review practices on software quality,” Empirical Software Engineering, vol. 21, pp. 2146–2189, 2016.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Jiaxin Yu (16 papers)
  2. Liming Fu (7 papers)
  3. Peng Liang (94 papers)
  4. Amjed Tahir (34 papers)
  5. Mojtaba Shahin (54 papers)
Citations (2)
View on Semantic Scholar