Factoring Expertise, Workload, and Turnover into Code Review Recommendation (2312.17236v1)
Abstract: Developer turnover is inevitable on software projects and leads to knowledge loss, a reduction in productivity, and an increase in defects. Mitigation strategies to deal with turnover tend to disrupt and increase workloads for developers. In this work, we suggest that through code review recommendation we can distribute knowledge and mitigate turnover while more evenly distributing review workload. We conduct historical analyses to understand the natural concentration of review workload and the degree of knowledge spreading that is inherent in code review. Even though review workload is highly concentrated, we show that code review natural spreads knowledge thereby reducing the files at risk to turnover. Using simulation, we evaluate existing code review recommenders and develop novel recommenders to understand their impact on the level of expertise during review, the workload of reviewers, and the files at risk to turnover. Our simulations use seeded random replacement of reviewers to allow us to compare the reviewer recommenders without the confounding variation of different reviewers being replaced for each recommender. Combining recommenders, we develop the SofiaWL recommender that suggests experts with low active review workload when none of the files under review are known by only one developer. In contrast, when knowledge is concentrated on one developer, it sends the review to other reviewers to spread knowledge. For the projects we study, we are able to globally increase expertise during reviews, +3%, reduce workload concentration, -12%, and reduce the files at risk, -28%. We make our scripts and data available in our replication package. Developers can optimize for a particular outcome measure based on the needs of their project, or use our GitHub bot to automatically balance the outcomes.
- F. Hajari, E. Mirsaeedi, and P. C. Rigby, “Replication package and relationalgit,” https://github.com/fahimeh1368/RelationalGit/tree/master/ReplicationPackage, 2021.
- E. Mirsaeedi and P. C. Rigby, “Mitigating turnover with code review recommendation: Balancing expertise, workload, and knowledge distribution,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, ser. ICSE ’20. New York, NY, USA: Association for Computing Machinery, 2020, p. 1183–1195. [Online]. Available: https://doi.org/10.1145/3377811.3380335
- L. Bao, Z. Xing, X. Xia, D. Lo, and S. Li, “Who will leave the company?: a large-scale industry study of developer turnover by mining monthly work report,” in 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR). IEEE, 2017, pp. 170–181.
- P. C. Rigby, Y. C. Zhu, S. M. Donadelli, and A. Mockus, “Quantifying and mitigating turnover-induced knowledge loss: Case studies of chrome and a project at avaya,” in 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), May 2016, pp. 1006–1016.
- N. Pekala, “Holding on to top talent,” Journal of Property management, vol. 66, no. 5, pp. 22–22, 2001.
- A. Mockus, “Organizational volatility and its effects on software defects,” in Proceedings of the eighteenth ACM SIGSOFT international symposium on Foundations of software engineering. ACM, 2010, pp. 117–126.
- D. Izquierdo-Cortazar, G. Robles, F. Ortega, and J. M. Gonzalez-Barahona, “Using software archaeology to measure knowledge loss in software projects due to developer turnover,” in 2009 42nd Hawaii International Conference on System Sciences. IEEE, 2009, pp. 1–10.
- A. Mockus, “Succession: Measuring transfer of code and developer productivity,” in Proceedings of the 31st International Conference on Software Engineering. IEEE Computer Society, 2009, pp. 67–77.
- M. A. Huselid, “The impact of human resource management practices on turnover, productivity, and corporate financial performance,” Academy of management journal, vol. 38, no. 3, pp. 635–672, 1995.
- M. Foucault, M. Palyart, X. Blanc, G. C. Murphy, and J.-R. Falleri, “Impact of developer turnover on quality in open-source software,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, 2015, pp. 829–841.
- B. Lin, G. Robles, and A. Serebrenik, “Developer turnover in global, industrial open source projects: Insights from applying survival analysis,” in 2017 IEEE 12th International Conference on Global Software Engineering (ICGSE). IEEE, 2017, pp. 66–75.
- E. Constantinou and T. Mens, “An empirical comparison of developer retention in the rubygems and npm software ecosystems,” Innovations in Systems and Software Engineering, vol. 13, no. 2-3, pp. 101–115, 2017.
- M. Stovel and N. Bontis, “Voluntary turnover: knowledge management–friend or foe?” Journal of intellectual Capital, vol. 3, no. 3, pp. 303–322, 2002.
- L. G. Pee, A. Kankanhalli, G. W. Tan, and G. Tham, “Mitigating the impact of member turnover in information systems development projects,” IEEE Transactions on Engineering Management, vol. 61, no. 4, pp. 702–716, 2014.
- M. Rashid, P. M. Clarke, and R. V. O’Connor, “Exploring knowledge loss in open source software (oss) projects,” in International conference on software process improvement and capability determination. Springer, 2017, pp. 481–495.
- A. Bacchelli and C. Bird, “Expectations, outcomes, and challenges of modern code review,” in Proceedings of the 2013 international conference on software engineering. IEEE Press, 2013, pp. 712–721.
- M. Greiler, C. Bird, M.-A. Storey, L. MacLeod, and J. Czerwonka, “Code reviewing in the trenches: Understanding challenges, best practices and tool needs,” 2016.
- A. Bosu, J. C. Carver, C. Bird, J. Orbeck, and C. Chockley, “Process aspects and social dynamics of contemporary code review: Insights from open source development and industrial practice at microsoft,” IEEE Transactions on Software Engineering, vol. 43, no. 1, pp. 56–75, 2016.
- P. Rigby, B. Cleary, F. Painchaud, M.-A. Storey, and D. German, “Contemporary peer review in action: Lessons from open source development,” IEEE software, vol. 29, no. 6, pp. 56–61, 2012.
- C. Sadowski, E. Söderberg, L. Church, M. Sipko, and A. Bacchelli, “Modern code review: a case study at google,” in Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice. ACM, 2018, pp. 181–190.
- P. Thongtanunam, S. McIntosh, A. E. Hassan, and H. Iida, “Revisiting code ownership and its relationship with software quality in the scope of modern code review,” in Proceedings of the 38th international conference on software engineering. ACM, 2016, pp. 1039–1050.
- X. Xia, D. Lo, X. Wang, and X. Yang, “Who should review this change?: Putting text and file location analyses together for more accurate recommendations,” in 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2015, pp. 261–270.
- Y. Yu, H. Wang, G. Yin, and T. Wang, “Reviewer recommendation for pull-requests in github: What can we learn from code review and bug assignment?” Information and Software Technology, vol. 74, pp. 204–218, 2016.
- M. M. Rahman, C. K. Roy, and J. A. Collins, “Correct: code reviewer recommendation in github based on cross-project and technology experience,” in 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C). IEEE, 2016, pp. 222–231.
- V. Kovalenko, N. Tintarev, E. Pasynkov, C. Bird, and A. Bacchelli, “Does reviewer recommendation help developers?” IEEE Transactions on Software Engineering, 2018.
- P. C. Rigby, D. M. German, L. Cowen, and M.-A. Storey, “Peer review on open-source software projects: Parameters, statistical models, and theory,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 23, no. 4, p. 35, 2014.
- O. Kononenko, O. Baysal, and M. W. Godfrey, “Code review quality: how developers see it,” in Proceedings of the 38th International Conference on Software Engineering, 2016, pp. 1028–1038.
- S. Asthana, R. Kumar, R. Bhagwan, C. Bird, C. Bansal, C. Maddila, S. Mehta, and B. Ashok, “Whodo: Automating reviewer suggestions at scale,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2019, pp. 937–945.
- C. Bird, N. Nagappan, B. Murphy, H. Gall, and P. Devanbu, “Don’t touch my code!: examining the effects of ownership on software quality,” in Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. ACM, 2011, pp. 4–14.
- F. Rahman and P. Devanbu, “Ownership, experience and defects: a fine-grained study of authorship,” in Proceedings of the 33rd International Conference on Software Engineering. ACM, 2011, pp. 491–500.
- A. Mockus and J. D. Herbsleb, “Expertise browser: a quantitative approach to identifying expertise,” in Proceedings of the 24th International Conference on Software Engineering. ICSE 2002. IEEE, 2002, pp. 503–512.
- H. Kagdi, M. Hammad, and J. I. Maletic, “Who can help me with this source code change?” in 2008 IEEE International Conference on Software Maintenance. IEEE, 2008, pp. 157–166.
- T. Fritz, G. C. Murphy, and E. Hill, “Does a programmer’s activity indicate knowledge of code?” in Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering. ACM, 2007, pp. 341–350.
- T. Girba, A. Kuhn, M. Seeberger, and S. Ducasse, “How developers drive software evolution,” in Eighth International Workshop on Principles of Software Evolution (IWPSE’05). IEEE, 2005, pp. 113–122.
- M. B. Zanjani, H. Kagdi, and C. Bird, “Automatically recommending peer reviewers in modern code review,” IEEE Trans. Softw. Eng., vol. 42, no. 6, pp. 530–543, Jun. 2016. [Online]. Available: https://doi.org/10.1109/TSE.2015.2500238
- P. Thongtanunam, C. Tantithamthavorn, R. G. Kula, N. Yoshida, H. Iida, and K.-i. Matsumoto, “Who should review my code? a file location-based code-reviewer recommendation approach for modern code review,” in 2015 IEEE 22nd International Conference on Software Analysis, Evolution, and Reengineering (SANER). IEEE, 2015, pp. 141–150.
- P. C. Rigby and C. Bird, “Convergent contemporary software peer review practices,” in Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering. ACM, 2013, pp. 202–212.
- P. N. Sharma, J. Hulland, and S. Daniel, “Examining turnover in open source software projects using logistic hierarchical linear modeling approach,” in IFIP International Conference on Open Source Systems. Springer, 2012, pp. 331–337.
- M. Zhou and A. Mockus, “Developer fluency: Achieving true mastery in software projects,” in Proceedings of the Eighteenth ACM SIGSOFT International Symposium on Foundations of Software Engineering, ser. FSE ’10. New York, NY, USA: Association for Computing Machinery, 2010, p. 137–146. [Online]. Available: https://doi.org/10.1145/1882291.1882313
- V. Balachandran, “Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation,” in Proceedings of the 2013 International Conference on Software Engineering. IEEE Press, 2013, pp. 931–940.
- M. Nassif and M. P. Robillard, “Revisiting turnover-induced knowledge loss in software projects,” in 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2017, pp. 261–272.
- M. E. Fagan, “Design and Code Inspections to Reduce Errors in Program Development,” IBM Systems Journal, vol. 15, no. 3, pp. 182–211, 1976.
- A. Bosu, M. Greiler, and C. Bird, “Characteristics of useful code reviews: An empirical study at microsoft,” in 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 2015, pp. 146–156.
- A. Mockus, R. T. Fielding, and J. Herbsleb, “A case study of open source software development: The apache server,” in Proceedings of the 22nd International Conference on Software Engineering, ser. ICSE ’00. New York, NY, USA: Association for Computing Machinery, 2000, p. 263–272. [Online]. Available: https://doi.org/10.1145/337180.337209
- G. Gousios, “The ghtorent dataset and tool suite,” in Proceedings of the 10th working conference on mining software repositories. IEEE Press, 2013, pp. 233–236.
- G. Gousios, M. Pinzger, and A. v. Deursen, “An exploratory study of the pull-based software development model,” in Proceedings of the 36th International Conference on Software Engineering. ACM, 2014, pp. 345–355.
- C. Bird, A. Gourley, P. Devanbu, M. Gertz, and A. Swaminathan, “Mining email social networks,” in Proceedings of the 2006 international workshop on Mining software repositories. ACM, 2006, pp. 137–143.
- G. Canfora, M. Di Penta, R. Oliveto, and S. Panichella, “Who is going to mentor newcomers in open source projects?” in Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering. ACM, 2012, p. 44.
- M. P. Robillard, M. Nassif, and S. McIntosh, “Threats of aggregating software repository data,” in 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2018, pp. 508–518.
- D. W. McDonald and M. S. Ackerman, “Expertise recommender: a flexible recommendation system and architecture,” in Proceedings of the 2000 ACM conference on Computer supported cooperative work. ACM, 2000, pp. 231–240.
- C. Hannebauer, M. Patalas, S. Stünkel, and V. Gruhn, “Automatically recommending code reviewers based on their expertise: An empirical comparison,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. ACM, 2016, pp. 99–110.
- G. Jeong, S. Kim, T. Zimmermann, and K. Yi, “Improving code review by predicting reviewers and acceptance of patches,” Research on software analysis for error-free computing center Tech-Memo (ROSAEC MEMO 2009-006), pp. 1–18, 2009.
- M. Fagan, “Design and code inspections to reduce errors in program development,” in Software pioneers. Springer, 2002, pp. 575–607.
- A. Porter, H. Siy, A. Mockus, and L. Votta, “Understanding the sources of variation in software inspections,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 7, no. 1, pp. 41–79, 1998.
- P. C. Rigby and M.-A. Storey, “Understanding broadcast based peer review on open source software projects,” in 2011 33rd International Conference on Software Engineering (ICSE). IEEE, 2011, pp. 541–550.
- S. McIntosh, Y. Kamei, B. Adams, and A. E. Hassan, “An Empirical Study of the Impact of Modern Code Review Practices on Software Quality,” Empirical Software Engineering, vol. 21, no. 5, pp. 2146–2189, 2016.
- A. Krutauz, T. Dey, P. C. Rigby, and A. Mockus, “Do code review measures explain the incidence of post-release defects?” Empirical Software Engineering, vol. 25, no. 5, pp. 3323–3356, 2020.
- P. Thongtanunam, S. McIntosh, A. E. Hassan, and H. Iida, “Review Participation in Modern Code Review: An Empirical Study of the Android, Qt, and OpenStack Projects,” Empirical Software Engineering, vol. 22, no. 2, pp. 768–817, 2017.
- Z. Ton and R. S. Huckman, “Managing the impact of employee turnover on performance: The role of process conformance,” Organization Science, vol. 19, no. 1, pp. 56–68, 2008.
- A. Chueshev, J. Lawall, R. Bendraou, and T. Ziadi, “Expanding the number of reviewers in open-source projects by recommending appropriate developers,” in 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), 2020, pp. 499–510.