Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Leveraging the Crowd for Dependency Management: An Empirical Study on the Dependabot Compatibility Score (2403.09012v1)

Published 14 Mar 2024 in cs.SE

Abstract: Dependabot, a popular dependency management tool, includes a compatibility score feature that helps client packages assess the risk of accepting a dependency update by leveraging knowledge from "the crowd". For each dependency update, Dependabot calculates this compatibility score as the proportion of successful updates performed by other client packages that use the same provider package as a dependency. In this paper, we study the efficacy of the compatibility score to help client packages assess the risks involved with accepting a dependency update. We analyze 579,206 pull requests opened by Dependabot to update a dependency, along with 618,045 compatibility score records calculated by Dependabot. We find that a compatibility score cannot be calculated for 83% of the dependency updates due to the lack of data from the crowd. Yet, the vast majority of the scores that can be calculated have a small confidence interval and are based on low-quality data, suggesting that client packages should have additional angles to evaluate the risk of an update and the trustworthiness of the compatibility score. To overcome these limitations, we propose metrics that amplify the input from the crowd and demonstrate the ability of those metrics to predict the acceptance of a successful update by client packages. We also demonstrate that historical update metrics from client packages can be used to provide a more personalized compatibility score. Based on our findings, we argue that, when leveraging the crowd, dependency management bots should include a confidence interval to help calibrate the trust clients can place in the compatibility score, and consider the quality of tests that exercise candidate updates.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. R. Abdalkareem, O. Nourry, S. Wehaibi, S. Mujahid, and E. Shihab, “Why do developers use trivial packages? an empirical case study on npm,” in Proceedings of the 11th Joint Meeting on Foundations of Software Engineering, 2017, pp. 385–395.
  2. R. Abdalkareem, V. Oda, S. Mujahid, and E. Shihab, “On the impact of using trivial packages: an empirical case study on npm and PyPI,” Empirical Software Engineering, vol. 25, no. 2, pp. 1168–1204, 2020.
  3. S. Mirhosseini and C. Parnin, “Can automated pull requests encourage software developers to upgrade out-of-date dependencies?” in Proceedings of the 32nd IEEE/ACM International Conference on Automated Software Engineering, 2017, pp. 84–94.
  4. A. Decan, T. Mens, and E. Constantinou, “On the impact of security vulnerabilities in the npm package dependency network,” in Proceedings of the 15th International Conference on Mining Software Repositories, 2018, pp. 181–191.
  5. C. Bogart, C. Kästner, J. Herbsleb, and F. Thung, “How to break an API: cost negotiation and community values in three software ecosystems,” in Proceedings of the 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2016, pp. 109–120.
  6. M. Hilton, T. Tunnell, K. Huang, D. Marinov, and D. Dig, “Usage, costs, and benefits of continuous integration in open-source projects,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, 2016, pp. 426–437.
  7. S. Raemaekers, A. van Deursen, and J. Visser, “Semantic versioning and impact of breaking changes in the Maven repository,” Journal of Systems and Software, vol. 129, pp. 140–158, 2017.
  8. S. Mujahid, R. Abdalkareem, E. Shihab, and S. McIntosh, “Using Others’ Tests to Identify Breaking Updates,” in Proceedings of the 17th International Conference on Mining Software Repositories, 2020, pp. 466–476.
  9. G. Mezzetti, A. Møller, and M. T. Torp, “Type Regression Testing to Detect Breaking Changes in Node.js Libraries,” in Proceedings of the 32nd European Conference on Object-Oriented Programming, 2018, p. 24 pages.
  10. E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German, and D. Damian, “The promises and perils of mining GitHub,” in Proceedings of the 11th Working Conference on Mining Software Repositories, 2014, pp. 92–101.
  11. M. Alfadel, D. E. Costa, E. Shihab, and M. Mkhallalati, “On the Use of Dependabot Security Pull Requests,” in Proceedings of the IEEE/ACM 18th International Conference on Mining Software Repositories, 2021, pp. 254–265.
  12. J. Hejderup and G. Gousios, “Can we trust tests to automate dependency updates? A case study of java projects,” Journal of Systems and Software, p. 111097, 2021.
  13. C. Bogart, C. Kastner, and J. Herbsleb, “When It Breaks, It Breaks: How Ecosystem Developers Reason about the Stability of Dependencies,” in Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering Workshop, 2015, pp. 86–89.
  14. J. Dietrich, D. Pearce, J. Stringer, A. Tahir, and K. Blincoe, “Dependency Versioning in the Wild,” in Proceedings of the IEEE/ACM 16th International Conference on Mining Software Repositories, 2019, pp. 349–359.
  15. A. Decan and T. Mens, “What do package dependencies tell us about semantic versioning?” IEEE Transactions on Software Engineering, pp. 1–1, 2020.
  16. C. Tantithamthavorn, S. McIntosh, A. E. Hassan, and K. Matsumoto, “An Empirical Comparison of Model Validation Techniques for Defect Prediction Models,” IEEE Transactions on Software Engineering, vol. 43, no. 1, pp. 1–18, 2017.
  17. D. Lee, G. K. Rajbahadur, D. Lin, M. Sayagh, C.-P. Bezemer, and A. E. Hassan, “An empirical study of the characteristics of popular Minecraft mods,” Empirical Software Engineering, vol. 25, no. 5, pp. 3396–3429, 2020.
  18. A. Hazra, “Using the confidence interval confidently,” Journal of Thoracic Disease, vol. 9, no. 10, pp. 4125–4130, 2017.
  19. T. Kulesza, S. Stumpf, M. Burnett, and I. Kwan, “Tell me more? the effects of mental model soundness on personalizing an intelligent agent,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2012, pp. 1–10.
  20. Y. Zhang, Q. V. Liao, and R. K. E. Bellamy, “Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making,” in Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 2020, pp. 295–305.
  21. E. Wittern, P. Suter, and S. Rajagopalan, “A look at the dynamics of the JavaScript package ecosystem,” in Proceedings of the 13th International Conference on Mining Software Repositories, 2016, pp. 351–361.
  22. A. M. Fard and A. Mesbah, “JavaScript: The (Un)Covered Parts,” in Proceedings of the 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST), 2017, pp. 230–240.
  23. M. A. R. Chowdhury, R. Abdalkareem, E. Shihab, and B. Adams, “On the Untriviality of Trivial Packages: An Empirical Study of npm JavaScript Packages,” IEEE Transactions on Software Engineering, pp. 1–1, 2021.
  24. A. Decan, T. Mens, and E. Constantinou, “On the Evolution of Technical Lag in the npm Package Dependency Network,” in Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME), 2018, pp. 404–414.
  25. J. M. Gonzalez-Barahona, P. Sherwood, G. Robles, and D. Izquierdo, “Technical Lag in Software Compilations: Measuring How Outdated a Software Deployment Is,” in Open Source Systems: Towards Robust Practices.   Springer International Publishing, 2017, vol. 496, pp. 182–192.
  26. A. Zerouali, E. Constantinou, T. Mens, G. Robles, and J. González-Barahona, “An Empirical Analysis of Technical Lag in npm Package Dependencies,” in New Opportunities for Software Reuse.   Springer International Publishing, 2018, vol. 10826, pp. 95–110.
  27. J. Cox, E. Bouwers, M. v. Eekelen, and J. Visser, “Measuring Dependency Freshness in Software Systems,” in Proceedings of the IEEE/ACM 37th IEEE International Conference on Software Engineering, 2015, pp. 109–118.
  28. F. R. Cogo, G. A. Oliva, and A. E. Hassan, “Deprecation of packages and releases in software ecosystems: A case study on npm,” IEEE Transactions on Software Engineering, pp. 1–1, 2021.
  29. A. Decan, T. Mens, and M. Claes, “An empirical comparison of dependency issues in OSS packaging ecosystems,” in Proceedings of the IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2017, pp. 2–12.
  30. A. Brito, L. Xavier, A. Hora, and M. T. Valente, “APIDiff: Detecting API breaking changes,” in Proceedings of the IEEE 25th International Conference on Software Analysis, Evolution and Reengineering, 2018, pp. 507–511.
  31. A. Møller and M. T. Torp, “Model-based testing of breaking changes in Node.js libraries,” in Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2019, pp. 409–419.
  32. L. Erlenhov, F. Gomes de Oliveira Neto, R. Scandariato, and P. Leitner, “Current and Future Bots in Software Development,” in Proceedings of the IEEE/ACM 1st International Workshop on Bots in Software Engineering, 2019, pp. 7–11.
  33. C. Lebeuf, A. Zagalsky, M. Foucault, and M.-A. Storey, “Defining and Classifying Software Bots: A Faceted Taxonomy,” in Proceedings of the IEEE/ACM 1st International Workshop on Bots in Software Engineering, 2019, pp. 1–6.
  34. M. Wessel, “Enhancing developers’ support on pull requests activities with software bots,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020, pp. 1674–1677.
  35. M. Wessel and I. Steinmacher, “The Inconvenient Side of Software Bots on Pull Requests,” in Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops, 2020, pp. 51–55.
  36. T. D. LaToza and A. van der Hoek, “Crowdsourcing in Software Engineering: Models, Motivations, and Challenges,” IEEE Software, vol. 33, no. 1, pp. 74–80, 2016.
  37. C. Treude, O. Barzilay, and M.-A. Storey, “How do programmers ask and answer questions on the web?: NIER track,” in Proceedings of the 33rd International Conference on Software Engineering, 2011, pp. 804–807.
  38. C. Rosen and E. Shihab, “What are mobile developers asking about? A large scale study using stack overflow,” Empirical Software Engineering, vol. 21, 2015.
  39. A. Barua, S. W. Thomas, and A. E. Hassan, “What are developers talking about? An analysis of topics and trends in Stack Overflow,” Empirical Software Engineering, vol. 19, no. 3, pp. 619–654, 2014.
  40. B. Vasilescu, V. Filkov, and A. Serebrenik, “StackOverflow and GitHub: Associations between Software Development and Crowdsourced Knowledge,” in Proceedings of the International Conference on Social Computing, 2013, pp. 188–195.
  41. R. Abdalkareem, E. Shihab, and J. Rilling, “What Do Developers Use the Crowd For? A Study Using Stack Overflow,” IEEE Software, vol. 34, no. 2, pp. 53–60, 2017.
  42. Y. M. Mileva, V. Dallmeier, M. Burger, and A. Zeller, “Mining trends of library usage,” in Proceedings of the joint international and annual ERCIM workshops on Principles of software evolution (IWPSE) and software evolution (Evol) workshops, 2009, pp. 57–62.
  43. L. Hespanhol, C. S. Vallio, L. M. Costa, and B. T. Saragiotto, “Understanding and interpreting confidence and credible intervals around effect estimates,” Brazilian Journal of Physical Therapy, vol. 23, no. 4, pp. 290–301, 2019.
  44. A. K. Gupta, “Beta Distribution,” in International Encyclopedia of Statistical Science, M. Lovric, Ed.   Berlin, Heidelberg: Springer Berlin Heidelberg, 2011, pp. 144–145.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets