Decade-long Utilization Patterns of ICSE Technical Papers and Associated Artifacts (2404.05826v1)
Abstract: Context: Annually, ICSE acknowledges a range of papers, a subset of which are paired with research artifacts such as source code, datasets, and supplementary materials, adhering to the Open Science Policy. However, no prior systematic inquiry dives into gauging the influence of ICSE papers using artifact attributes. Objective: We explore the mutual impact between artifacts and their associated papers presented at ICSE over ten years. Method: We collect data on usage attributes from papers and their artifacts, conduct a statistical assessment to identify differences, and analyze the top five papers in each attribute category. Results: There is a significant difference between paper citations and the usage of associated artifacts. While statistical analyses show no notable difference between paper citations and GitHub stars, variations exist in views and/or downloads of papers and artifacts. Conclusion: We provide a thorough overview of ICSE's accepted papers from the last decade, emphasizing the intricate relationship between research papers and their artifacts. To enhance the assessment of artifact influence in software research, we recommend considering key attributes that may be present in one platform but not in another.
- C. C. S. Liem and A. M. Demetriou, “Treat societally impactful scientific insights as open-source software artifacts,” in 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS), 2023, pp. 150–156.
- R. Heumüller, S. Nielebock, J. Krüger, and F. Ortmeier, “Publish or perish, but do not forget your software artifacts,” Empirical Software Engineering, vol. 25, no. 6, pp. 4585–4616, 2020.
- E. Frachtenberg, “Research artifacts and citations in computer systems papers,” PeerJ Computer Science, vol. 8, p. e887, 2022.
- S. Winter, C. S. Timperley, B. Hermann, J. Cito, J. Bell, M. Hilton, and D. Beyer, “A retrospective study of one decade of artifact evaluations,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2022. New York, NY, USA: ACM, 2022.
- S. Ahmed, A. Ahmed, and N. U. Eisty, “Automatic transformation of natural to unified modeling language: A systematic review,” in IEEE/ACIS 20th International Conference on Software Engineering Research, Management & Applications. IEEE, 2022, pp. 112–119.
- H. B. Mann and D. R. Whitney, “On a test of whether one of two random variables is stochastically larger than the other,” The annals of mathematical statistics, pp. 50–60, 1947.
- F. Wilcoxon, S. Katti et al., “Critical values and probability levels for the wilcoxon rank sum test and the wilcoxon signed rank test,” Selected tables in mathematical statistics, vol. 1, pp. 171–259, 1970.
- P. Virtanen, et al, and SciPy 1.0 Contributors, “SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python,” Nature Methods, vol. 17, 2020.
- N. Cliff, “Dominance statistics: Ordinal analyses to answer ordinal questions.” Psychological bulletin, vol. 114, no. 3, p. 494, 1993.
- M. R. Hess and J. D. Kromrey, “Robust confidence intervals for effect sizes: A comparative study of cohen’sd and cliff’s delta under non-normality and heterogeneous variances,” in annual meeting of the American Educational Research Association, vol. 1. Citeseer, 2004.