Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

OpenPerf: A Benchmarking Framework for the Sustainable Development of the Open-Source Ecosystem (2311.15212v1)

Published 26 Nov 2023 in cs.SE

Abstract: Benchmarking involves designing scientific test methods, tools, and frameworks to quantitatively and comparably assess specific performance indicators of certain test subjects. With the development of artificial intelligence, AI benchmarking datasets such as ImageNet and DataPerf have gradually become consensus standards in both academic and industrial fields. However, constructing a benchmarking framework remains a significant challenge in the open-source domain due to the diverse range of data types, the wide array of research issues, and the intricate nature of collaboration networks. This paper introduces OpenPerf, a benchmarking framework designed for the sustainable development of the open-source ecosystem. This framework defines 9 task benchmarking tasks in the open-source research, encompassing 3 data types: time series, text, and graphics, and addresses 6 research problems including regression, classification, recommendation, ranking, network building, and anomaly detection. Based on the above tasks, we implemented 3 data science task benchmarks, 2 index-based benchmarks, and 1 standard benchmark. Notably, the index-based benchmarks have been adopted by the China Electronics Standardization Institute as evaluation criteria for open-source community governance. Additionally, we have developed a comprehensive toolkit for OpenPerf, which not only offers robust data management, tool integration, and user interface capabilities but also adopts a Benchmarking-as-a-Service (BaaS) model to serve academic institutions, industries, and foundations. Through its application in renowned companies and institutions such as Alibaba, Ant Group, and East China Normal University, we have validated OpenPerf's pivotal role in the healthy evolution of the open-source ecosystem.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Cyber threat intelligence using pca-dnn model to detect abnormal network behavior. Egyptian Informatics Journal 23, 173–185.
  2. The strategic importance of benchmarking as a tool for achieving excellence in higher education. International Journal of Excellence in Education 2.
  3. Data analysis and data mining: An introduction. OUP USA.
  4. Bothawk: An approach for bots detection in open source software projects. arXiv preprint arXiv:2307.13386 .
  5. Measurement and meaningful management. Public Productivity & Management Review , 31–43.
  6. Celof: Effective and fast memory efficient local outlier detection in high-dimensional data streams. Applied Soft Computing 102, 107079.
  7. Temporal autoregressive matrix factorization for high-dimensional time series prediction of oss. IEEE Transactions on Neural Networks and Learning Systems .
  8. Towards a comprehensive approach for assessing open source projects, in: International Conference on Software Process and Product Measurement, Springer. pp. 316–330.
  9. Socio-technical evolution of the ruby ecosystem in github, in: 2017 IEEE 24th international conference on software analysis, evolution and reengineering (SANER), IEEE. pp. 34–44.
  10. Imagenet: A large-scale hierarchical image database, in: 2009 IEEE conference on computer vision and pattern recognition, Ieee. pp. 248–255.
  11. WordNet: An electronic lexical database. MIT press.
  12. Challenges in creating a sustainable generic research data infrastructure .
  13. Open source software ecosystems in health sector: A case study from sri lanka, in: Information and Communication Technologies for Development: 14th IFIP WG 9.4 International Conference on Social Implications of Computers in Developing Countries, ICT4D 2017, Yogyakarta, Indonesia, May 22-24, 2017, Proceedings 14, Springer. pp. 71–80.
  14. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems 33, 22118–22133.
  15. Is developer sentiment related to software bugs: An exploratory study on github commits, in: 2020 IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE. pp. 527–531.
  16. Analysis of factors influencing developers’ sentiments in commit logs: Insights from applying sentiment analysis. e-Informatica Software Engineering Journal 16, 220102.
  17. A study for verification procedures on open-source software via benchmark testing. Journal of Information Technology Services 5, 99–108.
  18. Saibench: Benchmarking ai for science. BenchCouncil Transactions on Benchmarks, Standards and Evaluations 2, 100063.
  19. Empirical research on the evaluation model and method of sustainability of the open source ecosystem. Symmetry 10, 747.
  20. Healthy or not: A way to predict ecosystem health in github. Symmetry 11, 144.
  21. Risk assessment in software supply chains using the bayesian method. International Journal of Production Research 59, 6758–6775.
  22. World of code: enabling a research workflow for mining and analyzing the universe of open source vcs data. Empirical Software Engineering 26, 1–42.
  23. Dataperf: Benchmarks for data-centric ai development. arXiv preprint arXiv:2207.10062 .
  24. A case study of open source software development: the apache server, in: Proceedings of the 22nd international conference on Software engineering, pp. 263–272.
  25. Backstabber’s knife collection: A review of open source software supply chain attacks, in: Detection of Intrusions and Malware, and Vulnerability Assessment: 17th International Conference, DIMVA 2020, Lisbon, Portugal, June 24–26, 2020, Proceedings 17, Springer. pp. 23–43.
  26. Security and emotion: sentiment analysis of security discussions on github, in: Proceedings of the 11th working conference on mining software repositories, pp. 348–351.
  27. Semeval-2016 task 5: Aspect based sentiment analysis, in: ProWorkshop on Semantic Evaluation (SemEval-2016), Association for Computational Linguistics. pp. 19–30.
  28. Semeval-2015 task 12: Aspect based sentiment analysis, in: Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pp. 486–495.
  29. Relationship between geographical location and evaluation of developer contributions in github, in: Proceedings of the 12th ACM/IEEE international symposium on empirical software engineering and measurement, pp. 1–8.
  30. Affective sentiment and emotional analysis of pull request comments on github. Master’s thesis. University of Waterloo.
  31. Automatic instrument segmentation in robot-assisted surgery using deep learning, in: 2018 17th IEEE international conference on machine learning and applications (ICMLA), IEEE. pp. 624–628.
  32. The small-world effect: The influence of macro-level properties of developer collaboration networks on open-source project success. ACM Transactions on Software Engineering and Methodology (TOSEM) 20, 1–27.
  33. Analyzing developer sentiment in commit logs, in: Proceedings of the 13th international conference on mining software repositories, pp. 520–523.
  34. Let’s talk about it: evaluating contributions through discussion in github, in: Proceedings of the 22nd ACM SIGSOFT international symposium on foundations of software engineering, pp. 144–154.
  35. A data set for social diversity studies of github teams, in: 2015 IEEE/ACM 12th working conference on mining software repositories, IEEE. pp. 514–517.
  36. Understanding emotions of developer community towards software documentation, in: 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS), IEEE. pp. 87–91.
  37. Recommendation of Homogeneous Open Source Projects based on Link Prediction. Ph.D. dissertation. East China Normal University. doi:10.27149/d.cnki.ghdsu.2022.003970.
  38. Accurate developer recommendation for bug resolution, in: 2013 20th Working Conference on Reverse Engineering (WCRE), IEEE. pp. 72–81.
  39. Lessons learned from the ant group open source program office. Computer 56, 92–97.
  40. Exploring activity and contributors on github: Who, what, when, and where, in: 2022 29th Asia-Pacific Software Engineering Conference (APSEC), IEEE. pp. 11–20.
  41. Understanding the archived projects on github, in: 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), IEEE. pp. 13–24.
  42. Ask-roberta: A pretraining model for aspect-based sentiment classification via sentiment knowledge mining. Knowledge-Based Systems 253, 109511.
  43. Call for establishing benchmark science and engineering.
  44. A benchcouncil view on benchmarking emerging and future computing.
  45. How do companies collaborate in open source ecosystems? an empirical study of openstack, in: Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp. 1196–1208.
  46. Evaluation of developer contributions within open source projects based on openrank. URL: https://blog.frankzhao.cn/openrank_in_project/. [EB/OL].
  47. Telegraph: A benchmark dataset for hierarchical link prediction. arXiv preprint arXiv:2204.07703 .

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now