Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Cycling on the Freeway: The Perilous State of Open Source Neuroscience Software (2403.19394v1)

Published 28 Mar 2024 in cs.CY and q-bio.OT

Abstract: Most scientists need software to perform their research (Barker et al., 2020; Carver et al., 2022; Hettrick, 2014; Hettrick et al., 2014; Switters and Osimo, 2019), and neuroscientists are no exception. Whether we work with reaction times, electrophysiological signals, or magnetic resonance imaging data, we rely on software to acquire, analyze, and statistically evaluate the raw data we obtain - or to generate such data if we work with simulations. In recent years there has been a shift toward relying on free, open-source scientific software (FOSSS) for neuroscience data analysis (Poldrack et al., 2019), in line with the broader open science movement in academia (McKiernan et al., 2016) and wider industry trends (Eghbal, 2016). Importantly, FOSSS is typically developed by working scientists (not professional software developers) which sets up a precarious situation given the nature of the typical academic workplace (wherein academics, especially in their early careers, are on short and fixed term contracts). In this paper, we will argue that the existing ecosystem of neuroscientific open source software is brittle, and discuss why and how the neuroscience community needs to come together to ensure a healthy growth of our software landscape to the benefit of all.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. Evaluation of Software Impact Designed for Biomedical Research: Are We Measuring What’s Meaningful? arXiv .
  2. A Novel Approach for Estimating Truck Factors In 2016 IEEE 24th International Conference on Program Comprehension (ICPC), pp. 1–10.
  3. Evidence for the importance of research software.
  4. Bartneck C, Kokkelmans S (2011) Detecting h-index manipulation through self-citation analysis. Scientometrics 87:85–98.
  5. A survey of the state of the practice for research software in the United States. PeerJ Computer Science 8:e963.
  6. Software Engineering Practices in Academia: Promoting the 3Rs—Readability, Resilience, and Reuse. Harvard Data Science Review 5.
  7. The Effects of Diversity in Global, Distributed Collectives: A Study of Open Source Project Success. Information Systems Research 24:312–333.
  8. Data Without Software Are Just Numbers. Data Science Journal 19:3.
  9. Earley CP, Mosakowski E (2000) Creating Hybrid Team Cultures: An Empirical Test of Transnational Team Functioning. Academy of Management Journal 43:26–49.
  10. Eghbal N (2016) The Unseen Labor Behind Our Digital Infrastructure Technical report.
  11. A degree-of-knowledge model to capture source code familiarity In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1, pp. 385–394, Cape Town South Africa. ACM.
  12. Gender and Participation in Open Source Software Development. Proceedings of the ACM on Human-Computer Interaction 6:1–31.
  13. Geiger RS (2022) Summary Analysis of the 2017 GitHub Open Source Survey .
  14. Gewin V (2022) Has the ’great resignation’ hit academia? Nature pp. 211–213.
  15. FLOSS Final report. Technical report.
  16. Goodhart CA (1984) Problems of Monetary Management: The UK Experience Springer.
  17. Guest O, Forbes SH (2023) Teaching coding inclusively: If this, then what? Preprint, SocArXiv.
  18. Hafer L, Kirkpatrick AE (2009) Assessing open source software as a scholarly contribution. Communications of the ACM 52:126–129.
  19. Hettrick S (2014) It’s impossible to conduct research without software, say 7 out of 10 UK researchers. Software Sustainability Institute. Retrieved 20.06.2023 20:2016.
  20. Uk research software survey 2014.
  21. The Impact of Gender Diversity on the Performance of Business Teams: Evidence from a Field Experiment. Management Science 59:1514–1528.
  22. Hostler TJ (2023) The Invisible Workload of Open Research. Journal of Trial and Error .
  23. Jackson SE, Joshi A (2004) Diversity in social context: A multi-attribute, multilevel analysis of team diversity and sales performance. Journal of Organizational Behavior 25:675–702.
  24. Jensen EA, Katz DS (2023) Charting the course: Policy and planning for sustainable research software. Technical report.
  25. Katz DS, Chue Hong NP (2018) Software Citation in Theory and Practice In Davenport JH, Kauers M, Labahn G, Urban J, editors, Mathematical Software – ICMS 2018, Vol. 10931, pp. 289–296. Springer International Publishing, Cham.
  26. Recognizing the value of software: A software citation guide. F1000Research 9.
  27. How open science helps researchers succeed. eLife 5:e16800.
  28. Better incentives are needed to reward academic software development. Nature Ecology & Evolution 7:626–627.
  29. Millman K, Pérez F (2018) Developing Open-Source Scientific Practice * In Stodden V, Leisch F, Peng RD, editors, Implementing Reproducible Research, pp. 149–183. Chapman and Hall/CRC, 1 edition.
  30. A manifesto for reproducible science. Nature Human Behaviour 1:0021.
  31. On the Relationship Between the Developer’s Perceptible Race and Ethnicity and the Evaluation of Contributions in OSS. IEEE Transactions on Software Engineering pp. 1–1.
  32. Nafus D (2012) ‘Patches don’t have gender’: What is not open in open source software. New Media & Society 14:669–683.
  33. Changing computational research. The challenges ahead. Source Code for Biology and Medicine 7:2, 1751–0473–7–2.
  34. Pérez F (2011) Ten years of (interactive) scientific Python.
  35. Computational and Informatic Advances for Reproducible Data Analysis in Neuroimaging. Annual Review of Biomedical Data Science 2:119–138.
  36. Purvis A (2006) The h index: Playing the numbers game. Trends in Ecology & Evolution 21:422.
  37. Roberson QM (2019) Diversity in the Workplace: A Review, Synthesis, and Future Research Agenda. Annual Review of Organizational Psychology and Organizational Behavior 6:69–88.
  38. Self-citations as strategic response to the use of metrics for career decisions. Research Policy 48:478–491.
  39. Optimal Career Strategies and Brain Drain in Academia. Journal of Optimization Theory and Applications 168:268–295.
  40. Singh V, Bongiovanni B (2021) Motivated and Capable but No Space for Error: Women’s Experiences of Contributing to Open Source Software. The International Journal of Information, Diversity, & Inclusion (IJIDI) 5.
  41. Software citation principles. PeerJ Computer Science 2:e86.
  42. Journal of Open Source Software (JOSS): Design and first-year review. PeerJ Computer Science 4:e147.
  43. Switters J, Osimo D (2019) Recognising the importance of software in research: Research Software Engineers (RSEs), a UK example. Technical report, European Commission. Directorate General for Research and Innovation., LU.
  44. Astropy: A community Python package for astronomy. Astronomy & Astrophysics 558:A33.
  45. Perceptions of Diversity on Git Hub: A User Survey In 2015 IEEE/ACM 8th International Workshop on Cooperative and Human Aspects of Software Engineering, pp. 50–56, Florence, Italy. IEEE.
  46. Gender and Tenure Diversity in GitHub Teams In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 3789–3798, Seoul Republic of Korea. ACM.
  47. Whitaker K, Guest O (2020) #bropenscience is broken science .
  48. Zhivotovsky LA, Krutovsky KV (2008) Self-citation can inflate h-index. Scientometrics 77:373–375.

Summary

  • The paper shows that most projects have a truck factor of one to three, underscoring a precarious reliance on a few key contributors.
  • The paper finds that academic incentives favor new projects over maintaining existing software, hindering sustained development.
  • The paper identifies funding shortfalls and inclusivity challenges, proposing professionalization and improved training to stabilize ecosystems.

Assessing the Fragility of Open Source Neuroscience Software Ecosystems

Introduction

The reliance on free, open-source scientific software (FOSSS) within the neuroscience community is well-documented, reflecting a broader trend towards open science. However, this shift towards open-source tools for data acquisition, analysis, and statistical evaluation comes with significant risks and challenges. The paper presented by Britta U. Westner and colleagues explores the precarious nature of the open source software ecosystem in neuroscience, pointing out the brittle infrastructure that currently supports it. This post aims to provide a comprehensive summary of their findings, focusing on the truck factor, academic incentives for software development, funding issues, diversity in development teams, and the path forward to a more stable and inclusive open source ecosystem.

The Truck Factor and Its Implications

The concept of the Truck Factor, used to measure the number of individuals with critical knowledge necessary to maintain a project, reveals a concerning fragility within the neuroscience software community. With most projects having a Truck Factor of one to three, the continuity of these essential tools relies heavily on a minimal number of contributors. This situation underscores the vulnerability of the ecosystem and highlights the undue pressure on software maintainers, raising questions about the sustainability and reliability of these tools.

The Academic Incentive Structure

The current academic incentive model, heavily centered around publication counts, disproportionately disadvantages the development and maintenance of open-source software. The lack of citations for latter contributors and the underappreciation of software papers in tenure and promotion discussions discourage ongoing development efforts. This structure inadvertently promotes starting new projects over contributing to existing ones, exacerbating the risk of software becoming unmaintained.

Funding Challenges

The scarcity of stable, long-term funding for software development and maintenance poses another significant hurdle. While there are emerging support mechanisms from various organizations, the predominantly short-term nature of grants fails to provide lasting solutions, leaving the ecosystem vulnerable once the funding period ends.

Diversity and Inclusivity Issues

Research consistently shows that diversity within development teams leads to better project outcomes. However, structural barriers such as misogyny, racism, and a lack of representation severely hinder the diversification of the developer pool in open-source projects. These issues not only compromise project quality but also reflect broader systemic issues within the open-source community and academia at large.

Pathways to Sustainability

The paper outlines several strategies to address these challenges and strengthen the open-source neuroscience software ecosystem:

  • Professionalizing Software Development: Increasing institutional support and funding for maintenance activities, as well as recognizing and rewarding contributions to software development, are essential steps towards sustainability.
  • Adopting New Citation Practices: Encouraging the practice of citing the specific version of the software used, rather than the seminal paper, could better reflect and reward contributions across a project's lifespan.
  • Valuing Software Contributions: Funding bodies, tenure, and promotion committees should regard significant open-source software work as equivalent in value to academic publications.
  • Improving Training: Incorporating programming literacy and software development skills into neuroscience education will prepare future generations for meaningful contributions to the ecosystem.
  • Enhancing Inclusivity: By changing the incentive structure and actively fostering an environment that welcomes and supports underrepresented groups, the diversity and strength of the developer community can be significantly improved.

Conclusion

The insights presented in this paper underline a critical need for cultural and structural changes within academia and the neuroscience community regarding open-source software. It's a collective call to action: recognizing software development as a vital scientific contribution, reevaluating the current incentive structure, and actively working towards a more inclusive, sustainable open-source ecosystem. Failure to address these concerns not only risks the integrity and progress of neuroscience research but also overlooks the potential to fully harness the power of collaborative, open science.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown