State of the Practice for Medical Imaging Software (2405.12171v1)
Abstract: We selected 29 medical imaging projects from 48 candidates, assessed 10 software qualities by answering 108 questions for each software project, and interviewed 8 of the 29 development teams. Based on the quantitative data, we ranked the MI software with the Analytic Hierarchy Process (AHP). The four top-ranked software products are 3D Slicer, ImageJ, Fiji, and OHIF Viewer. Generally, MI software is in a healthy state as shown by the following: we observed 88% of the documentation artifacts recommended by research software development guidelines, 100% of MI projects use version control tools, and developers appear to use the common quasi-agile research software development process. However, the current state of the practice deviates from the existing guidelines because of the rarity of some recommended artifacts, low usage of continuous integration (17% of the projects), low use of unit testing (about 50% of projects), and room for improvement with documentation (six of nine developers felt their documentation was not clear enough). From interviewing the developers, we identified five pain points and two qualities of potential concern: lack of development time, lack of funding, technology hurdles, ensuring correctness, usability, maintainability, and reproducibility. The interviewees proposed strategies to improve the state of the practice, to address the identified pain points, and to improve software quality. Combining their ideas with ours, we have the following list of recommendations: increase documentation, increase testing by enriching datasets, increase continuous integration usage, move to web applications, employ linters, use peer reviews, design for change, add assurance cases, and incorporate a "Generate All Things" approach.
- Safia Abdalla. 2016. A template for creating open source contributor guidelines. https://opensource.com/life/16/3/contributor-guidelines-template-and-tips.
- Scientific Software Development at a Research Facility. IEEE Software 25, 4 (July/August 2008), 44–51.
- U.S. Food & Drug Administration. 2021. Medical Imaging. https://www.fda.gov/radiation-emitting-products/radiation-emitting-products-and-procedures/medical-imaging. [Online; accessed 25-July-2021].
- Aysel Afsar. 2021. DICOM Viewer. https://github.com/ayselafsar/dicomviewer. [Online; accessed 27-May-2021].
- ParaView: An End-User Tool for Large Data Visualization. Visualization Handbook (01 2005).
- Yasmin AlNoamany and John A. Borghi. 2018. Towards computational reproducibility: researcher perspectives on the use and sharing of software. PeerJ Computer Science 4, e163 (September 2018), 1–25.
- InVesalius: An Interactive Rendering Framework for Health Care Support. 10. https://doi.org/10.1007/978-3-319-27857-5_5
- Identifying, Categorizing and Mitigating Threats to Validity in Software Engineering Secondary Studies. Information and Software Technology 106 (02 2019). https://doi.org/10.1016/j.infsof.2018.10.006
- Mathematical methods in medical image processing. Bulletin (new series) of the American Mathematical Society 43 (07 2006), 365–396. https://doi.org/10.1090/S0273-0979-06-01104-9
- Kevin Archie and Daniel Marcus. 2012. DicomBrowser: Software for Viewing and Modifying DICOM Metadata. Journal of digital imaging : the official journal of the Society for Computer Applications in Radiology 25 (02 2012), 635–45. https://doi.org/10.1007/s10278-012-9462-x
- Medical Imaging Technology Association. 2021. About DICOM: Overview. https://www.dicomstandard.org/about-home. [Online; accessed 11-August-2021].
- The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). IEEE Transactions on Medical Imaging 34, 10 (Oct. 2015), 1993–2024.
- Isaac N. Bankman. 2000. Preface. In Handbook of Medical Imaging, Isaac N. Bankman (Ed.). Academic Press, San Diego, xi – xii. https://doi.org/10.1016/B978-012077790-7/50001-1
- F. Benureau and N. Rougier. 2017. Re-run, Repeat, Reproduce, Reuse, Replicate: Transforming Code into Scientific Contributions. ArXiv e-prints (Aug. 2017). arXiv:1708.08205 [cs.GL]
- The Liver Tumor Segmentation Benchmark (LiTS). CoRR abs/1901.04056 (2019). arXiv:1901.04056 http://arxiv.org/abs/1901.04056
- Christian Bird and Alberto Bacchelli. 2013. Expectations, Outcomes, and Challenges of Modern Code Review. In Proceedings of the International Conference on Software Engineering (proceedings of the international conference on software engineering ed.). IEEE. https://www.microsoft.com/en-us/research/publication/expectations-outcomes-and-challenges-of-modern-code-review/
- Kari Björn. 2017. Evaluation of Open Source Medical Imaging Software: A Case Study on Health Technology Student Learning Experience. Procedia Computer Science 121 (01 2017), 724–731. https://doi.org/10.1016/j.procs.2017.11.094
- Barry W Boehm. 2007. Software engineering: Barry W. Boehm’s lifetime contributions to software development, management, and research. Vol. 69. John Wiley & Sons.
- Ben Boyter. 2021. Sloc Cloc and Code. https://github.com/boyter/scc. [Online; accessed 27-May-2021].
- Scottish Covid-19 Response Consortium. https://github.com/ScottishCovidResponse/modelling-software-checklist/blob/main/software-checklist.md.
- Titus Brown. 2015. Notes from“How to grow a sustainable software development process (for scientific software)”. http://ivory.idyll.org/blog/2015-growing-sustainable-software-development-process.html.
- Free DICOM-Viewers for Veterinary Medicine: Survey and Comparison of Functionality and User-Friendliness of Medical Imaging PACS-DICOM-Viewer Freeware for Specific Use in Veterinary Medicine Practices. Journal of Digital Imaging (03 2019). https://doi.org/10.1007/s10278-019-00194-3
- Jacques Carette and Oleg Kiselyov. 2011. Multi-stage programming with Functors and Monads: Eliminating abstraction overhead from generic code. Sci. Comput. Program. 76, 5 (2011), 349–375.
- Drasil. https://github.com/JacquesCarette/Drasil/tree/v0.1-alpha
- David Carty. 2020. Follow Google’s lead with programming style guides. https://www.techtarget.com/searchsoftwarequality/feature/Follow-Googles-lead-with-programming-style-guides.
- Software Development Environments for Scientific and Engineering Software: A Series of Case Studies. In ICSE ’07: Proceedings of the 29th International Conference on Software Engineering. IEEE Computer Society, Washington, DC, USA, 550–559. https://doi.org/10.1109/ICSE.2007.77
- A Survey of the State of the Practice for Research Software in the United States. PeerJ Computer Science 8:e963 (2022). https://doi.org/10.7717/peerj-cs.963
- A lightweight rapid application development framework for biomedical image analysis. Computer Methods and Programs in Biomedicine 164 (07 2018). https://doi.org/10.1016/j.cmpb.2018.07.011
- Picture archiving and communication systems: an overview. Radiographics : a review publication of the Radiological Society of North America, Inc 12 (02 1992), 127–9. https://doi.org/10.1148/radiographics.12.1.1734458
- James Edward Corbly. 2014. The Free Software Alternative: Freeware, Open Source Software, and Libraries. Information Technology and Libraries 33, 3 (Sep. 2014), 65–75. https://doi.org/10.6017/ital.v33i3.5105
- What Makes Research Software Sustainable? An Interview Study With Research Software Engineers. CoRR abs/1903.06039 (2019). arXiv:1903.06039 http://arxiv.org/abs/1903.06039
- Ao Dong. 2021a. Assessing the State of the Practice for Medical Imaging Software. Master’s thesis. McMaster University, Hamilton, ON, Canada.
- Ao Dong. 2021b. Software Quality Grades for MI Software. Mendeley Data, V1, doi: 10.17632/k3pcdvdzj2.1. https://doi.org/10.17632/k3pcdvdzj2.1
- Steve M. Easterbrook and Timothy C. Johns. 2009. Engineering the Software for Understanding Climate Change. Comuting in Science & Engineering 11, 6 (November/December 2009), 65–74. https://doi.org/10.1109/MCSE.2009.193
- C. Ebert and C. Jones. 2009. Embedded Software: Facts, Figures, and Future. Computer 42, 4 (April 2009), 42–52. https://doi.org/10.1109/MC.2009.118
- Semi-formal design of reliable mesh generation systems. Advances in Engineering Software 35, 12 (2004), 827–841.
- Steve Emms. 2019. 16 Best Free Linux Medical Imaging Software. https://www.linuxlinks.com/medicalimaging/. [Online; accessed 02-February-2020].
- ESA. February 1991. ESA Software Engineering Standards, PSS-05-0 Issue 2. Technical Report. European Space Agency.
- M. E. Fagan. 1976. Design and code inspections to reduce errors in program development. IBM Systems Journal 15, 3 (1976), 182–211. https://doi.org/10.1147/sj.153.0182
- Scientific Computing’s Productivity Gridlock: How Software Engineering Can Help. Computing in Science Engineering 11, 6 (Nov 2009), 30–39. https://doi.org/10.1109/MCSE.2009.205
- Medinria: DT-MRI processing and visualization software. (04 2012).
- Karl Fogel. 2005. Producing Open Source Software: How to Run a Successful Free Software Project. O’Reilly Media, Inc.
- Martin Fowler. 2006. Continuous Integration. https://martinfowler.com/articles/continuousIntegration.html.
- Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley Professional.
- Marc-Oliver Gewaltig and Robert Cannon. 2012. Quality and sustainability of software tools in neuroscience. Cornell University Library (May 2012), 20 pp.
- Fundamentals of Software Engineering (2nd ed.). Prentice Hall, Upper Saddle River, NJ, USA.
- Tomasz Gieniusz. 2019. GitStats. https://github.com/tomgi/git_stats. [Online; accessed 27-May-2021].
- Ray Givler. 2020. A Checklist of Basic Software Engineering Practices for Data Analysts and Data Scientists. https://www.linkedin.com/pulse/checklist-basic-software-engineering-practices-data-analysts-givler/?articleId=6681691007303630849.
- GNU. 2019. Categories of free and nonfree software. https://www.gnu.org/philosophy/categories.html. [Online; accessed 20-May-2021].
- Carole Goble. 2014. Better Software, Better Research. IEEE Internet Computing 18, 5 (2014), 4–8. https://doi.org/10.1109/MIC.2014.88
- A Survey of DICOM Viewer Software to Integrate Clinical Research and Medical Imaging. Journal of digital imaging 29 (10 2015). https://doi.org/10.1007/s10278-015-9833-1
- Daniel Haehn. 2013. Slice:drop: collaborative medical imaging in the browser. 1–1. https://doi.org/10.1145/2503541.2503645
- Paul Hamill. 2004. Unit test frameworks: Tools for high-quality software development. O’Reilly Media.
- How do scientists develop and use scientific software?. In 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering. 1–8. https://doi.org/10.1109/SECSE.2009.5069155
- How Do Scientists Develop and Use Scientific Software?. In Proceedings of the 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering (SECSE ’09). IEEE Computer Society, Washington, DC, USA, 1–8. https://doi.org/10.1109/SECSE.2009.5069155
- Mehedi Hasan. 2020. Top 25 Best Free Medical Imaging Software for Linux System. https://www.ubuntupit.com/top-25-best-free-medical-imaging-software-for-linux-system/. [Online; accessed 30-January-2020].
- Dustin Heaton and Jeffrey C. Carver. 2015. Claims About the Use of Software Engineering Practices in Science. Inf. Softw. Technol. 67, C (Nov. 2015), 207–219. https://doi.org/10.1016/j.infsof.2015.07.011
- Michael A. Heroux and David E. Bernholdt. 2018. Better (Small) Scientific Software Teams, tutorial in Argonne Training Program on Extreme-Scale Computing (ATPESC). https://press3.mcs.anl.gov//atpesc/files/2018/08/ATPESC_2018_Track-6_3_8-8_1030am_Bernholdt-Better_Scientific_Software_Teams.pdf. https://doi.org/articles/journal_contribution/ATPESC_Software_Productivity_03_Better_Small_Scientific_Software_Teams/6941438
- Trilinos Developers Guide Part II: ASC Softwar Quality Engineering Practices Version 2.0. https://faculty.csbsju.edu/mheroux/fall2012_csci330/TrilinosDevGuide2.pdf.
- Usage, costs, and benefits of continuous integration in open-source projects. In 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE). 426–437.
- Daniel M. Hoffman and Paul A. Strooper. 1995. Software Design, Automated Testing, and Maintenance: A Practical Approach. International Thomson Computer Press, New York, NY, USA.
- horosproject.org. 2020. Horos. https://github.com/horosproject/horos. [Online; accessed 27-May-2021].
- James Howison and Julia Bullard. 2016. Software in the Scientific Literature: Problems with Seeing, Finding, and Using Software Mentioned in the Biology Literature. J. Assoc. Inf. Sci. Technol. 67, 9 (sep 2016), 2137–2155. https://doi.org/10.1002/asi.23538
- Jez Humble and David G. Farley. 2010. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley, Upper Saddle River, NJ. http://my.safaribooksonline.com/9780321601919
- IEEE. 1991. IEEE Standard Glossary of Software Engineering Terminology. Standard. IEEE.
- IEEE. 1998. Recommended Practice for Software Requirements Specifications. IEEE Std 830-1998 (Oct. 1998), 1–40. https://doi.org/10.1109/IEEESTD.1998.88286
- Joint Task Force on Software Engineering Ethics IEEE-CS/ACM and Professional Practices. 1999. Code of Ethics, IEEE Computer Society. https://www.computer.org/education/code-of-ethics.
- Parallax Innovations. 2020. Microview. https://github.com/parallaxinnovations/MicroView/. [Online; accessed 27-May-2021].
- Software Sustainability Institute. 2022. Online sustainability evaluation. https://www.software.ac.uk/resources/online-sustainability-evaluation.
- Alessio Ishizaka and Markus Lusti. 2006. How to derive priorities in AHP: A comparative study. Central European Journal of Operations Research 14 (12 2006), 387–400. https://doi.org/10.1007/s10100-006-0012-9
- ISO. 2001. Iec 9126-1: Software engineering-product quality-part 1: Quality model. Geneva, Switzerland: International Organization for Standardization 21 (2001).
- ISO/IEC. 2011. Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - System and software quality models. Standard. International Organization for Standardization.
- ISO/TR. 2002. Ergonomics of human-system interaction — Usability methods supporting human-centred design. Standard. International Organization for Standardization.
- ISO/TR. 2018. Ergonomics of human-system interaction — Part 11: Usability: Definitions and concepts. Standard. International Organization for Standardization.
- GATE: a simulation toolkit for PET and SPECT. Physics in medicine and biology 49 (11 2004), 4543–61. https://doi.org/10.1088/0031-9155/49/19/007
- Arne N. Johanson and Wilhelm Hasselbring. 2018. Software Engineering for Computational Science: Past, Present, Future. Computing in Science & Engineering Accepted (2018), 1–31.
- Capers Jones. 2008. Measuring Defect Potentials and Defect Removal Efficiency. Crosstalk, The Journal of Defense Software Engineering 21, 6 (June 2008), 11–13.
- Unified Framework for Development, Deployment and Robust Testing of Neuroimaging Algorithms. Neuroinformatics 9 (03 2011), 69–84. https://doi.org/10.1007/s12021-010-9092-8
- Thematic Domain Analysis for Ocean Modeling. Environmental Modelling & Software (Jan 2022), 105323. https://doi.org/10.1016/j.envsoft.2022.105323
- Panagiotis Kalagiakos. 2003. The Non-Technical Factors of Reusability. In Proceedings of the 29th Conference on EUROMICRO. IEEE Computer Society, 124.
- U. Kanewala and J. M. Bieman. 2013. Techniques for testing scientific programs without an oracle. In Software Engineering for Computational Science and Engineering (SE-CSE), 2013 5th International Workshop on. 48–57. https://doi.org/10.1109/SECSE.2013.6615099
- Neeraj Kashyap. 2020. GitHub’s Path to 128M Public Repositories. https://towardsdatascience.com/githubs-path-to-128m-public-repositories-f6f656ab56b1.
- Matthias Katerbow and Georg Feulner. 2018. Recommendations on the development, use and provision of Research Software. https://doi.org/10.5281/zenodo.1172988
- Diane Kelly. 2013. Industrial Scientific Software: A Set of Interviews on Software Development. In Proceedings of the 2013 Conference of the Center for Advanced Studies on Collaborative Research (Ontario, Canada) (CASCON ’13). IBM Corp., Riverton, NJ, USA, 299–310. http://dl.acm.org/citation.cfm?id=2555523.2555555
- Diane Kelly. 2015. Scientific software development viewed as knowledge acquisition: Towards understanding the development of risk-averse scientific software. Journal of Systems and Software 109 (2015), 50–61. https://doi.org/10.1016/j.jss.2015.07.027
- Diane Kelly and Terry Shepard. 2000. Task-directed software inspection technique: an experiment and case study. In CASCON ’00: Proceedings of the 2000 conference of the Centre for Advanced Studies on Collaborative research (Mississauga, Ontario, Canada). IBM Press, 6. http://portal.acm.org/citation.cfm?id=782040#
- Diane F. Kelly. 2007. A Software Chasm: Software Engineering and Scientific Computing. IEEE Software 24, 6 (2007), 120–119. https://doi.org/10.1109/MS.2007.155
- Software Engineering for Scientists. Computing in Science & Engineering 13, 5 (Oct. 2011), 7–11.
- 3D Slicer: A Platform for Subject-Specific Image Analysis, Visualization, and Clinical Support. Vol. 3. 277–289. https://doi.org/10.1007/978-1-4614-7657-3_19
- The Recent Progress in Quantitative Medical Image Analysis for Computer Aided Diagnosis Systems. Healthcare informatics research 17 (09 2011), 143–9. https://doi.org/10.4258/hir.2011.17.3.143
- Technical debt: From metaphor to theory and practice. IEEE Software 29, 6 (2012), 18–21.
- Chris Rorden’s Lab. 2021. MRIcroGL. https://github.com/rordenlab/MRIcroGL. [Online; accessed 27-May-2021].
- Measuring the installability of service orchestrations using the square method. In 2013 IEEE 6th International Conference on Service-Oriented Computing and Applications. IEEE, 118–125.
- How software engineers use documentation: the state of the practice. IEEE Software 20, 6 (2003), 35–39. https://doi.org/10.1109/MS.2003.1241364
- Ajay Limaye. 2012. Drishti, A Volume Exploration and Presentation Tool. Proc SPIE 8506, 85060X. https://doi.org/10.1117/12.935640
- Fast Realistic MRI Simulations Based on Generalized Multi-Pool Exchange Tissue Model. IEEE Transactions on Medical Imaging PP (10 2016), 1–1. https://doi.org/10.1109/TMI.2016.2620961
- Andy Loening. 2017. AMIDE. https://sourceforge.net/p/amide/code/ci/default/tree/amide-current/. [Online; accessed 27-May-2021].
- Automated Solution of Differential Equations by the Finite Element Method. Lecture Notes in Computational Science and Engineering, Vol. 84. Springer. https://doi.org/10.1007/978-3-642-23099-8
- Detecting Near Duplicates in Software Documentation. Programming and Computer Software 44, 5 (01 Sep 2018), 335–343. https://doi.org/10.1134/S0361768818050079
- Yves Martelli. 2021. dwv. https://github.com/ivmartel/dwv. [Online; accessed 27-May-2021].
- Development High Performance Scientific Computing Application Using Model-Driven Architecture. Applied Mathematical Sciences 7, 100 (2013), 4961–4974.
- ITK: Enabling Reproducible Research and Open Science. Frontiers in neuroinformatics 8 (02 2014), 13. https://doi.org/10.3389/fninf.2014.00013
- Application Programming Interface Documentation: What Do Software Developers Want? Journal of Technical Writing and Communication 48, 3 (2018), 295–330. https://doi.org/10.1177/0047281617721853 arXiv:https://doi.org/10.1177/0047281617721853
- Peter Michalski. 2021. State of The Practice for Lattice Boltzmann Method Software. Master’s thesis. McMaster University, Hamilton, Ontario, Canada.
- Hamza Mu. 2019. 20 Free & open source DICOM viewers for Windows. https://medevel.com/free-dicom-viewers-for-windows/. [Online; accessed 31-January-2020].
- Product Roadmap – From Vision to Reality: A Systematic Literature Review. In 2019 IEEE International Conference on Engineering, Technology and Innovation (ICE/ITMC). 1–8. https://doi.org/10.1109/ICE.2019.8792654
- Software reliability: prediction and application.
- NASA. 1989. Software requirements DID, SMAP-DID-P200-SW, Release 4.3. Technical Report. National Aeronautics and Space Agency.
- D Nevcas and P Klapetek. 2012. Gwyddion: an open-source software for spm data analysis. Cent Eur J Phys 10 (01 2012).
- A Survey of Scientific Software Development. In Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (Bolzano-Bozen, Italy) (ESEM ’10). ACM, New York, NY, USA, Article 12, 10 pages. https://doi.org/10.1145/1852786.1852802
- (X)MedCon * An OpenSource Medical Image Conversion Toolkit. European Journal of Nuclear Medicine and Molecular Imaging 30 (08 2003), S246. https://doi.org/10.1007/s00259-003-1284-0
- On the use of models for high-performance scientific computing applications: an experience report. Software & Systems Modeling 17, 1 (01 Feb 2018), 319–342. https://doi.org/10.1007/s10270-016-0518-0
- A set of common software quality assurance baseline criteria for research projects. https://doi.org/10.20350/digitalCSIC/12543
- State of Sustainability for Research Software (poster). In SIAM-CSE 2021 Conference on Computational Science and Engineering, Minisymposterium: Software Productivity and Sustainability for CSE. https://doi.org/10.6084/m9.figshare.14039888.v2
- A. Panchal and R. Keyes. 2010. SU-GG-T-260: Dicompyler: An Open Source Radiation Therapy Research Platform with a Plugin Architecture. Medical Physics - MED PHYS 37 (06 2010). https://doi.org/10.1118/1.3468652
- BioImage Suite: An integrated medical image analysis suite. 1 (01 2005).
- The modular structure of complex systems. In International Conference on Software Engineering. 408–419.
- David L. Parnas. 1972. On the Criteria To Be Used in Decomposing Systems into Modules. Comm. ACM 15, 2 (Dec. 1972), 1053–1058.
- David Lorge Parnas and Paul C Clements. 1986. A rational design process: How and why to fake it. IEEE transactions on software engineering 2 (1986), 251–257.
- Developing a technology roadmapping system. In A Unifying Discipline for Melting the Boundaries Technology Management:. 99–111. https://doi.org/10.1109/PICMET.2005.1509680
- Roman Pichler. 2012. Working with an Agile Product Roadmap. https://www.romanpichler.com/blog/agile-product-roadmap/.
- More Common Than You Think: An In-depth Study of Casual Contributors. In 2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER), Vol. 1. 112–123. https://doi.org/10.1109/SANER.2016.68
- How Do Scientists Develop and Use Scientific Software? An External Replication. In Proceedings of 25th IEEE International Conference on Software Analysis, Evolution and Reengineering. 582–591. https://doi.org/10.1109/SANER.2018.8330263
- A Survey of the Practice of Computational Science (SC ’11). Association for Computing Machinery, New York, NY, USA, Article 19, 12 pages. https://doi.org/10.1145/2063348.2063374
- Categorizing the Content of GitHub README Files. arXiv:1802.06997 [cs.SE]
- The public cancer radiology imaging collections of The Cancer Imaging Archive. Scientific Data 4 (09 2017), sdata2017124. https://doi.org/10.1038/sdata.2017.124
- Professional Engineers Act. 2021. Professional Engineers Act, RSO 1990, c P. 28. https://canlii.ca/t/5568z.
- The Linux Information Project. 2006. Freeware Definition. http://www.linfo.org/freeware.html. [Online; accessed 20-May-2021].
- Fast Automatic Generation of DSP Algorithms. In International Conference on Computational Science (ICCS) (Lecture Notes In Computer Science, Vol. 2073). Springer, 97–106.
- UTHSCSA Research Imaging Institute. 2019. Papaya. https://github.com/rii-mango/Papaya. [Online; accessed 27-May-2021].
- Peter C. Rigby and Christian Bird. 2013. Convergent Contemporary Software Peer Review Practices. In Proceedings of the 2013 9th Joint Meeting on Foundations of Software Engineering (Saint Petersburg, Russia) (ESEC/FSE 2013). Association for Computing Machinery, New York, NY, USA, 202–212. https://doi.org/10.1145/2491411.2491444
- Current Practices in Constructing and Evaluating Assurance Cases with Applications to Aviation. Technical Report CR-2014-218678. National Aeronautics and Space Administration (NASA), Langley Research Centre, Hampton, Virginia.
- Suzanne Robertson and James Robertson. 1999. Mastering the Requirements Process. ACM Press/Addison-Wesley Publishing Co, New York, NY, USA, Chapter Volere Requirements Specification Template, 353–391.
- Nicolas Roduit. 2021. Weasis. https://github.com/nroduit/nroduit.github.io. [Online; accessed 27-May-2021].
- J. Rokicki and L. Laniewski-Wollk. 2016. Adjoint lattice Boltzmann for topology optimization on multi-GPU architecture. Computers & Mathematics with Applications 71, 3 (2016), 833–848.
- ImageJ2: ImageJ for the next generation of scientific image data. BMC Bioinformatics 18 (11 2017). https://doi.org/10.1186/s12859-017-1934-z
- Per Runeson and Martin Höst. 2009. Guidelines for conducting and reporting case study research in software engineering. Empirical Software Engineering 14, 2 (19 Dec 2009), 131–164. https://doi.org/10.1007/s10664-008-9102-8
- Thomas L. Saaty. 1990. How to make a decision: The analytic hierarchy process. European Journal of Operational Research 48, 1 (1990), 9–26. https://doi.org/10.1016/0377-2217(90)90057-I Desicion making by the analytic hierarchy process: Theory and applications.
- Modern code review: a case study at google. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice. 181–190.
- Ravi Samala. 2014. Can anyone suggest free software for medical images segmentation and volume? https://www.researchgate.net/post/Can_anyone_suggest_free_software_for_medical_images_segmentation_and_volume. [Online; accessed 31-January-2020].
- Rebecca Sanders and Diane Kelly. 2008. Dealing with Risk in Scientific Software Development. IEEE Software 4 (July/August 2008), 21–28.
- Pixmeo SARL. 2019. OsiriX Lite. https://github.com/pixmeo/osirix. [Online; accessed 27-May-2021].
- Fiji: An Open-Source Platform for Biological-Image Analysis. Nature methods 9 (06 2012), 676–82. https://doi.org/10.1038/nmeth.2019
- DLR Software Engineering Guidelines. https://doi.org/10.5281/zenodo.1344612
- The visualization toolkit. Kitware.
- Judith Segal. 2005. When Software Engineers Met Research Scientists: A Case Study. Empirical Software Engineering 10, 4 (Oct. 2005), 517–536. https://doi.org/10.1007/s10664-005-3865-y
- Judith Segal and Chris Morris. 2008. Developing Scientific Software. IEEE Software 25, 4 (July/August 2008), 18–20.
- Continuous Integration, Delivery and Deployment: A Systematic Review on Approaches, Tools, Challenges and Practices. IEEE Access 5 (2017), 3909–3943. https://doi.org/10.1109/ACCESS.2017.2685629
- Codes of conduct in Open Source Software—for warm and fuzzy feelings or equality in community? Software Quality Journal (2021). https://doi.org/10.1007/s11219-020-09543-w
- Continuous Integration, In-Code Documentation, and Automation for Nuclear Quality Assurance Conformance. Nuclear Technology 207 (01 2021), 1–8. https://doi.org/10.1080/00295450.2020.1826804
- James Smirniotopoulos. 2014. MedPix Medical Image Database. https://doi.org/10.13140/2.1.3403.3608
- xSDK Community Package Policies. https://doi.org/10.6084/m9.figshare.4495136.v6
- Quality Definitions of Qualities. (2020). Manuscript in preparation.
- Spencer Smith and Peter Michalski. 2022. Digging Deeper Into the State of the Practice for Domain Specific Research Software. In Proceedings of the International Conference on Computational Science, ICCS. 1–15.
- State of the Practice for Lattice Boltzmann Method Software. Archives of Computational Methods in Engineering 31, 1 (Jan 2024), 313–350. https://doi.org/10.1007/s11831-023-09981-2
- Statistical Software for Psychology: Comparing Development Practices Between CRAN and Other Communities. arXiv:1802.07362 [cs.SE]
- Seismology software: state of the practice. Journal of Seismology 22 (05 2018). https://doi.org/10.1007/s10950-018-9731-3
- W. Spencer Smith. 2016. A Rational Document Driven Design Process for Scientific Computing Software. In Software Engineering for Science, Jeffrey C. Carver, Neil Chue Hong, and George Thiruvathukal (Eds.). Taylor & Francis, Chapter Section I – Examples of the Application of Traditional Software Engineering Practices to Science, 33–63.
- W. Spencer Smith. 2018. Beyond Software Carpentry. In 2018 International Workshop on Software Engineering for Science (held in conjunction with ICSE’18). 1–8.
- W. Spencer Smith and Jacques Carette. 2021. Sustainable Software via Generation. In Proceedings of the 1st Annual Booth Resource and Innovation Cluster (BRIC) Symposium. 21.
- Methodology for Assessing the State of the Practice for Domain X. https://arxiv.org/abs/2110.11575.
- W. Spencer Smith and Nirmitha Koothoor. 2016. A Document-Driven Method for Certifying Scientific Computing Software for Use in Nuclear Safety Analysis. Nuclear Engineering and Technology 48, 2 (April 2016), 404–418. https://doi.org/10.1016/j.net.2015.11.008
- Requirements Analysis for Engineering Computation: A Systematic Approach for Improving Software Reliability. Reliable Computing, Special Issue on Reliable Engineering Computation 13, 1 (Feb. 2007), 83–107. https://doi.org/10.1007/s11155-006-9020-7
- State of Practice for Mesh Generation Software. Advances in Engineering Software 100 (Oct. 2016), 53–71.
- State of the Practice for GIS Software. arXiv:1802.03422 [cs.SE]
- State of the practice for mesh generation and mesh processing software. Advances in Engineering Software 100 (2016), 53–71.
- Raising the Bar: Assurance Cases for Scientific Computing Software. Computing in Science and Engineering 23, 1 (Feb. 2020), 47–57.
- Statistical Software for Psychology: Comparing Development Practices Between CRAN and Other Communities. https://arxiv.org/abs/1802.07362. 33 pp..
- W. Spencer Smith and Wen Yu. 2009. A Document Driven Methodology for Improving the Quality of a Parallel Mesh Generation Toolbox. Advances in Engineering Software 40, 11 (Nov. 2009), 1155–1167. https://doi.org/10.1016/j.advengsoft.2009.05.003
- SourceLevel. 2022. What is a linter and why your team should use it? https://sourcelevel.io/blog/what-is-a-linter-and-why-your-team-should-use-it.
- John Spriggs. 2012. GSN - The Goal Structuring Notation. Springer-Verlag, London.
- Graeme Stewart et al. 2017. A Roadmap for HEP Software and Computing R&D for the 2020s. arXiv (2017). arXiv:1712.06982 [physics.comp-ph]
- Tim Storer. 2017. Bridging the Chasm: A Survey of Software Engineering Practice in Scientific Programming. ACM Comput. Surv. 50, 4, Article 47 (Aug. 2017), 32 pages. https://doi.org/10.1145/3084225
- Keenan Szulik. 2017. Don’t judge a project by its GitHub stars alone. https://blog.tidelift.com/dont-judge-a-project-by-its-github-stars-alone.
- Position Paper: A Knowledge-Based Approach to Scientific Software Development. In Proceedings of SE4Science’16 in conjunction with the International Conference on Software Engineering (ICSE). In conjunction with ICSE 2016, Austin, Texas, United States. 4 pp..
- TESCAN. 2020. 3DimViewer. https://bitbucket.org/3dimlab/3dimviewer/src/master/. [Online; accessed 27-May-2021].
- Carsten Thiel. 2020. EURISE Network Technical Reference. https://technical-reference.readthedocs.io/en/latest/.
- Code of conduct in open source projects. In 2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER). 24–33. https://doi.org/10.1109/SANER.2017.7884606
- USGS. 2019. USGS (United States Geological Survey) Software Plannning Checklist. https://www.usgs.gov/media/files/usgs-software-planning-checklist.
- An Approach to Product Roadmapping in Small Software Product Businesses. University of Technologie, Software Business and Engineering Institute, Helsinki, Finland.
- Omkarprasad S. Vaidya and Sushil Kumar. 2006. Analytic hierarchy process: An overview of applications. European Journal of Operational Research 169, 1 (2006), 1–29. https://doi.org/10.1016/j.ejor.2004.04.028
- Guidelines for Software Quality, CLARIAH Task Force 54.100. https://github.com/CLARIAH/software-quality-guidelines/blob/master/softwareguidelines.pdf.
- Hans van Vliet. 2000. Software Engineering (2nd ed.): Principles and Practice. John Wiley & Sons, Inc., New York, NY, USA.
- Todd. L. Veldhuizen. 1998. Arrays in Blitz++. In Proceedings of the 2nd International Scientific Computing in Object-Oriented Parallel Environments (ISCOPE’98), Lecture Notes in Computer Science. Springer-Verlag.
- ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. arXiv:1705.02315 (05 2017).
- B. Douglas Ward. 2000. Program 3dfim+. Biophysics Research Institute, Medical College of Wisconsin. https://afni.nimh.nih.gov/afni/doc/manual/3dfim+.pdf
- Can Product-Specific Assurance Case Templates Be Used as Medical Device Standards? IEEE Design & Test 32, 5 (2015), 45–55. https://doi.org/10.1109/MDAT.2015.2462720
- Automated empirical optimization of software and the ATLAS project. Parallel Comput. 27, 1–2 (2001), 3–35.
- Ross Whitehouse. 2018. Setting up ESLint in React. https://medium.com/@RossWhitehouse/setting-up-eslint-in-react-c20015ef35f7.
- Naming the Pain in Developing Scientific Software. IEEE Software (2019), 1–1. https://doi.org/10.1109/MS.2019.2899838
- Wikipedia. 2022. Lint (software). https://en.wikipedia.org/wiki/Lint_(software).
- Wikipedia contributors. 2021a. Fault injection — Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Fault_injection&oldid=1039005082 [Online; accessed 28-August-2021].
- Wikipedia contributors. 2021b. Fuzzing — Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Fuzzing&oldid=1039424308 [Online; accessed 28-August-2021].
- Wikipedia contributors. 2021c. Medical image computing — Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Medical_image_computing&oldid=1034877594 [Online; accessed 25-July-2021].
- Wikipedia contributors. 2021d. Medical imaging — Wikipedia, The Free Encyclopedia. https://en.wikipedia.org/w/index.php?title=Medical_imaging&oldid=1034887445 [Online; accessed 25-July-2021].
- Best Practices for Scientific Computing. PLoS Biol 12, 1 (Jan. 2014), e1001745. https://doi.org/10.1371/journal.pbio.1001745
- Good Enough Practices in Scientific Computing. CoRR abs/1609.00037 (2016). http://arxiv.org/abs/1609.00037
- Gregory V. Wilson. 2006. Where’s the Real Bottleneck in Scientific Computing? Scientists would do well to pick some tools widely used in the software industry. American Scientist 94, 1 (2006). http://www.americanscientist.org/issues/pub/wheres-the-real-bottleneck-in-scientific-computing
- Gert Wollny. 2020. Ginkgo CADx. https://github.com/gerddie/ginkgocadx. [Online; accessed 27-May-2021].
- User-Guided 3D Active Contour Segmentation of Anatomical Structures: Significantly Improved Efficiency and Reliability. Neuroimage 31, 3 (2006), 1116–1128.
- Moshe Zadka. 2018. How to open source your Python library. https://opensource.com/article/18/12/tips-open-sourcing-python-libraries.
- 1 - Medical Imaging. In Biomedical Information Technology, David Dagan Feng (Ed.). Academic Press, Burlington, 3–27. https://doi.org/10.1016/B978-012373583-6.50005-0
- Open Health Imaging Foundation Viewer: An Extensible Open-Source Framework for Building Web-Based Imaging Applications to Support Cancer Research. JCO Clinical Cancer Informatics 4 (2020), 336–345. https://doi.org/10.1200/CCI.19.00131 arXiv:https://doi.org/10.1200/CCI.19.00131 PMID: 32324447.