Reproducibility, Replicability, and Transparency in Research: What 430 Professors Think in Universities across the USA and India (2402.08796v1)
Abstract: In the past decade, open science and science of science communities have initiated innovative efforts to address concerns about the reproducibility and replicability of published scientific research. In some respects, these efforts have been successful, yet there are still many pockets of researchers with little to no familiarity with these concerns, subsequent responses, or best practices for engaging in reproducible, replicable, and reliable scholarship. In this work, we survey 430 professors from Universities across the USA and India to understand perspectives on scientific processes and identify key points for intervention. Our findings reveal both national and disciplinary gaps in attention to reproducibility and replicability, aggravated by incentive misalignment and resource constraints. We suggest that solutions addressing scientific integrity should be culturally-centered, where definitions of culture should include both regional and domain-specific elements.
- Herman Aguinis and Angelo M Solarino. 2019. Transparency and replicability in qualitative research: The case of interviews with elite informants. Strategic Management Journal 40, 8 (2019), 1291–1315.
- Monya Baker. 2016a. 1,500 scientists lift the lid on reproducibility. Nature 533, 7604 (2016).
- Monya Baker. 2016b. Reproducibility crisis. Nature 533, 26 (2016), 353–66.
- Publications Output: U.S. Trends and International Comparisons. NSB-2023-33. Science and Engineering Indicators 2024. (2023).
- Qualitative HCI research: Going behind the scenes. Morgan & Claypool Publishers.
- Reducing bias and improving transparency in medical research: a critical overview of the problems, progress and suggested next steps. Journal of the Royal Society of Medicine 113, 11 (2020), 433–443.
- Evaluating replicability of laboratory experiments in economics. Science 351, 6280 (2016), 1433–1436.
- Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour 2, 9 (2018), 637–644.
- Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349, 6251 (2015), aac4716. https://doi.org/10.1126/science.aac4716 arXiv:https://www.science.org/doi/pdf/10.1126/science.aac4716
- Estimating the reproducibility of experimental philosophy. Review of Philosophy and Psychology 12 (2021), 9–44.
- Examining the generalizability of research findings from archival data. Proceedings of the National Academy of Sciences 119, 30 (2022), e2120377119.
- Florian Echtler and Maximilian Häußler. 2018. Open source, open science, and the replication crisis in HCI. In Extended abstracts of the 2018 CHI conference on human factors in computing systems. 1–8.
- Challenges for assessing replicability in preclinical cancer biology. Elife 10 (2021), e67995.
- An open investigation of the reproducibility of cancer biology research. Elife 3 (2014), e04333.
- Fiona Fidler and John Wilcox. 2018. Reproducibility of scientific results. (2018).
- Fraida Fund. 2023. We Need More Reproducibility Content Across the Computer Science Curriculum. In Proceedings of the 2023 ACM Conference on Reproducibility and Replicability. 97–101.
- Stephan Guttinger and Alan C Love. 2019. Characterizing scientific failure: putting the replication crisis in context. EMBO reports 20, 9 (2019), e48765.
- The extent and consequences of p-hacking in science. PLoS biology 13, 3 (2015), e1002106.
- Sayash Kapoor and Arvind Narayanan. 2023. Leakage and the reproducibility crisis in machine-learning-based science. Patterns 4, 9 (2023).
- J André Knottnerus and Peter Tugwell. 2016. Promoting transparency of research and data needs much more attention. Journal of clinical epidemiology 70 (2016), 1–3.
- Eric Loken and Andrew Gelman. 2017. Measurement error and the replication crisis. Science 355, 6325 (2017), 584–585.
- Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist 70, 6 (2015), 487.
- Athanasios Mazarakis and Paula Bräuer. 2020. Gamification of an open access quiz with badges and progress bars: An experimental study with scientists.. In GamiFIN. 62–71.
- Cormac McGrath and Gustav Nilsonne. 2018. Data sharing in qualitative research: Opportunities and concerns. MedEdPublish 7, 255 (2018), 255.
- The “replication crisis” in the public eye: Germans’ awareness and perceptions of the (ir) reproducibility of scientific research. Public Understanding of Science 30, 1 (2021), 91–102.
- Promoting transparency in social science research. Science 343, 6166 (2014), 30–31.
- Tsuyoshi Miyakawa. 2020. No raw data, no science: another possible source of the reproducibility crisis. , 6 pages.
- Reproducibility and replicability in science. National Academies Press.
- US News. 2023. Best National University Rankings 2023. https://www.usnews.com/best-colleges/rankings/national-universities
- Transparency and openness promotion (TOP) guidelines. (2016).
- Promoting an open research culture. Science 348, 6242 (2015), 1422–1425.
- Replicability, robustness, and reproducibility in psychological science. (2021).
- Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science 7, 6 (2012), 615–631.
- Analysis of open data and computational reproducibility in registered reports in psychology. Advances in Methods and Practices in Psychological Science 3, 2 (2020), 229–237.
- Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program). The Journal of Machine Learning Research 22, 1 (2021), 7459–7478.
- Edward Raff. 2023. Does the market of citations reward reproducible work?. In Proceedings of the 2023 ACM Conference on Reproducibility and Replicability. 89–96.
- Edward Raff and Andrew L Farris. 2023. A siren song of open source reproducibility, examples from machine learning. In Proceedings of the 2023 ACM Conference on Reproducibility and Replicability. 115–120.
- Home Ranking. 2018. National institutional ranking framework. Methodology for Ranking of Academic Institutions in India (Ranking Metrics for Medical) (2018).
- Anthony Rowe. 2023. Recommendations to improve use and reporting of statistics in animal experiments. Laboratory Animals 57, 3 (2023), 224–235.
- Did awarding badges increase data sharing in BMJ Open? A randomized controlled trial. Royal Society open science 7, 3 (2020), 191818.
- Anisa Rowhani-Farid and Adrian G Barnett. 2018. Badges for sharing data and code at Biostatistics: an observational study. F1000Research 7 (2018).
- Felix Schönbrodt and David Mellor. 2018. Academic job offers that mentioned open science. Open Science Framework.
- Jonathan W Schooler. 2014. Metascience could rescue the ‘replication crisis’. Nature 515, 7525 (2014), 9–9.
- Reproducibility in Machine Learning-Driven Research. arXiv preprint arXiv:2307.10320 (2023).
- Patrick E Shrout and Joseph L Rodgers. 2018. Psychology, science, and knowledge construction: Broadening perspectives from the replication crisis. Annual review of psychology 69 (2018), 487–510.
- A guide for social science journal editors on easing into open science. Research Integrity and Peer Review (2024).
- Transparency in qualitative research: Increasing fairness in the CHI review process. In Extended abstracts of the 2020 chi conference on human factors in computing systems. 1–6.
- Thematic analysis. The SAGE handbook of qualitative research in psychology 2 (2017), 17–37.
- UNESCO. 2021. UNESCO recommendation on open science. Zenodo (2021).
- Lars Vilhuber. 2020. Reproducibility and replicability in economics. Harvard Data Science Review 2, 4 (2020), 1–39.
- Craig Willis and Victoria Stodden. 2020. Trust but verify: How to leverage policies, workflows, and infrastructure to ensure computational reproducibility in publication. (2020).