Papers
Topics
Authors
Recent
Search
2000 character limit reached

How to Tell When a Result Will Replicate: Significance and Replication in Distributional Null Hypothesis Tests

Published 4 Nov 2022 in stat.ME, math.ST, and stat.TH | (2211.02609v3)

Abstract: There is a well-known problem in Null Hypothesis Significance Testing: many statistically significant results fail to replicate in subsequent experiments. We show that this problem arises because standard point-form null' significance tests consider only within-experiment but ignore between-experiment variation, and so systematically underestimate the degree of random variation in results. We give an extension to standard significance testing that addresses this problem by analysing both within- and between-experiment variation. Thisdistributional null' approach does not underestimate experimental variability and so is not overconfident in identifying significance; because this approach addresses between-experiment variation, it gives mathematically coherent estimates for the probability of replication of significant results. Using a large-scale replication dataset (the first Many Labs' project), we show that many experimental results that appear statistically significant in standard tests are in fact consistent with random variation when both within- and between-experiment variation are taken into account in this approach. Further, grouping experiments in this dataset intopredictor-target' pairs we show that the predicted replication probabilities for target experiments produced in this approach (given predictor experiment results and the sample sizes of the two experiments) are strongly correlated with observed replication rates. Distributional null hypothesis testing thus gives researchers a statistical tool for identifying statistically significant and reliably replicable results.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. Remove, Rather Than Redefine, Statistical Significance. Nature Human Behaviour, 2(1):4.
  2. The Intrinsic Bayes Factor for Model Selection and Prediction. Journal of the American Statistical Association, 91(433):109–122.
  3. Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
  4. Evaluating the Replicability of Social Science Experiments in Nature and Science Between 2010 and 2015. Nature Human Behaviour, 2(9):637–644.
  5. Carver, R. (1978). The Case Against Statistical Significance Testing. Harvard Educational Review, 48(3):378–399.
  6. Christensen, R. (2005). Testing Fisher, Neyman, Pearson, and Bayes. The American Statistician, 59(2):121–126.
  7. Cohen, J. (1992). A Power Primer. Psychological Bulletin, 112(1):155.
  8. Cohen, J. (2013). Statistical Power Analysis for the Behavioral Sciences. Routledge.
  9. Costello, F. (2022). Significance and replication with distributional null hypothesis testing. http://osf.io/ep86y.
  10. Xtable: Export Tables to LaTeX or HTML. R package version 1.8-4.
  11. Unified conditional frequentist and Bayesian testing of composite hypotheses. Scandinavian Journal of Statistics, 30(1):193–210.
  12. Diaconis, P. et al. (1991). [Replication and Meta-Analysis in Parapsychology]: Comment. Statistical Science, 6(4):386–386.
  13. Probability of Replication Revisited: Comment on An Alternative to Null-Hypothesis Significance Tests. Psychological Science, 16(12):1005–1006.
  14. Effect Size Estimates: Current Use, Calculations, and Interpretation. Journal of Experimental Psychology: General, 141(1):2.
  15. Visualization in Bayesian workflow. Journal of the Royal Statistical Society Series A: Statistics in Society, 182(2):389–402.
  16. The Bayesian Two-Sample T Test. The American Statistician, 59(3):252–257.
  17. Hedges, L. V. (1983). A Random Effects Model for Effect Sizes. Psychological Bulletin, 93(2):388.
  18. Hunter, J. E. (1997). Needed: A Ban on the Significance Test. Psychological Science, 8(1):3–7.
  19. Prep Misestimates the Probability of Replication. Psychonomic Bulletin & Review, 16(2):424–429.
  20. The Random Effects Prep Continues to Mispredict the Probability of Replication. Psychonomic Bulletin & Review, 17(2):270–272.
  21. Prep: An Agony in Five Fits. Journal of Mathematical Psychology, 53(4):195–202.
  22. A Model-Averaging Approach to Replication: The Case of Prep. Psychological Methods, 15(2):172.
  23. On Effect Size. Psychological Methods, 17(2):137.
  24. Killeen, P. R. (2005a). An Alternative to Null-Hypothesis Significance Tests. Psychological Science, 16(5):345–353.
  25. Killeen, P. R. (2005b). Replicability, Confidence, and Priors. Psychological Science, 16(12):1009–1012.
  26. Killeen, P. R. (2007). Replication Statistics. Best Practices in Quantitative Methods, pages 103–124.
  27. Investigating Variation in Replicability. Social Psychology, 45(3):142–152.
  28. Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances in Methods and Practices in Psychological Science, 1(4):443–490.
  29. Translating the Statistical Representation of the Effects of Education Interventions Into More Readily Interpretable Forms. National Center for Special Education Research.
  30. Harold Jeffreys?s Default Bayes Factor Hypothesis Tests: Explanation, Extension, and Application in Psychology. Journal of Mathematical Psychology, 72:19–32.
  31. Macdonald, R. R. (2005). Why Replication Probabilities Depend on Prior Probability Distributions. Psychological Science-Cambridge-, 16(12):1007.
  32. Mandel, J. (2012). The Statistical Analysis of Experimental Data. Courier Corporation.
  33. Killeen’s (2005) Prep Coefficient: Logical and Mathematical Problems. Psychological Methods, 15(2):182.
  34. Abandon Statistical Significance. The American Statistician, 73(sup1):235–245.
  35. Meehl, P. E. (1990a). Appraising and Amending Theories: The Strategy of Lakatosian Defense and Two Principles That Warrant It. Psychological Inquiry, 1(2):108–141.
  36. Meehl, P. E. (1990b). Why Summaries of Research on Psychological Theories Are Often Uninterpretable. Psychological Reports, 66(1):195–244.
  37. Miller, J. (2009). What Is the Probability of Replicating a Statistically Significant Effect? Psychonomic Bulletin & Review, 16(4):617–640.
  38. Aggregate and Individual Replication Probability Within an Explicit Model of the Research Process. Psychological Methods, 16(3):337.
  39. BayesFactor: Computation of Bayes Factors for Common Designs. R package version 0.9.12-4.3.
  40. Murphy, K. P. (2007). Conjugate Bayesian Analysis of the Gaussian Distribution. Technical report, University of British Columbia.
  41. Open Science Collaboration et al. (2015). Estimating the Reproducibility of Psychological Science. Science, 349(6251):aac4716.
  42. Crud (Re) Defined. Advances in Methods and Practices in Psychological Science, 3(2):238–247.
  43. Editors’ Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? Perspectives on Psychological Science, 7(6):528–530.
  44. R Core Team (2021). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.
  45. Bayesian T Tests for Accepting and Rejecting the Null Hypothesis. Psychonomic Bulletin & Review, 16(2):225–237.
  46. Better Statistics for Better Decisions: Rejecting Null Hypotheses Statistical Tests in Favor of Replication Statistics. Psychology in the Schools, 44(5):471–481.
  47. Toward a principled Bayesian workflow in cognitive science. Psychological Methods, 26(1):103.
  48. Statistical Methods.
  49. Thompson, B. (1998). In Praise of Brilliance: Where That Praise Really Belongs. American Psychologist, 53(7):799–800.
  50. Bayesian Statistics and Modelling. Nature Reviews Methods Primers, 1(1):1–26.
  51. Methods to Estimate the Between-Study Variance and Its Uncertainty in Meta-Analysis. Research Synthesis Methods, 7(1):55–79.
  52. Viechtbauer, W. (2005). Bias and Efficiency of Meta-Analytic Variance Estimators in the Random-Effects Model. Journal of Educational and Behavioral Statistics, 30(3):261–293.
  53. Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic bulletin & review, 14(5):779–804.
  54. Bayesian inference for psychology. part i: Theoretical advantages and practical ramifications. Psychonomic bulletin & review, 25:35–57.
  55. Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 T Tests. Perspectives on Psychological Science, 6(3):291–298.
  56. Wickham, H. (2016). Ggplot2: Elegant Graphics for Data Analysis. Springer-Verlag New York.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free