Statistical Reliability of 10 Years of Cyber Security User Studies (Extended Version)

Abstract: Background. In recent years, cyber security security user studies have been appraised in meta-research, mostly focusing on the completeness of their statistical inferences and the fidelity of their statistical reporting. However, estimates of the field's distribution of statistical power and its publication bias have not received much attention. Aim. In this study, we aim to estimate the effect sizes and their standard errors present as well as the implications on statistical power and publication bias. Method. We built upon a published systematic literature review of $146$ user studies in cyber security (2006--2016). We took into account $431$ statistical inferences including $t$-, $\chi^2$-, $r$-, one-way $F$-tests, and $Z$-tests. In addition, we coded the corresponding total sample sizes, group sizes and test families. Given these data, we established the observed effect sizes and evaluated the overall publication bias. We further computed the statistical power vis-{`a}-vis of parametrized population thresholds to gain unbiased estimates of the power distribution. Results. We obtained a distribution of effect sizes and their conversion into comparable log odds ratios together with their standard errors. We, further, gained funnel-plot estimates of the publication bias present in the sample as well as insights into the power distribution and its consequences. Conclusions. Through the lenses of power and publication bias, we shed light on the statistical reliability of the studies in the field. The upshot of this introspection is practical recommendations on conducting and evaluating studies to advance the field.