Measuring impact in research evaluations: A thorough discussion of methods for, effects of, and problems with impact measurements

Abstract: Impact of science is one of the most important topics in scientometrics. Recent developments show a fundamental change in impact measurements from impact on science to impact on society. Since impact measurement is currently in a state of far reaching changes, this paper describes recent developments and facing problems in this area. For that the results of key publications (dealing with impact measurement) are discussed. The paper discusses how impact is generally measured within science and beyond (section 2), which effects impact measurements have on the science system (section 3), and which problems are associated with impact measurement (section 4). The problems associated with impact measurement constitute the focus of this paper: Science is marked by inequality, random chance, anomalies, the right to make mistakes, unpredictability, and a high significance of extreme events, which might distort impact measurements. Scientometricians as the producer of impact scores and decision makers as their consumers should be aware of these problems and should consider them in the generation and interpretation of bibliometric results, respectively.