"How trustworthy is this research?" Designing a Tool to Help Readers Understand Evidence and Uncertainty in Science Journalism

Abstract: This article reports on a Research through Design study exploring how to design a tool for helping readers of science journalism understand the strength and uncertainty of scientific evidence in news stories about health science, using both textual and visual information. A central aim has been to teach readers about criteria for assessing scientific evidence, in particular in order to help readers differentiate between science and pseudoscience. Working in a research-in-the-wild collaboration with a website for popular science, the study presents the design and evaluation of the Scientific Evidence Indicator, which uses metadata about scientific publications to present an assessment of evidence strength to the readers. Evaluations of the design demonstrate some success in helping readers recognize whether studies have undergone scientific peer review or not, but point to challenges in facilitating a more in-depth understanding. Insights from the study point to a potential for developing similar tools aimed at journalists rather than directly at audiences.