Standard meta-analysis methods are not robust (2110.14511v1)

Abstract: P values or risk ratios from multiple, independent studies, observational or randomized, can be computationally combined to provide an overall assessment of a research question in meta-analysis. There is a need to examine the reliability of these methods of combination. It is typical in observational studies to statistically test many questions and not correct the analysis results for multiple testing or multiple modeling, MTMM. The same problem can happen for randomized, experimental trials. There is the additional problem that some of the base studies may be using fabricated or fraudulent data. If there is no attention to MTMM or fraud in the base studies, there is no guarantee that the results to be combined are unbiased, the key requirement for the valid combining of results. We note that methods of combination are not robust; even one extreme base study value can overwhelm standard methods of combination. It is possible that multiple, extreme (MTMM or fraudulent) results can feed from the base studies to bias the combined result. A meta-analysis of observational (or even randomized studies) may not be reliable. Examples are given along with some methods to evaluate existing base studies and meta-analysis studies.