The Discordance of Individual Risk Estimates and the Reference Class Problem

Abstract: Multivariate methods that relate outcomes to risk factors have been adopted clinically to individualize treatment. This has promoted the belief that individuals have a true or unique risk. The logic of assigning an individual a single risk value has been criticized since 1866. The reason is that any individual can be simultaneously considered a member of different groups, with each group having its own risk level (the reference class problem). Lemeshow et al. provided well-documented examples of remarkable discordance between predictions for an individual by different valid predictive methods utilizing different risk factors. The prevalence of such discordance is unknown as it is rarely evaluated, but must be substantial due to the abundance of risk factors. Lemeshow et al. cautioned against using ICU mortality predictions for the provision of care to individual patients. If individual risk estimates are used clinically, users should be aware that valid methods may give very different results.