Sample Size Calculations for the Development of Risk Prediction Models that Account for Performance Variability

Abstract: Existing approaches to sample size calculations for developing clinical prediction models have focused on ensuring that the expected value of a chosen performance measure meets a pre-specified target. For example, to limit model-overfitting, the sample size is commonly chosen such that the expected calibration slope (CS) is 0.9, close to 1 for a perfectly calibrated model. In practice, due to sampling variability, model performance can vary considerably across different development samples of the recommended size. If this variability is high, the probability of obtaining a model with performance close to the target for a given measure may be unacceptably low. To address this, we propose an adapted approach to sample size calculations that explicitly incorporates performance variability by targeting the probability of acceptable performance (PrAP). For example, in the context of calibration, we may define a model as acceptably calibrated if CS falls in a pre-defined range, e.g. between 0.85 and 1.15. Then we choose the required sample size to ensure that PrAP(CS)=80%. For binary outcomes we implemented our approach for CS within a simulation-based framework via the R package `samplesizedev'. Additionally, for CS specifically, we have proposed an equivalent analytical calculation which is computationally efficient. While we focused on CS, the simulation-based framework is flexible and can be easily extended to accommodate other performance measures and types of outcomes. When adhering to existing recommendations, we found that performance variability increased substantially as the number of predictors, p, decreased. Consequently, PrAP(CS) was often low. For example, with 5 predictors, PrAP(CS) was around 50%. Our adapted approach resulted in considerably larger sample sizes, especially for p<10. Applying shrinkage tends to improve PrAP(CS).