Bootstrap Cross-validation Improves Model Selection in Pharmacometrics

Abstract: Cross-validation assesses the predictive ability of a model, allowing one to rank models accordingly. Although the nonparametric bootstrap is almost always used to assess the variability of a parameter, it can be used as the basis for cross-validation if one keeps track of which items were not selected in a given bootstrap iteration. The items which were selected constitute the training data and the omitted items constitute the testing data. This bootstrap cross-validation (BS-CV) allows model selection to be made on the basis of predictive ability by comparing the median values of ensembles of summary statistics of testing data. BS-CV is herein demonstrated using several summary statistics, including a new one termed the simple metric for prediction quality (SMPQ), and using the warfarin data included in the Monolix distribution with 13 pharmacokinetics (PK) models and 12 pharmacodynamics (PD) models. Of note the two best PK models by AIC had the worst predictive ability, underscoring the danger of using single realizations of a random variable (such as AIC) as the basis for model selection. Using these data BS-CV was able to discriminate between similar indirect response models (inhibition of input versus stimulation of output). This could be useful in situations in which the mechanism of action is unknown (unlike warfarin).