Establish a practical standard for useful out-of-sample prediction

Develop a domain-appropriate standard for determining how large an out-of-sample R^2 must be to be useful, and express that standard as a target for which the study-design formulas can be solved.

Background

The paper derives expected out-of-sample R2 and correlation formulas for ordinary least squares and ridge regression under leave-one-out cross-validation. These formulas identify whether a design is expected to outperform the sample mean, but they do not determine whether the resulting predictive accuracy is substantively useful for a scientific or practical decision.

The authors note that clinical prediction and statistical genetics have domain-specific standards for judging predictive performance, whereas the social sciences lack a comparable benchmark. The unresolved problem is therefore to develop such a standard and formulate it in a way that can be incorporated into prospective design calculations.

References

The break-even threshold asks only whether a design can be expected to predict better than the sample mean; it does not say how large an out-of-sample $R2$ must be to be worth having. Fields that predict routinely have their own answers --- clinical prediction sets sample sizes so that the expected shrinkage of the fitted model is small \citep{riley2019a}, and statistical genetics judges a polygenic score against the heritability of the trait \citep{daetwyler2008,dudbridge2013} --- but the social sciences do not yet have a comparable standard. Developing one, and expressing it as a target the design formulas can be solved for, is a question for future research.

— Designing for Prediction: Planning Formulas for Expected Out-of-Sample $R^2$ and $r$  (2610.18781 - Tipton, 13 Oct 2026) in Section 5, Discussion, subsection “Limitations”

Whether an out-of-sample $R2$ of this size is large enough to be useful for any decision made from these predictions remains a separate question, one the design formulas cannot answer and to which we return in Section~\ref{sec:discussion}.

— Designing for Prediction: Planning Formulas for Expected Out-of-Sample $R^2$ and $r$  (2610.18781 - Tipton, 13 Oct 2026) in Section 4, An Illustration: Predicting Personality from Smartphone Behavior, subsection “A post-hoc analysis”