A critical assessment of conformal prediction methods applied in binary classification settings

Abstract: In recent years there has been an increase in the number of scientific papers that suggest using conformal predictions in drug discovery. We consider that some versions of conformal predictions applied in binary settings are embroiled in pitfalls, not obvious at first sight, and that it is important to inform the scientific community about them. In the paper we first introduce the general theory of conformal predictions and follow with an explanation of the versions currently dominant in drug discovery research today. Finally, we provide cases for their critical assessment in binary classification settings.