2000 character limit reached
No Free Lunch versus Occam's Razor in Supervised Learning (1111.3846v1)
Published 16 Nov 2011 in cs.LG, cs.IT, and math.IT
Abstract: The No Free Lunch theorems are often used to argue that domain specific knowledge is required to design successful algorithms. We use algorithmic information theory to argue the case for a universal bias allowing an algorithm to succeed in all interesting problem domains. Additionally, we give a new algorithm for off-line classification, inspired by Solomonoff induction, with good performance on all structured problems under reasonable assumptions. This includes a proof of the efficacy of the well-known heuristic of randomly selecting training data in the hope of reducing misclassification rates.