Entropy and Compression: A simple proof of an inequality of Khinchin-Ornstein-Shields (1907.04713v4)

Abstract: This paper concerns the folklore statement that ``entropy is a lower bound for compression''. More precisely we derive from the entropy theorem a simple proof of a pointwise inequality firstly stated by Ornstein and Shields and which is the almost-sure version of an average inequality firstly stated by Khinchin in 1953. We further give an elementary proof of original Khinchin inequality that can be used as an exercise for Information Theory students and we conclude by giving historical and technical notes of such inequality.