Shannon Information Capacity of Discrete Synapses

Abstract: There is evidence that biological synapses have only a fixed number of discrete weight states. Memory storage with such synapses behaves quite differently from synapses with unbounded, continuous weights as old memories are automatically overwritten by new memories. We calculate the storage capacity of discrete, bounded synapses in terms of Shannon information. For optimal learning rules, we investigate how information storage depends on the number of synapses, the number of synaptic states and the coding sparseness.