On the Capacity of Constrained Systems

Abstract: In the first chapter of Shannon's "A Mathematical Theory of Communication," it is shown that the maximum entropy rate of an input process of a constrained system is limited by the combinatorial capacity of the system. Shannon considers systems where the constraints define regular languages and uses results from matrix theory in his derivations. In this work, the regularity constraint is dropped. Using generating functions, it is shown that the maximum entropy rate of an input process is upper-bounded by the combinatorial capacity in general. The presented results also allow for a new approach to systems with regular constraints. As an example, the results are applied to binary sequences that fulfill the (j,k) run-length constraint and by using the proposed framework, a simple formula for the combinatorial capacity is given and a maxentropic input process is defined.