Equivalence Principle of the $P$-value and Mutual Information

Abstract: In this paper, we propose a novel equivalence between probability theory and information theory. For a single random variable, Shannon's self-information, $I=-\log {p}$, is an alternative expression of a probability $p$. However, for two random variables, no information equivalent to the $p$-value has been identified. Here, we prove theorems that demonstrate that mutual information (MI) is equivalent to the $p$-value irrespective of prior information about the distribution of the variables. If the maximum entropy principle can be applied, our equivalence theorems allow us to readily compute the $p$-value from multidimensional MI. By contrast, in a contingency table of any size with known marginal frequencies, our theorem states that MI asymptotically coincides with the logarithm of the $p$-value of Fisher's exact test, divided by the sample size. Accordingly, the theorems enable us to perform a meta-analysis to accurately estimate MI with a low $p$-value, thereby calculating informational interdependence that is robust against sample size variation. Thus, our theorems demonstrate the equivalence of the $p$-value and MI at every dimension, use the merits of both, and provide fundamental information for integrating probability theory and information theory.