Some new insights into information decomposition in complex systems based on common information

Abstract: We take a closer look at the structure of bivariate dependency induced by a pair of predictor random variables $(X_1, X_2)$ trying to synergistically, redundantly or uniquely encode a target random variable $Y$. We evaluate a recently proposed measure of redundancy based on the G\'acs-K\"{o}rner common information (Griffith et al., Entropy 2014) and show that the measure, in spite of its elegance is degenerate for most non-trivial distributions. We show that Wyner's common information also fails to capture the notion of redundancy as it violates an intuitive monotonically non-increasing property. We identify a set of conditions when a conditional version of G\'acs and K\"{o}rner's common information is an ideal measure of unique information. Finally, we show how the notions of approximately sufficient statistics and conditional information bottleneck can be used to quantify unique information.