Unique Information Decomposition

• Unique information decomposition is a framework that partitions mutual information into unique, redundant, and synergistic components to reveal how data from multiple predictors contributes to a target.
• It employs a formal bivariate method, using constrained optimization to minimize conditional mutual information and quantify specific decision-theoretic advantages.
• The framework supports efficient algorithmic computation and highlights challenges in extending to multivariate cases, impacting fields like neuroscience, communication theory, and machine learning.

Unique information decomposition is a framework for dissecting the mutual information between a set of predictor (source) variables and a target variable into constituent components—specifically, into unique, shared (redundant), and synergistic (complementary) parts. This structural disaggregation addresses fundamental questions about how information is distributed, lost, or synergistically generated among multiple variables, with profound implications for neuroscience, cryptography, machine learning, communication theory, and the foundations of information theory. Theoretical and algorithmic advances have been largely centered on the bivariate case (two predictors) due to deep challenges in extending to higher-order decompositions.

1. Formal Framework: Bivariate Decomposition

Consider three discrete random variables $X$, $Y$, and $Z$ with joint law $P(X,Y,Z)$. The unique information of $Y$ about $X$ with respect to $Z$, denoted $\widetilde{UI}(X\,;\,Y \setminus Z)$, is defined as the minimum conditional mutual information $MI_Q(X:Y|Z)$ over the set of joint distributions $\Delta_P$ that preserve the observed $Y$0 and $Y$1 pairwise marginals: $Y$2 where

$Y$3

Analogously, shared (redundant) and complementary (synergistic) information are defined by: $Y$4 where $Y$5 denotes coinformation, and $Y$6 denotes mutual information.

The decomposition holds: $Y$7 with the corresponding consistency equations for each marginal.

This framework, due to Bertschinger et al. (Bertschinger et al., 2013, Rauh et al., 2014), is both operationally and structurally motivated.

2. Key Properties and Interpretation

The unique, shared, and synergistic information quantities exhibit several essential properties (Rauh et al., 2014):

• Nonnegativity: All four quantities ($Y$8, $Y$9, $Z$0) are nonnegative.
• Marginal-invariance: Unique and shared information depend only on the observed pairwise marginals $Z$1 and $Z$2.
• Monotonicity: Unique information is monotonic in each argument—enlarging $Z$3 (adding predictors) cannot increase the unique information of $Z$4, while enlarging $Z$5 or $Z$6 cannot decrease it.
• Special cases:
  • For $Z$7 (XOR of independent bits): $Z$8, $Z$9 bit (pure synergy).
  • For $P(X,Y,Z)$0 determines $P(X,Y,Z)$1 and $P(X,Y,Z)$2 is independent: all information unique to $P(X,Y,Z)$3, with $P(X,Y,Z)$4.

Operationally, the unique information quantifies the advantage held in a decision problem where an agent has access to $P(X,Y,Z)$5 in place of $P(X,Y,Z)$6, or vice versa, formalized via the decision-theoretic Blackwell order (Bertschinger et al., 2013).

3. Algorithmic Computation and Numerical Methods

The unique information involves a constrained optimization (minimization of $P(X,Y,Z)$7) over the polytope $P(X,Y,Z)$8, which is typically high-dimensional (Banerjee et al., 2017). Convex optimization methods—including alternating divergence minimization and generalized iterative scaling—have been developed for efficient, convergent computation: $\widetilde{UI}(X\,;\,Y \setminus Z)$4 This method guarantees global convergence and is substantially faster than unconstrained solvers for system sizes up to $P(X,Y,Z)$9.

For continuous or parametric distributions (e.g., Gaussian, Poisson, multinomial), analytic solutions or adaptations exist when the structure of the marginal polytope allows one unique information term to vanish, massively simplifying the decomposition (Goswami et al., 2023).

4. Theoretical Limitations and Multivariate Decomposition

A central result shows that any attempt to extend the bivariate SI/UI/CI decomposition to more than two predictors via the Williams–Beer partial information lattice leads to an incompatibility: no redundancy function $Y$0 can simultaneously satisfy nonnegativity, monotonicity, symmetry, and the identity axiom for $Y$1 (Rauh et al., 2014).

Theorem (Rauh et al.): For $Y$2, any redundancy function $Y$3 designed to generalize bivariate SI/UI/CI with nonnegative atomic decomposition must violate at least one natural axiom.

This precludes a nonnegative information decomposition situated on the PI-lattice for $Y$4. The precise combinatorial structure enabling a consistent multivariate unique information decomposition remains an open problem.

Nevertheless, pairwise ("partitioned") unique information can still be meaningfully defined, and nonnegative inequalities such as

$Y$5

are conjectured to hold, offering a possible route to reconstructing atomic structure in larger systems (Rauh et al., 2014).

5. Operational and Decision-Theoretic Foundations

The decision-theoretic underpinning of unique information is rooted in the Blackwell order: $Y$6 carries no unique information about $Y$7 relative to $Y$8 if and only if $Y$9 is a garbling (randomized copy) of $X$0 in the sense of statistical decision theory (Bertschinger et al., 2013, Rauh et al., 2014). More specifically, unique information vanishes precisely when no decision-maker observing $X$1 can improve upon optimal choices based solely on $X$2 for any loss function.

The structure of the optimization set $X$3 ensures that $X$4, $X$5, and $X$6 depend only on the two relevant decision problems defined by the pairwise marginals, and that synergy is forced to zero for some $X$7, capturing the channel degradation structure (Rauh et al., 2014).

6. Examples and Interpretation of Decomposition

The following table summarizes several archetypal examples:

Distribution	$X$8	$X$9 (both)	$Z$0	Key Feature
$Z$1 (COPY)	$Z$2	$Z$3	$Z$4	All redundancy/unique
$Z$5 (XOR)	$Z$6	$Z$7	$Z$8	Pure synergy
$Z$9 (AND)	$\widetilde{UI}(X\,;\,Y \setminus Z)$0	$\widetilde{UI}(X\,;\,Y \setminus Z)$1	$\widetilde{UI}(X\,;\,Y \setminus Z)$2	Partial redundancy/synergy

These confirm the interpretive design: UI captures only information exclusive to a given predictor, SI the redundancy, and CI the higher-order synergy.

7. Practical Considerations and Open Directions

Practically, the unique information framework is robust in the bivariate setting, with efficient algorithms and a clear operational meaning. However, current limitations include:

• Non-uniqueness of solutions: For certain marginal cardinalities or conditional independence cases, the optimizer for $\widetilde{UI}(X\,;\,Y \setminus Z)$3 is not unique, resulting in ambiguity in the attribution of unique information (Rauh et al., 2019).
• Incomplete multivariate extensions: There is no general nonnegative decomposition into unique, redundant, and synergistic parts for more than two predictors based on the PI-lattice. Structuring the appropriate combinatorial or geometric foundation for the multivariate case remains unresolved (Rauh et al., 2014).
• Application to practice: Despite these theoretical obstacles, bivariate unique information can be computed for arbitrary partitions in multivariate systems, providing a partial picture of information flow in complex systems.

Current research seeks to overcome these barriers via richer combinatorial structures, and to establish new nonnegative atomic decompositions compatible with natural axioms for systems with three or more predictors (Rauh et al., 2014).

---

References:

• "Quantifying unique information" (Bertschinger et al., 2013)
• "Reconsidering unique information: Towards a multivariate information decomposition" (Rauh et al., 2014)
• "Computing the Unique Information" (Banerjee et al., 2017)
• "Properties of Unique Information" (Rauh et al., 2019)