--- title: 'Geometric PID: Bivariate Info Decomposition' url: https://www.emergentmind.com/topics/geometric-partial-information-decomposition-pid type: topic --- # Geometric PID: Bivariate Info Decomposition Geometric Partial Information Decomposition (PID) is an information-theoretic framework designed to disentangle the contributions of multiple information sources to a target variable in terms of redundancy, unique information, and synergy. The Geometric PID formalism offers an operational and mathematically principled construction of redundancy for bivariate systems, rooted in the geometry of probability distributions and Kullback–Leibler (KL) projections. It is notable for its rigorous axiomatic foundation, clear geometric interpretation, and explicit computability, though it is inherently restricted to systems with exactly two sources [2603.06678]. ## 1. Formal Definition of Geometric PID Consider two discrete source variables $X_1$, $X_2$ and a target $Y$ with joint distribution $P(x_1, x_2, y)$. For each $x_1$ in the support of $X_1$, the conditional distribution $p_{y|x_1} := P(Y=y|X_1=x_1)$ is viewed as a point in the probability simplex $\Delta_Y$. Similarly, for $x_2$ in the support of $X_2$, $p_{y|x_2}$ is defined. Define for $X_2$ the convex hull $C_2 = \operatorname{conv}\{ p_{y|x_2} : x_2 \in \operatorname{supp}(X_2) \}$ and analogously $C_1$ for $X_1$. The information projection (I-projection) of $p_{y|x_1}$ onto $C_2$ is $$ \pi_{C_2}(p_{y|x_1}) = \arg\min_{r \in C_2} D_{KL}\left[ p_{y|x_1} \,\|\, r \right], $$ yielding the projected conditional $p_{(x_1 \to X_2)}(y)$. The directed projected information from $X_1$ into $X_2$ is then $$ I_Y^\pi(X_1 \to X_2) = \sum_{x_1, y} P(x_1,y) \log \frac{ p_{(x_1 \to X_2)}(y) }{ P(y) }. $$ Redundant information is given by $$ I^{\text{Geom}}_\cap(X_1, X_2; Y) = \min \left\{ I_Y^\pi(X_1 \to X_2),\; I_Y^\pi(X_2 \to X_1) \right\}. $$ The atoms of the PID lattice are then given by Möbius inversion: - Redundancy: $R = I^{\text{Geom}}_\cap(X_1, X_2; Y)$ - Unique information of $X_1$: $U_1 = I(X_1; Y) - R$ - Unique information of $X_2$: $U_2 = I(X_2; Y) - R$ - Synergy: $S = I(X_1, X_2; Y) - R - U_1 - U_2$ ## 2. Geometric Interpretation and Information Projection Each conditional $p_{y|x_i}$ ($i=1,2$) can be interpreted as a point on the $(|Y|-1)$-simplex. The set of conditionals $\{ p_{y|x_2} \}$ spans a convex polytope $C_2$. The projection $\pi_{C_2}(p_{y|x_1})$ finds the point in $C_2$ that is closest (in the KL sense) to $p_{y|x_1}$. Intuitively, this projects the information that $X_1$ has about $Y$ onto the “statistical structure” available from $X_2$. This geometry underpins the “shared” content: only information already expressible by $X_2$ conditionals is counted as redundant. Symmetry is enforced by minimizing the directed projections in both possible directions. ## 3. Computational Workflow Computation of the Geometric PID proceeds as follows [2603.06678]: 1. **Marginal and Conditional Computation**: Compute $p(y)$, $p(x_1, y)$, $p(x_2, y)$, then the conditionals $p_{y|x_1}$ and $p_{y|x_2}$. 2. **Convex Hull Construction**: Form $C_2$ as the convex hull of $\{ p_{y|x_2} \}$. For each $x_1$, solve the convex projection (e.g., using Blahut–Arimoto or gradient methods) to find $\pi_{C_2}(p_{y|x_1})$. 3. **Projected Information Calculation**: Compute $I_Y^\pi(X_1 \to X_2)$ using the projected conditionals. 4. **Symmetry Step**: Repeat for $I_Y^\pi(X_2 \to X_1)$. 5. **Redundancy and Atom Derivation**: Assign $R = \min \{I_Y^\pi(X_1 \to X_2), I_Y^\pi(X_2 \to X_1)\}$ and derive $U_1, U_2, S$ as above. The following table summarizes the definitions of the bivariate PID atoms: | Atom | Formula | Description | |:----------:|:---------------------------------------------------------------:|:-----------------------------------------------| | Redundancy | $R = I^{\text{Geom}}_\cap$ | Information shared by $X_1$, $X_2$ about $Y$ | | Unique $X_1$ | $U_1 = I(X_1; Y) - R$ | Unique information of $X_1$ | | Unique $X_2$ | $U_2 = I(X_2; Y) - R$ | Unique information of $X_2$ | | Synergy | $S = I(X_1,X_2;Y) - R - U_1 - U_2$ | Information only available jointly | ## 4. Axiomatic Properties and Limiting Results The Geometric redundancy $I^{\text{Geom}}_\cap$ satisfies the following axioms and properties: - **Self-redundancy (SR)**: $I^{\text{Geom}}_\cap(X_i; Y) = I(X_i; Y)$ - **(Weak) Symmetry (S₀)**: Invariant under swapping $X_1 \leftrightarrow X_2$ - **(Weak) Monotonicity (M₀)**: Redundancy does not increase when adding a source, $R \leq I(X_i; Y)$ - **Subset-Equality (SE)**: If $X_2 = f(X_1)$ then $R(X_1,X_2;Y) = I(X_1;Y)$ - **Nonnegativity (GP)**: $R \geq 0$ - **Local Positivity (LP)**: All PID atoms are nonnegative - **Identity (ID)**: For $Y=(X_1,X_2)$, $R = I(X_1;X_2)$ - **Independent-Identity (IID)**: If $X_1 \perp X_2$, $R(X_1,X_2;Y=X_1X_2)=0$ - **Lower Bound (LB)**: Redundancy lower-bounded by less-informative surrogates - **Equivalence-Invariance (EI)**: Invariant to relabeling of variable values Crucially, several no-go results establish that Geometric PID cannot be consistently extended to more than two sources while retaining all the aforementioned properties plus chain-rule (TC) or target monotonicity (TM). Indeed, Geometric PID fails TM/TC: adding more of $Y$ can decrease redundancy. ## 5. Illustrative Example: XOR Gate For $X_1$ and $X_2$ independent fair bits, $Y = X_1 \oplus X_2$: - $p(y=0|x_i) = p(y=1|x_i) = \frac{1}{2}$ for $i=1,2$; thus $p_{y|x_i}$ for all $x_i$ is the simplex center. - The projections $\pi_{C_2}(p_{y|x_1}) = (1/2, 1/2)$, so $I_Y^\pi(X_1 \to X_2) = 0$. - $R=0$, $U_1=U_2=0$, $I(X_1,X_2;Y)=1$ bit, so $S=1$ bit: all information is synergistic, no redundancy. This aligns with the expected behavior for the XOR structure [2603.06678]. ## 6. Advantages, Limitations, and Applications ### Advantages - **Identity Validity**: Satisfies the ID axiom; independent copies do not yield spurious redundancy. - **Nonnegativity and Interpretability**: All PID atoms are nonnegative and possess a geometric interpretation as KL projections. - **Label Invariance**: Equivalence-invariant under invertible relabeling of variable values. ### Limitations - **Bivariate Only**: Formalism is restricted to two sources; no generalization exists to higher dimensions that preserves all core properties and ID. - **Violation of Target-Monotonicity**: Adding more of $Y$ can decrease redundancy (TM fails). - **Computational Overhead**: For large support on $Y$, repeated convex optimizations may become computationally expensive. ### Use Cases - **Bivariate PID**: Settings where two source variables are analyzed for contributions to a target. - **Contexts Demanding Identity and Nonnegativity**: Experimental systems needing strict adherence to these axioms. - **Low-dimensional Targets**: These facilitate practical geometric projection computation. ## 7. Relation to Alternative Geometric PID Approaches A related but distinct geometric PID approach leverages information geometry over partially ordered sets (posets) of variable subsets [1601.05533]. This framework generalizes Amari's hierarchy to enable decomposition on structured spaces, constructing a dually-flat manifold (with $e$- and $m$-coordinates) for arbitrary posets and deriving PID atoms through Möbius inversion on KL divergence projections. While more general and multivariate, the practical and conceptual constraints differ from the bivariate-focused Geometric PID defined by Harder et al. Thus, users should be cautious to distinguish these two flavors of "geometric" PID, as only the latter corresponds precisely to the KL-projection and simplex geometry described in [2603.06678]. Source: https://www.emergentmind.com/topics/geometric-partial-information-decomposition-pid