O-information: Redundancy and Synergy

• O-information is a multivariate measure defined as the difference between total correlation and dual total correlation, distinguishing redundancy from synergy.
• It employs entropy-based formulations and gradient analyses to estimate high-order informational couplings in both discrete and continuous data.
• O-information has been applied across neuroscience, macroeconomics, and machine learning to uncover nuanced group interactions and dynamic system behaviors.

O-information is a multivariate information-theoretic quantity that quantifies the overall balance between redundancy- and synergy-dominated dependencies in groups of $n \ge 3$ variables. It is defined as the difference between total correlation and dual total correlation, so that positive values indicate redundancy-dominated interactions and negative values indicate synergy-dominated interactions. In the triplet case, O-information coincides with interaction information (co-information), which anchors its interpretation to the classical three-variable redundancy–synergy dichotomy (Rosas et al., 2019, Scagliarini et al., 2022).

1. Formal definition and sign semantics

For random variables $(X_1,\ldots,X_n)$, total correlation and dual total correlation are defined as

$$\mathrm{TC}(X_1,\ldots,X_n)=\sum_{i=1}^n H(X_i)-H(X_1,\ldots,X_n),$$

and

$$\mathrm{DTC}(X_1,\ldots,X_n)=H(X_1,\ldots,X_n)-\sum_{i=1}^n H(X_i\mid X_{-i}), \qquad X_{-i}=(X_1,\ldots,X_{i-1},X_{i+1},\ldots,X_n).$$

The O-information is

$$\Omega(X_1,\ldots,X_n)=\mathrm{TC}(X_1,\ldots,X_n)-\mathrm{DTC}(X_1,\ldots,X_n).$$

An equivalent entropy-only form is

$$\Omega(\mathbf{X}^n)=(n-2)H(\mathbf{X}^n)+\sum_{i=1}^n\big[H(X_i)-H(\mathbf{X}^n_{-i})\big].$$

These equivalent forms are standard in the literature and motivate both discrete and continuous estimators (Rosas et al., 2019, Scagliarini et al., 2022).

The sign of $\Omega$ is the central interpretive device. If $\Omega>0$, the system is redundancy dominated; if $\Omega<0$, it is synergy dominated; and if $\Omega=0$, redundancy and synergy are balanced or absent. In the triplet case,

$(X_1,\ldots,X_n)$0

so O-information reduces exactly to interaction information. This identity is used widely because it preserves the classical interpretation of positive values as overlapping, coordinated information and negative values as complementary, distributed information not reducible to lower-order parts (Pomarico et al., 31 Jul 2025).

The components entering $(X_1,\ldots,X_n)$1 admit complementary interpretations. In the discrete multivariate setting, total correlation quantifies collective constraints, whereas dual total correlation is also termed binding information or shared randomness. In Gaussian closed-form treatments, dual total correlation is also interpreted as total partial correlation/coherence through the standardized inverse covariance matrix $(X_1,\ldots,X_n)$2, yielding $(X_1,\ldots,X_n)$3 in the complex case and $(X_1,\ldots,X_n)$4 in the real case (Rosas et al., 2019, Pascual-Marqui et al., 11 Jul 2025).

2. Canonical regimes, bounds, and structural properties

Several basic regimes fix the interpretation of O-information. If the variables are mutually independent, then both total correlation and dual total correlation vanish, hence $(X_1,\ldots,X_n)$5. For $(X_1,\ldots,X_n)$6, O-information is identically zero, so nontrivial redundancy–synergy balance begins only at order three. If all variables are perfectly redundant copies, then $(X_1,\ldots,X_n)$7; if they form an XOR-like parity relation, then $(X_1,\ldots,X_n)$8 (Rosas et al., 2019, Scagliarini et al., 2021).

These canonical constructions also realize the extremal bounds. For finite alphabets of size $(X_1,\ldots,X_n)$9,

$$\mathrm{TC}(X_1,\ldots,X_n)=\sum_{i=1}^n H(X_i)-H(X_1,\ldots,X_n),$$0

For balanced binary variables, the $$\mathrm{TC}(X_1,\ldots,X_n)=\sum_{i=1}^n H(X_i)-H(X_1,\ldots,X_n),$$1-copy construction attains the upper extremum and the $$\mathrm{TC}(X_1,\ldots,X_n)=\sum_{i=1}^n H(X_i)-H(X_1,\ldots,X_n),$$2-XOR construction attains the lower extremum. In the copy case, $$\mathrm{TC}(X_1,\ldots,X_n)=\sum_{i=1}^n H(X_i)-H(X_1,\ldots,X_n),$$3 bits; in the XOR case, $$\mathrm{TC}(X_1,\ldots,X_n)=\sum_{i=1}^n H(X_i)-H(X_1,\ldots,X_n),$$4 bits. These examples are important because they show that O-information scales with system size in a way that preserves the direction of redundancy or synergy (Rosas et al., 2019, Scagliarini et al., 2021).

O-information is permutation-invariant, because both total correlation and dual total correlation are permutation-symmetric. It is additive over independent subsystems: if the joint distribution factorizes into independent blocks, then the total O-information is the sum of the blockwise O-informations. In particular, if a system factorizes into disjoint pairs, then $$\mathrm{TC}(X_1,\ldots,X_n)=\sum_{i=1}^n H(X_i)-H(X_1,\ldots,X_n),$$5, which formalizes the statement that O-information is blind to purely pairwise organization when no genuine higher-order coupling remains (Rosas et al., 2019).

A recurrent interpretive caveat is that $$\mathrm{TC}(X_1,\ldots,X_n)=\sum_{i=1}^n H(X_i)-H(X_1,\ldots,X_n),$$6 is a net index. A value near zero can arise because higher-order effects are absent, because the distribution decomposes into independent subsets of size at most two, or because redundancy- and synergy-dominated contributions cancel. Likewise, the local O-information literature emphasizes that global bounds do not necessarily constrain pointwise values: local pattern-level values can exceed the global finite-alphabet scale in magnitude, even though their expectation equals the global O-information (Scagliarini et al., 2021).

3. Local, gradient, structured, and dynamic generalizations

Several extensions were introduced to localize or specialize the redundancy–synergy balance.

The local O-information assigns a value to each observed pattern $$\mathrm{TC}(X_1,\ldots,X_n)=\sum_{i=1}^n H(X_i)-H(X_1,\ldots,X_n),$$7:

$$\mathrm{TC}(X_1,\ldots,X_n)=\sum_{i=1}^n H(X_i)-H(X_1,\ldots,X_n),$$8

with the equivalent form

$$\mathrm{TC}(X_1,\ldots,X_n)=\sum_{i=1}^n H(X_i)-H(X_1,\ldots,X_n),$$9

Its expectation recovers the global O-information, $$\mathrm{DTC}(X_1,\ldots,X_n)=H(X_1,\ldots,X_n)-\sum_{i=1}^n H(X_i\mid X_{-i}), \qquad X_{-i}=(X_1,\ldots,X_{i-1},X_{i+1},\ldots,X_n).$$0, which makes it a genuine pattern-level decomposition of the global quantity (Scagliarini et al., 2021).

The gradients of O-information localize high-order effects to variables and pairs. The first-order gradient is

$$\mathrm{DTC}(X_1,\ldots,X_n)=H(X_1,\ldots,X_n)-\sum_{i=1}^n H(X_i\mid X_{-i}), \qquad X_{-i}=(X_1,\ldots,X_{i-1},X_{i+1},\ldots,X_n).$$1

so $$\mathrm{DTC}(X_1,\ldots,X_n)=H(X_1,\ldots,X_n)-\sum_{i=1}^n H(X_i\mid X_{-i}), \qquad X_{-i}=(X_1,\ldots,X_{i-1},X_{i+1},\ldots,X_n).$$2 means that adding $$\mathrm{DTC}(X_1,\ldots,X_n)=H(X_1,\ldots,X_n)-\sum_{i=1}^n H(X_i\mid X_{-i}), \qquad X_{-i}=(X_1,\ldots,X_{i-1},X_{i+1},\ldots,X_n).$$3 builds redundancy-dominated circuits, whereas $$\mathrm{DTC}(X_1,\ldots,X_n)=H(X_1,\ldots,X_n)-\sum_{i=1}^n H(X_i\mid X_{-i}), \qquad X_{-i}=(X_1,\ldots,X_{i-1},X_{i+1},\ldots,X_n).$$4 means that adding $$\mathrm{DTC}(X_1,\ldots,X_n)=H(X_1,\ldots,X_n)-\sum_{i=1}^n H(X_i\mid X_{-i}), \qquad X_{-i}=(X_1,\ldots,X_{i-1},X_{i+1},\ldots,X_n).$$5 fosters synergy-dominated interdependencies. Second-order gradients quantify the non-additive effect of jointly including a pair $$\mathrm{DTC}(X_1,\ldots,X_n)=H(X_1,\ldots,X_n)-\sum_{i=1}^n H(X_i\mid X_{-i}), \qquad X_{-i}=(X_1,\ldots,X_{i-1},X_{i+1},\ldots,X_n).$$6 beyond their separate contributions, and higher-order gradients follow an inclusion–exclusion chain rule (Scagliarini et al., 2022).

Structured O-information addresses grouped variables. If variables are partitioned into non-overlapping groups and within-group interactions should be discounted, then structured total correlation and structured dual total correlation are built from $$\mathrm{DTC}(X_1,\ldots,X_n)=H(X_1,\ldots,X_n)-\sum_{i=1}^n H(X_i\mid X_{-i}), \qquad X_{-i}=(X_1,\ldots,X_{i-1},X_{i+1},\ldots,X_n).$$7 and $$\mathrm{DTC}(X_1,\ldots,X_n)=H(X_1,\ldots,X_n)-\sum_{i=1}^n H(X_i\mid X_{-i}), \qquad X_{-i}=(X_1,\ldots,X_{i-1},X_{i+1},\ldots,X_n).$$8, and the structured O-information is

$$\mathrm{DTC}(X_1,\ldots,X_n)=H(X_1,\ldots,X_n)-\sum_{i=1}^n H(X_i\mid X_{-i}), \qquad X_{-i}=(X_1,\ldots,X_{i-1},X_{i+1},\ldots,X_n).$$9

This quantity isolates between-group redundancy or synergy, so $$\Omega(X_1,\ldots,X_n)=\mathrm{TC}(X_1,\ldots,X_n)-\mathrm{DTC}(X_1,\ldots,X_n).$$0 indicates redundancy-dominated interactions between groups and $$\Omega(X_1,\ldots,X_n)=\mathrm{TC}(X_1,\ldots,X_n)-\mathrm{DTC}(X_1,\ldots,X_n).$$1 indicates synergy-dominated interactions between groups. The grouped construction is explicitly motivated by situations in which global $$\Omega(X_1,\ldots,X_n)=\mathrm{TC}(X_1,\ldots,X_n)-\mathrm{DTC}(X_1,\ldots,X_n).$$2 is dominated by within-group redundancy and therefore misses between-group synergy (Pascual-Marqui et al., 11 Jul 2025).

Dynamic extensions replace entropies by entropy rates. For a stationary multivariate process $$\Omega(X_1,\ldots,X_n)=\mathrm{TC}(X_1,\ldots,X_n)-\mathrm{DTC}(X_1,\ldots,X_n).$$3, the O-information rate is

$$\Omega(X_1,\ldots,X_n)=\mathrm{TC}(X_1,\ldots,X_n)-\mathrm{DTC}(X_1,\ldots,X_n).$$4

For $$\Omega(X_1,\ldots,X_n)=\mathrm{TC}(X_1,\ldots,X_n)-\mathrm{DTC}(X_1,\ldots,X_n).$$5, this reduces to the interaction information rate. The O-information rate can be decomposed into Granger-causal and instantaneous components, and frequency-domain expansions yield spectral O-information rate functions that reveal band-specific redundancy or synergy invisible in whole-band summaries (Faes et al., 2022).

Dynamic network formulations further define a node-wise OIR-gradient and a link-wise local OIR. In that framework, node colors encode redundancy- or synergy-dominated contributions of each node, edge colors encode whether pairwise links participate in redundant or synergistic higher-order interactions with the rest of the network, and a global annotation encodes the network-wise OIR (Mijatovic et al., 2024).

Quantity	Definition/setting	Interpretation
Global $$\Omega(X_1,\ldots,X_n)=\mathrm{TC}(X_1,\ldots,X_n)-\mathrm{DTC}(X_1,\ldots,X_n).$$6	$$\Omega(X_1,\ldots,X_n)=\mathrm{TC}(X_1,\ldots,X_n)-\mathrm{DTC}(X_1,\ldots,X_n).$$7	Net redundancy vs synergy
Local $$\Omega(X_1,\ldots,X_n)=\mathrm{TC}(X_1,\ldots,X_n)-\mathrm{DTC}(X_1,\ldots,X_n).$$8	Pattern-level pointwise form	Redundancy/synergy of one pattern
First-order gradient $$\Omega(X_1,\ldots,X_n)=\mathrm{TC}(X_1,\ldots,X_n)-\mathrm{DTC}(X_1,\ldots,X_n).$$9	$$\Omega(\mathbf{X}^n)=(n-2)H(\mathbf{X}^n)+\sum_{i=1}^n\big[H(X_i)-H(\mathbf{X}^n_{-i})\big].$$0	Variable contribution
O-information rate	Entropy-rate analog	Dynamic higher-order balance
Structured $$\Omega(\mathbf{X}^n)=(n-2)H(\mathbf{X}^n)+\sum_{i=1}^n\big[H(X_i)-H(\mathbf{X}^n_{-i})\big].$$1	Group-adjusted $$\Omega(\mathbf{X}^n)=(n-2)H(\mathbf{X}^n)+\sum_{i=1}^n\big[H(X_i)-H(\mathbf{X}^n_{-i})\big].$$2	Between-group balance

4. Estimation regimes and computational practice

For discrete data, the standard route is plug-in estimation from empirical joint and marginal frequencies. One computes the required entropies or conditional entropies directly, then assembles total correlation, dual total correlation, and O-information. The same count tables support local O-information, because pointwise surprisals $$\Omega(\mathbf{X}^n)=(n-2)H(\mathbf{X}^n)+\sum_{i=1}^n\big[H(X_i)-H(\mathbf{X}^n_{-i})\big].$$3, $$\Omega(\mathbf{X}^n)=(n-2)H(\mathbf{X}^n)+\sum_{i=1}^n\big[H(X_i)-H(\mathbf{X}^n_{-i})\big].$$4, and $$\Omega(\mathbf{X}^n)=(n-2)H(\mathbf{X}^n)+\sum_{i=1}^n\big[H(X_i)-H(\mathbf{X}^n_{-i})\big].$$5 can be read from empirical frequencies over observed states (Scagliarini et al., 2021).

For continuous data, several estimators are used in the literature. The Gaussian copula approach estimates mutual information and derived quantities after marginal Gaussianization via empirical CDFs, and is used both for continuous real-valued analyses and for neural activation studies that combine continuous activations with a discrete label (Scagliarini et al., 2022, Clauw et al., 2022). In the grokking study on tensor-network classifiers, O-information among three continuous class-output processes is estimated without discretization via the Kraskov–Stögbauer–Grassberger $$\Omega(\mathbf{X}^n)=(n-2)H(\mathbf{X}^n)+\sum_{i=1}^n\big[H(X_i)-H(\mathbf{X}^n_{-i})\big].$$6-nearest-neighbor estimator with $$\Omega(\mathbf{X}^n)=(n-2)H(\mathbf{X}^n)+\sum_{i=1}^n\big[H(X_i)-H(\mathbf{X}^n_{-i})\big].$$7 and the maximum norm, and the reported curves are averaged over all six permutations of variable order with one standard deviation across permutations (Pomarico et al., 31 Jul 2025).

Closed-form estimators are available under Gaussianity. For real Gaussian data with covariance $$\Omega(\mathbf{X}^n)=(n-2)H(\mathbf{X}^n)+\sum_{i=1}^n\big[H(X_i)-H(\mathbf{X}^n_{-i})\big].$$8, correlation matrix $$\Omega(\mathbf{X}^n)=(n-2)H(\mathbf{X}^n)+\sum_{i=1}^n\big[H(X_i)-H(\mathbf{X}^n_{-i})\big].$$9, and standardized inverse covariance $\Omega$0,

$\Omega$1

while for complex Gaussian data

$\Omega$2

The same log-determinant framework extends to total correlation, dual total correlation, structured O-information, and frequency-specific analyses based on cross-spectral covariance matrices (Pascual-Marqui et al., 11 Jul 2025).

A more recent nonparametric route is score-based estimation. "S$\Omega$3I: Score-based O-INFORMATION Estimation" (Bounoua et al., 2024) turns the KL-divergence expressions underlying total correlation and dual total correlation into time integrals of squared differences between score functions of noised densities. In that construction, a single amortized neural network learns joint, marginal, and conditional denoisers; inference requires $\Omega$4 forward passes through the score network per sampled diffusion time. With exact scores, the resulting total correlation, dual total correlation, and O-information estimates are exact; with learned scores, the method is approximate but was reported to outperform MI-decomposition baselines in high-dimensional and synergy-dominated regimes (Bounoua et al., 2024).

Dynamic linear-Gaussian implementations use VAR or state-space models. Entropy rates are then computed from restricted residual covariance matrices, mutual information rates from determinant ratios of residual covariances, and O-information rate, local OIR, and OIR-gradients are assembled from those primitives. Bootstrap on VAR innovations is used for OIR-derived measures, while iAAFT surrogates are used for pairwise MIR significance in dynamic network applications (Faes et al., 2022, Mijatovic et al., 2024).

Estimator sensitivity remains a standard caveat. Small sample sizes, near-deterministic late-training regimes, high dimensionality, and search over large subsets all increase variance or computational burden. Several papers therefore recommend bootstrap confidence intervals, surrogate testing, permutation averaging, or greedy subset search to mitigate overinterpretation of weak or unstable effects (Scagliarini et al., 2022, Pomarico et al., 31 Jul 2025, Clauw et al., 2022).

5. Empirical domains and recurring interpretations

The original large-scale empirical demonstration concerned Baroque music. In Bach’s four-voice chorales, the global O-information is negative, whereas in Corelli’s Opus 1 and 3–6 it is positive. The local analysis further showed that redundancy-dominated patterns concentrate at stable, consonant, and harmonically central events, while synergy-dominated patterns accompany dissonances, inversions, and harmonic excursions (Rosas et al., 2019, Scagliarini et al., 2021).

In statistical physics and macroeconomics, gradients of O-information were used to localize higher-order informational circuits. In a frustrated seven-spin Ising model, peripheral spins contributed positively while the central spin contributed negatively, localizing synergy to the frustration-inducing spin. In quarterly US macroeconomic indicators, first-order gradients identified several redundantly connected indicators, while GPDI emerged with a significantly negative first-order gradient and GDP was the most connected node for pairwise synergy under the second-order gradient analysis (Scagliarini et al., 2022).

In neuroscience and physiology, dynamic O-information measures reveal structure that is not captured by pairwise links. The O-information rate framework documented frequency-specific redundancy and synergy in simulated oscillatory networks, cardiovascular variability, and ECoG during anesthesia; notably, whole-band integration could mask opposite-sign higher-order interactions present in distinct frequency bands (Faes et al., 2022). The network-representation framework then mapped global OIR, node-wise OIR-gradients, and link-wise local OIR onto network diagrams, showing redundancy in source-hub star networks, synergy in sink-hub star networks, and condition-specific higher-order reconfiguration in cardiovascular physiology (Mijatovic et al., 2024).

In neural representation analysis, O-information has been used to study groups of neurons in trained classifiers. In a fully connected MNIST network, the first hidden layer was found to be redundancy dominated, consistent with shared general features, while the last hidden layer was synergy dominated, consistent with local class-specific features. Selecting the most synergistic multi-neuron subsets and retraining only those sub-networks yielded smaller performance drops than random or single-neuron MI-based baselines (Clauw et al., 2022).

Dynamic conditioned O-information has also been applied to neural spiking. In monkey spiking data during a perceptual discrimination task, both redundancy- and synergy-dominated incoming multiplets toward informative target neurons peaked around $\Omega$5 ms after the go cue, and synergistic multiplets increasingly included neurons previously categorized as containing little relevant information individually. This is a direct example of an empirically important point: neurons or variables that appear weak at the bivariate level can become critical once high-order synergy is considered (Stramaglia et al., 2020).

6. O-information in grokking and tensor-network multi-class classification

A recent application uses O-information to detect and interpret grokking in Matrix Product State classifiers trained on three-class problems. In that setting, the variables are the three class-output score processes $\Omega$6, $\Omega$7, and $\Omega$8 tracked across training sweeps, and the tri-variate O-information is estimated from the sweep-indexed samples as

$\Omega$9

which is valid because for three variables O-information coincides with interaction information. The outputs are continuous real-valued scores, no softmax is applied, and the estimator is the KSG $\Omega>0$0NN estimator with $\Omega>0$1 and Chebyshev distance; the reported value is averaged over all six permutations of variable order (Pomarico et al., 31 Jul 2025).

The key empirical result is a task-dependent sign transition. In the three-class fashion MNIST problem, $\Omega>0$2 starts negative under random initialization, then around the entanglement transition and onset of grokking, at sweeps approximately $\Omega>0$3–$\Omega>0$4, it undergoes a sharp jump into positive values, with a moderate peak preceding the transition by one sweep. This redundancy peak co-occurs with a collapse in entanglement entropy from an initial volume-law to a lower-entropy regime, directional transfer entropy dominated by the $\Omega>0$5 mask interaction at short delays $\Omega>0$6–$\Omega>0$7, and improved generalization for sneaker and dress while bag remains confounding (Pomarico et al., 31 Jul 2025).

The hyperspectral land-cover task behaves differently. There, $\Omega>0$8 begins negative and remains negative even after a sharp training transition around sweeps approximately $\Omega>0$9–$\Omega<0$0, entanglement entropy drops simultaneously across masks, local magnetizations polarize synchronously, and transfer-entropy curves remain lower and flatter across mask pairs. The paper interprets this regime as overfitted and disordered: persistent negative O-information aligns with synergy-dominated output interactions, weak directional dependencies, and limited generalization (Pomarico et al., 31 Jul 2025).

This use case clarifies both the strength and the limitation of O-information. Its strength is that a single signed multivariate observable can track a shift from synergy-dominated to redundancy-dominated organization in the model outputs and align that shift with entanglement restructuring and causal mask interactions. Its limitation is interpretive: positive $\Omega<0$1 can in principle arise from trivial output correlations or label imbalance, and negative $\Omega<0$2 can reflect either healthy complementary structure or disordered behavior. The tensor-network study addresses this by contextualizing O-information with transfer entropy, entanglement entropy, magnetization, and accuracy rather than treating the sign of $\Omega<0$3 as self-sufficient (Pomarico et al., 31 Jul 2025).