Causally Conditioned Directed Info Rate

• Causally conditioned directed information rate is a metric that measures the average per-symbol flow of information under causal constraints using side information.
• It naturally decomposes into lagged transfer entropy and instantaneous exchange, thereby clarifying directional dependencies in stochastic processes.
• Applications include estimating feedback channel capacity, effective connectivity in neuroscience and econometrics, and assessing privacy leakage in control systems.

The causally conditioned directed information rate quantifies the average per-symbol amount of information that a sequence $X^n$ causally provides about another sequence $Y^n$, given a third side-information sequence $Z^n$. This measure plays a central role in quantifying directional, often time-asymmetric, dependencies in stochastic processes and serves as a unifying framework for transfer entropy, feedback information, graphical causal inference, privacy assessment in control, and channel capacity in feedback communication. The rate is defined for stationary ergodic processes as the normalized limit of causally conditioned directed information over increasing block-lengths, providing rigorous operational and statistical interpretations for information flow under causal constraints.

1. Formal Definitions and Basic Properties

For sequences $X^n = (X_1,\ldots,X_n)$, $Y^n = (Y_1,\ldots,Y_n)$, and a third “side information” sequence $Z^n = (Z_1,\ldots,Z_n)$, the causally conditioned directed information from $X^n$ to $Y^n$ given $Z^n$ is

$$I\bigl(X^n \to Y^n \parallel Z^n\bigr) = \sum_{i=1}^n I\bigl(X^i; Y_i \mid Y^{i-1}, Z^i\bigr).$$

This is equivalently expressible via causally conditioned entropies: $$I(X^n \to Y^n\!\parallel Z^n) = H(Y^n \parallel Z^n) - H(Y^n \parallel X^n, Z^n),$$ with the notation $$H(Y^n \parallel X^n, Z^n) = -\mathbb{E}\big[\log p_{Y^n|X^n,Z^n}(Y^n|X^n,Z^n)\big]$$.

If the joint process $(X_t, Y_t, Z_t)$ is stationary and ergodic, the causally conditioned directed information rate is defined as

$$\bar I(X \to Y \parallel Z) = \lim_{n \to \infty} \frac{1}{n} I(X^n \to Y^n\!\parallel Z^n) = \lim_{n\to\infty} I(X^n; Y_n \mid Y^{n-1}, Z^n),$$

where the existence of the limit is guaranteed by subadditivity and ergodicity (Amblard et al., 2010, Tanaka et al., 2017, Derpich et al., 2021, Raginsky, 2011). This rate is always non-negative and invariant under nonsingular time transformations for processes in continuous time (Weissman et al., 2011).

2. Decomposition, Operational Interpretations, and Connections

The causally conditioned directed information rate decomposes naturally into two components:

• Conditional transfer entropy rate: quantifies information transfer from the past of $X$ to the present of $Y$, given $Z$;
• Instantaneous exchange rate: quantifies instantaneous or contemporaneous coupling between $X$ and $Y$, given $Z$.

Explicitly,

$$\bar I(X \to Y \parallel Z) = \bar T_{X \to Y \parallel Z} + \bar I_{\text{inst}}(X \to Y \parallel Z),$$

where

$$\bar T_{X \to Y \parallel Z} = \lim_{n\to\infty} I(X^{n-1}; Y_n \mid Y^{n-1}, Z^n), \qquad \bar I_{\text{inst}} = \lim_{n\to\infty} I(X_n; Y_n \mid X^{n-1}, Y^{n-1}, Z^n).$$

This decomposition is crucial for mapping the zeros of $\bar T_{X \to Y \parallel Z}$ and $\bar I_{\text{inst}}$ to the absence of, respectively, lagged and contemporaneous edges in Granger causality graphs (Amblard et al., 2010).

Operationally, $\bar I(X \to Y \parallel Z)$ coincides with feedback channel capacity when causal side information $Z$ is available at both encoder and decoder (Raginsky, 2011). In causal inference, if $Z$ blocks all backdoor paths from $X$ to $Y$, then $\bar I(X \to Y \parallel Z) = 0$, precisely matching the information-theoretic analogue of Pearl's "back-door" criterion (Raginsky, 2011).

3. Estimation and Universal Consistency

When the underlying processes are finite-alphabet and stationary ergodic, four universal estimators for the directed information rate have been introduced, each based on different functionals of universal or context-tree weighting (CTW) probability assignments (Jiao et al., 2012):

• Shannon–McMillan–Breiman type estimator;
• Entropy-functional estimator (smoothed);
• Forward–backward relative-entropy estimator (nonnegative);
• Joint-to-product relative-entropy estimator (bounded and nonnegative).

These estimators achieve $O(n^{-1/2} \log n)$ rates of convergence under suitable conditions, and the minimax lower bound is also $n^{-1/2}$. The CTW algorithm allows for efficient, online, and strongly consistent estimation of directed information rates (Jiao et al., 2012).

For high-dimensional or real-valued (e.g., Gaussian) data, the rate formula reduces to the difference in log-determinant prediction error covariances from vector autoregressions with and without the input process $X$, yielding an estimator that achieves $O\left(N^{-1/2}\log N\right)$ non-asymptotic error with high probability (Zheng et al., 6 Dec 2025).

4. Implications for Causality, Network Structure, and Hypothesis Testing

Causally conditioned directed information rate provides a nonparametric, model-free measure for inferring effective connectivity, especially in neuroscience and econometrics (Amblard et al., 2010). If $\bar I(X \to Y \parallel Z) = 0$, there is no directed edge from $X$ to $Y$ in the Granger-causality graph conditioned on $Z$. The dynamic/transfer entropy component matches lagged causality (Granger), while the instantaneous component captures contemporaneous relationships.

Testing for nonzero rates involves statistical hypothesis tests, including surrogate-data approaches, block-bootstrap, or permutation tests, since no analytical null distributions generally exist for these estimators (Amblard et al., 2010). Estimation must carefully address boundary effects and window selection in practice.

5. Axiomatic and Privacy-Theoretic Foundations

An axiomatic derivation uniquely characterizes causally conditioned directed information as the operational measure of privacy leakage in feedback and control systems. This follows from postulates about Bayes risk reduction, causal data-processing, separation of private/public data over time, and additivity (Tanaka et al., 2017). In Linear-Quadratic-Gaussian (LQG) settings, the optimal privacy-control tradeoff can be solved as a semidefinite program that minimizes the causally conditioned directed information rate subject to performance constraints (Tanaka et al., 2017).

Lower bounds on estimation or decoding error under prescribed information rate constraints are provided via rate-distortion and data-processing arguments, ensuring operational relevance in control and privacy (Tanaka et al., 2017).

6. Computational Methods and Channel Capacity

For feedback channels with memory and causal side-information, capacity maximization over causal input distributions can be performed via an extension of the Blahut–Arimoto algorithm. This algorithm uses alternating maximization over causal input laws and backward-indexed auxiliary PMFs, and provides finite-block, tight upper/lower bounds converging to the causal directed information rate (Naiss et al., 2010).

In continuous-time processes, the causally conditioned directed information rate exists for broad classes (including stationary ergodic Gaussian or Poisson channels with feedback), and fundamental limits on reliable feedback communication equal this rate (Weissman et al., 2011).

7. Applications and Experimental Characterization

Applications include:

• Inference of effective connectivity in neural spike and LFP recordings (Amblard et al., 2010);
• Detection of causal directionality in econometric time series, e.g., stock indices (Jiao et al., 2012);
• Quantification of privacy leakage in cloud-based and cyber-physical control (Tanaka et al., 2017);
• Calculation of feedback channel capacity with memory and delayed/causal side information (Naiss et al., 2010, Weissman et al., 2011);
• Estimation of Granger-type causal links via statistical thresholding of estimated conditional directed information rates (Amblard et al., 2010, Raginsky, 2011).

Empirical experiments confirm that universal and VAR-based estimators reliably recover true rates and successfully localize directionality and delays in both synthetic and real high-dimensional datasets (Zheng et al., 6 Dec 2025, Jiao et al., 2012).

---

References:

• (Amblard et al., 2010) On directed information theory and Granger causality graphs
• (Tanaka et al., 2017) Directed Information as Privacy Measure in Cloud-based Control
• (Derpich et al., 2021) Directed Data-Processing Inequalities for Systems with Feedback
• (Naiss et al., 2010) Extension of the Blahut-Arimoto algorithm for maximizing directed information
• (0802.1383) On Directed Information and Gambling
• (Weissman et al., 2011) Directed Information, Causal Estimation, and Communication in Continuous Time
• (Raginsky, 2011) Directed information and Pearl's causal calculus
• (Jiao et al., 2012) Universal Estimation of Directed Information
• (Zheng et al., 6 Dec 2025) Non-Asymptotic Error Bounds for Causally Conditioned Directed Information Rates of Gaussian Sequences