Transfer Entropy and Directed Information in Gaussian diffusion processes

Abstract: Transfer Entropy and Directed Information are information-theoretic measures of the directional dependency between stochastic processes. Following the definitions of Schreiber and Massey in discrete time, we define and evaluate these measures for the components of multidimensional Gaussian diffusion processes. When the components are jointly Markov, the Transfer Entropy and Directed Information are both measures of influence according to a simple physical principle. More generally, the effect of other components has to be accounted for, and this can be achieved in more than one way. We propose two definitions, one of which preserves the properties of influence of the jointly Markov case. The Transfer Entropy and Directed Information are expressed in terms of the solutions of matrix Riccati equations, and so are easy to compute. The definition of continuous-time Directed Information we propose differs from that previously appearing in the literature. We argue that the latter is not strictly directional.