Directed Rate-Distortion in Causal Source Coding

• Directed rate-distortion is defined as a generalization of classical rate-distortion theory, replacing mutual information with directed information under causal feed-forward constraints.
• It employs a multi-letter formulation that captures causal encoding restrictions and relates to channel capacity with feedback, highlighting its role in sources with memory.
• Practical applications include compression for Gaussian and binary Markov sources, multi-terminal extensions, quantum generalizations, and learning-based reinterpretations.

Directed rate-distortion is the rate-distortion function for source coding with feed-forward, and, in this setting, the standard mutual information in the rate-distortion function is replaced by directed information from the source sequence to the reconstruction sequence. In the formulation summarized for sources with feed-forward, the resulting quantity is a multi-letter expression, and the same directed-information formalism appears in the dual problem of channel capacity with feedback. The topic therefore occupies a central position among causally constrained information-theoretic problems: it generalizes ordinary rate-distortion theory when temporal directionality matters, yet it reduces to the standard rate-distortion function in important special cases such as memoryless sources with single-letter distortion measures [0702009].

1. Foundational causal formulation

For a source sequence $X^n=(X_1,\ldots,X_n)$ and a reconstruction $\hat{X}^n=(\hat{X}_1,\ldots,\hat{X}_n)$, directed information is given by

$$I(X^n \to \hat{X}^n)=\sum_{i=1}^n I(X^i;\hat{X}_i\mid \hat{X}^{i-1}).$$

This quantity captures causality in scenarios where the relevant conditional law is causal rather than unrestricted. In the standard description of the feed-forward setting, the rate-distortion function with feed-forward, sometimes called the directed rate-distortion function, is

$$R_{ff}(D)=\lim_{n\to\infty}\frac{1}{n}\min_{p(\hat{x}^n\|x^n)} I(X^n\to\hat{X}^n)$$

subject to the expected distortion constraint

$\mathbb{E}[d(X^n,\hat{X}^n)]\le D,$

where $p(\hat{x}^n\|x^n)$ denotes a causal conditional distribution of the form

$$p(\hat{x}^n|x^n)=\prod_{i=1}^n p(\hat{x}_i\mid \hat{x}^{i-1},x^i).$$

These formulas place causality inside the optimization itself, rather than treating it as an external implementation restriction [0702009].

The conceptual shift relative to classical rate-distortion theory is precise. Ordinary rate-distortion theory minimizes mutual information under a distortion constraint; directed rate-distortion minimizes a causal information measure under the analogous constraint. This suggests that directed rate-distortion is not merely a reformulation of the classical problem, but a characterization of lossy compression when admissible reconstructions are sequentially constrained.

2. Multi-letter structure, single-letterization, and computation

The rate-distortion function with feed-forward is described as a multi-letter expression and cannot be computed easily in general. The 2007 formulation studies conditions under which these expressions can be computed for a large class of sources or channels with memory and distortion or cost measures, emphasizing that computability is itself a primary theoretical issue rather than a secondary numerical detail [0702009].

In general, the directed rate-distortion function does not admit a simple single-letter formula for arbitrary sources and distortion measures. However, for special cases such as memoryless sources and single-letter distortion measures, or certain Markov sources under additive distortion, the function can sometimes be single-letterized. When the source is memoryless and causal and non-causal encoding coincide, the expression reduces to

$$R_{ff}(D)=\min_{p(\hat{x}|x):\mathbb{E}[d(X,\hat{X})]\le D} I(X;\hat{X}),$$

which is the standard rate-distortion function [0702009].

This reduction addresses a common misconception. Directed formulations do not automatically imply a different operational rate for every source class; for i.i.d. models with the appropriate distortion structure, the classical Shannon formula can re-emerge. The distinction becomes substantive for sources with memory, causal reconstruction constraints, or distortion criteria that interact with temporal structure.

Illustrative instances mentioned in the general theory include Gaussian sources with quadratic distortion, for which the optimal test channel is often Gaussian and causal, and Binary Markov Sources, for which exact directed rate-distortion functions can sometimes be derived by exploiting the Markov structure and using dynamic programming methods. Blahut-Arimoto type algorithms for directed information can also be used for numerical computation in certain cases [0702009].

3. Directed information, feedback duality, and causal information flow

Directed rate-distortion is explicitly paired with the capacity of channels with feedback. The latter can be written as

$$C_{fb}=\lim_{n\to\infty}\frac{1}{n}\max_{p(x^n\|y^{n-1})} I(X^n\to Y^n),$$

while the feed-forward rate-distortion function takes the minimization form over causal conditionals. This symmetry highlights the generalized role of directed information as an extension of mutual information to causally constrained problems [0702009].

A related causal quantity used in later work is transfer entropy, introduced in the neural speech-tracking setting as

$$\mathbb{TE}(X^n\to Z^n)\triangleq I(X^{n-1};Z_n\mid Z^{n-1}).$$

In that context, transfer entropy is described as a directed, causal information-theoretic quantity measuring how much the past of $\hat{X}^n=(\hat{X}_1,\ldots,\hat{X}_n)$0 helps predict the present of $\hat{X}^n=(\hat{X}_1,\ldots,\hat{X}_n)$1, over and above $\hat{X}^n=(\hat{X}_1,\ldots,\hat{X}_n)$2’s own past. Classical mutual information is said to quantify pairwise redundancy but to fail to capture redundancy across sets of signals, especially with temporal-causal relationships (Østergaard et al., 28 Jan 2025).

The connection is structural rather than identity-based. Directed rate-distortion is formulated through directed information under a distortion constraint; transfer-entropy-based constructions study causal information flow and redundancy in time series. This suggests a broader causal information-theoretic ecosystem in which directed rate-distortion is one of the canonical optimization problems and transfer entropy is one of the associated diagnostic measures.

4. General-source and multi-terminal extensions

The successive refinement problem provides a multi-terminal setting in which directionality appears through layered reconstructions. A source sequence is encoded into two codewords by two encoders, one reconstruction is obtained from one codeword, and the other reconstruction is obtained from both codewords. In the formulation for general sources under the maximum distortion criterion, the rate-distortion region is expressed through an information-spectrum formula involving an auxiliary test channel $\hat{X}^n=(\hat{X}_1,\ldots,\hat{X}_n)$3, distortion constraints on the reconstructions, and spectral sup-mutual information rates such as $\hat{X}^n=(\hat{X}_1,\ldots,\hat{X}_n)$4 and $\hat{X}^n=(\hat{X}_1,\ldots,\hat{X}_n)$5 (Matsuta et al., 2018).

The same work gives non-asymptotic inner and outer bounds on achievable pairs of codeword numbers under prescribed excess-distortion probability levels. The finite-blocklength bounds are stated in terms of smooth max Rényi divergence mutual information, denoted $\hat{X}^n=(\hat{X}_1,\ldots,\hat{X}_n)$6, and the asymptotic general formula is recovered via spectral mutual information rates. For memoryless sources, the region recovers Rimoldi’s classic characterization; for mixed sources, the region is the intersection of individual regions (Matsuta et al., 2018).

These results broaden the scope of directed rate-distortion in two ways. First, they show that directional structure is not confined to single-stream causal coding, but also appears in layered and multi-terminal lossy compression. Second, they integrate non-asymptotic analysis with information-spectrum methods, indicating that directed and multi-terminal rate-distortion problems can be studied beyond stationary or memoryless assumptions.

5. Learning-theoretic and empirical reinterpretations

A learning-based reinterpretation appears in Rate-Distortion Auto-Encoders, which minimize the mutual information between the inputs and the outputs of the auto-encoder subject to a fidelity constraint. The central rate-distortion objective is stated as

$\hat{X}^n=(\hat{X}_1,\ldots,\hat{X}_n)$7

The approach uses a matrix-based analogue of Rényi’s $\hat{X}^n=(\hat{X}_1,\ldots,\hat{X}_n)$8-entropy computed from Gram matrices of kernel evaluations, with the stated purpose of avoiding the plug in estimation of densities. The entropy term acts as a regularization term, while the fidelity constraint can be understood as a risk functional in the conventional statistical learning setting (Giraldo et al., 2013).

The same source states that the general approach fits into the Directed Rate-Distortion framework: one seeks mappings, possibly deterministic or stochastic, in a directed fashion from input to code to output, with mutual information measured between input and code or output, constrained by distortion. It also relates this perspective to Directed Information Bottleneck methods, emphasizing that compression and regularization are unified within rate-distortion theory rather than relying on heuristics such as explicit weight decay (Giraldo et al., 2013).

A different empirical reinterpretation appears in neural tracking of speech, where directed redundancy is used for causal, multivariate, time-series systems. In a competing talker scenario, the rate-redundancy in the left temporal EEG region is operationalized through transfer-entropy terms from stimulus to reconstruction, from each electrode to reconstruction, and from stimulus to each electrode. The distortion measure is

$\hat{X}^n=(\hat{X}_1,\ldots,\hat{X}_n)$9

where $$I(X^n \to \hat{X}^n)=\sum_{i=1}^n I(X^i;\hat{X}_i\mid \hat{X}^{i-1}).$$0 is the Pearson correlation between true and reconstructed speech envelopes. For attended speech, both the transfer entropy rate and the directed redundancy in the left temporal EEG signals are linearly, inversely related to distortion, and the relationship is statistically significant for both $$I(X^n \to \hat{X}^n)=\sum_{i=1}^n I(X^i;\hat{X}_i\mid \hat{X}^{i-1}).$$1 and $$I(X^n \to \hat{X}^n)=\sum_{i=1}^n I(X^i;\hat{X}_i\mid \hat{X}^{i-1}).$$2. For distracting speech, no such clear relationship is observed (Østergaard et al., 28 Jan 2025).

The significance of these learning-oriented and neuroscience-oriented formulations is methodological. They do not replace the classical causal source-coding problem, but they show that directed rate-distortion ideas can be reinterpreted as objectives for representation learning and as operational descriptors of information fidelity in neural systems.

6. Quantum generalization and adjacent operational developments

A quantum generalization is given by the rate-distortion version of quantum state redistribution, described as a quantum generalisation of the classical Wyner-Ziv problem. The source is a tripartite pure state shared between Alice $$I(X^n \to \hat{X}^n)=\sum_{i=1}^n I(X^i;\hat{X}_i\mid \hat{X}^{i-1}).$$3, Bob $$I(X^n \to \hat{X}^n)=\sum_{i=1}^n I(X^i;\hat{X}_i\mid \hat{X}^{i-1}).$$4, and a reference $$I(X^n \to \hat{X}^n)=\sum_{i=1}^n I(X^i;\hat{X}_i\mid \hat{X}^{i-1}).$$5, and the distortion is encoded in an observable on $$I(X^n \to \hat{X}^n)=\sum_{i=1}^n I(X^i;\hat{X}_i\mid \hat{X}^{i-1}).$$6. For entanglement-assisted schemes, the rate-distortion function is

$$I(X^n \to \hat{X}^n)=\sum_{i=1}^n I(X^i;\hat{X}_i\mid \hat{X}^{i-1}).$$7

and the single-letter auxiliary quantity is

$$I(X^n \to \hat{X}^n)=\sum_{i=1}^n I(X^i;\hat{X}_i\mid \hat{X}^{i-1}).$$8

Because of a possible discontinuity at the minimum achievable distortion, the relevant function is the right-continuous extension

$$I(X^n \to \hat{X}^n)=\sum_{i=1}^n I(X^i;\hat{X}_i\mid \hat{X}^{i-1}).$$9

with the main theorem stating $$R_{ff}(D)=\lim_{n\to\infty}\frac{1}{n}\min_{p(\hat{x}^n\|x^n)} I(X^n\to\hat{X}^n)$$0 (Khanian et al., 2021).

This framework includes as special cases most quantum rate-distortion problems considered in the past, as well as Schumacher compression and state redistribution with per-copy fidelity. At the same time, the formula presents two technical difficulties: optimization over an unbounded auxiliary register and a continuity issue at zero distortion. The work states that these difficulties can be overcome in certain situations, including important special cases in which the auxiliary system can be bounded (Khanian et al., 2021).

Recent compression papers also illuminate adjacent operational themes. A learned linear block transform called the Rate Distortion Learned Transform is directly optimized from an RD perspective, but the paper explicitly states that it does not directly implement or extend Directed Rate-Distortion; rather, its design resonates with DRD’s core principle of optimizing operational rate-distortion, not just MSE (Gnutti et al., 2024). Likewise, neural $$R_{ff}(D)=\lim_{n\to\infty}\frac{1}{n}\min_{p(\hat{x}^n\|x^n)} I(X^n\to\hat{X}^n)$$1-domain rate control for deep video compression learns frame-wise rate-distortion-$$R_{ff}(D)=\lim_{n\to\infty}\frac{1}{n}\min_{p(\hat{x}^n\|x^n)} I(X^n\to\hat{X}^n)$$2 relationships to achieve target bitrate and mitigate inter-frame quality fluctuations, which is a rate-distortion control problem rather than a formal DRD characterization (Gu et al., 2024).

Taken together, these developments indicate that directed rate-distortion has two enduring roles. One is foundational: a causal generalization of lossy source coding governed by directed information, information spectrum quantities, or conditional mutual information in quantum settings. The other is methodological: a template for designing objectives and diagnostic measures in sequential learning, neural information processing, and practical compression systems.