Innovation-Block Differential Entropy

• Innovation-block differential entropy is an extension of classical entropy that quantifies uncertainty in blocks of innovation processes using Doob decompositions and whitened projections.
• It underpins practical applications in nonlinear filtering, reservoir computing, and compressibility analysis by linking the block structure and innovation capacity to sample complexity.
• It also guides adaptive rate control and decision-based metrics by leveraging the geometric and statistical properties of innovations in dynamical learning systems.

Innovation-block differential entropy quantifies the information content or uncertainty associated with blocks (finite or infinite sequences) of innovation processes arising in filtered probability spaces, dynamical learning systems, signal processing, and statistical mechanics. It generalizes classical differential entropy to settings where innovations are defined as Doob components orthogonal to past filtrations, with applications ranging from nonlinear filtering and reservoir computing to the compressibility analysis of stochastic processes. The metric reflects not only the inherent randomness of the innovation process but also its block structure, rate dimension, and capacity constraints.

1. Formal Definitions and Doob Innovations

Innovation-block differential entropy is constructed by considering a process $X_t \in \mathbb{R}^d$, its input-generated filtration $\mathscr{F}^{\rm in}_t=\sigma(u_s:s \le t)$, and the one-step Doob decomposition: $$\langle X_t \rangle = E[X_t \mid \mathscr{F}^{\rm in}_t], \qquad \Delta X_t = X_t - \langle X_t \rangle,$$ where $\Delta X_t$ is the innovation, defined as the unpredictable component orthogonal to the history. When the covariance $\Sigma_{XX} = E[X_t X_t^\top]$ is invertible, innovations are "whitened": $\Delta Z_t = \Sigma_{XX}^{+/2} \Delta X_t,$ and further projected onto a trimmed innovation subspace $\mathcal{U}_\tau$ by $P_\tau$ (orthonormal projector), yielding $Y_t = P_\tau \Delta Z_t$. The block of innovations,

$$Y_t^{(b)} = [Y_{t-b+1}^\top, ..., Y_t^\top]^\top \in \mathbb{R}^{L_\tau b},$$

serves as the fundamental object whose differential entropy is given by

$\mathscr{F}^{\rm in}_t=\sigma(u_s:s \le t)$0

with $\mathscr{F}^{\rm in}_t=\sigma(u_s:s \le t)$1 (Polloreno, 12 Jan 2026).

In path-space nonlinear filtering, the innovation process $\mathscr{F}^{\rm in}_t=\sigma(u_s:s \le t)$2 associated with an observed signal $\mathscr{F}^{\rm in}_t=\sigma(u_s:s \le t)$3 (driven by Brownian motion $\mathscr{F}^{\rm in}_t=\sigma(u_s:s \le t)$4 and drift $\mathscr{F}^{\rm in}_t=\sigma(u_s:s \le t)$5) is given by (Ustunel, 2013)

$\mathscr{F}^{\rm in}_t=\sigma(u_s:s \le t)$6

For block entropy on $\mathscr{F}^{\rm in}_t=\sigma(u_s:s \le t)$7, the relative entropy of the innovation law w.r.t. Wiener measure $\mathscr{F}^{\rm in}_t=\sigma(u_s:s \le t)$8 is

$\mathscr{F}^{\rm in}_t=\sigma(u_s:s \le t)$9

which equals the block "kinetic energy" of the best predictable drift over the interval.

2. Block Entropy Rate, Quantization, and Entropy Dimension

For stationary innovation processes in continuous time, the block differential-entropy rate is formalized by quantizing time ($$\langle X_t \rangle = E[X_t \mid \mathscr{F}^{\rm in}_t], \qquad \Delta X_t = X_t - \langle X_t \rangle,$$0), amplitude ($$\langle X_t \rangle = E[X_t \mid \mathscr{F}^{\rm in}_t], \qquad \Delta X_t = X_t - \langle X_t \rangle,$$1), and block length ($$\langle X_t \rangle = E[X_t \mid \mathscr{F}^{\rm in}_t], \qquad \Delta X_t = X_t - \langle X_t \rangle,$$2) (Ghourchian et al., 2017): $$\langle X_t \rangle = E[X_t \mid \mathscr{F}^{\rm in}_t], \qquad \Delta X_t = X_t - \langle X_t \rangle,$$3 and block entropy

$$\langle X_t \rangle = E[X_t \mid \mathscr{F}^{\rm in}_t], \qquad \Delta X_t = X_t - \langle X_t \rangle,$$4

The block differential-entropy rate is obtained as

$$\langle X_t \rangle = E[X_t \mid \mathscr{F}^{\rm in}_t], \qquad \Delta X_t = X_t - \langle X_t \rangle,$$5

In regimes where random variables are discrete-continuous, the block entropy contains a "rate dimension" $$\langle X_t \rangle = E[X_t \mid \mathscr{F}^{\rm in}_t], \qquad \Delta X_t = X_t - \langle X_t \rangle,$$6 analogous to Rényi's entropy dimension.

Closed-form asymptotics for $$\langle X_t \rangle = E[X_t \mid \mathscr{F}^{\rm in}_t], \qquad \Delta X_t = X_t - \langle X_t \rangle,$$7-stable innovation processes with stability $$\langle X_t \rangle = E[X_t \mid \mathscr{F}^{\rm in}_t], \qquad \Delta X_t = X_t - \langle X_t \rangle,$$8 yield

$$\langle X_t \rangle = E[X_t \mid \mathscr{F}^{\rm in}_t], \qquad \Delta X_t = X_t - \langle X_t \rangle,$$9

while for impulsive Poisson innovations with rate $\Delta X_t$0 and jump law $\Delta X_t$1,

$\Delta X_t$2

with lower entropy rate signifying higher compressibility (Ghourchian et al., 2017).

3. Capacity, Entropy Growth, and Geometric Structure

Innovation capacity $\Delta X_t$3 is defined as the trace of the expected conditional covariance projected onto the active subspace: $\Delta X_t$4 partitioning the observable rank into predictable and innovation components. In linear-Gaussian (Johnson–Nyquist) regimes with $\Delta X_t$5,

$\Delta X_t$6

where $\Delta X_t$7 are nonzero eigenvalues of $\Delta X_t$8.

The entropy bound is extensive: $\Delta X_t$9 so block entropy grows linearly in effective innovation dimension and block length (Polloreno, 12 Jan 2026).

Geometrically, in whitened coordinates, complementary ellipsoids represent predictable and innovation directions: $\Sigma_{XX} = E[X_t X_t^\top]$0 with innovation axes $\Sigma_{XX} = E[X_t X_t^\top]$1.

4. Filtering, Learning, and Application Domains

Innovation-block differential entropy provides operational control over nonlinear filtering and signal estimation tasks, particularly by quantifying the information content contributed by unpredictable innovations relative to a reference process (such as Wiener measure) (Ustunel, 2013). For practical filtering:

• The block entropy per step approximates $\Sigma_{XX} = E[X_t X_t^\top]$2;
• Low-entropy blocks signify low innovation-energy and correspond to high estimation quality.

Extensive innovation-block entropy also underpins sample complexity in generative modeling: learning the induced block law to $\Sigma_{XX} = E[X_t X_t^\top]$3 total variation error requires $\Sigma_{XX} = E[X_t X_t^\top]$4 samples, supporting generative reservoir learning (Polloreno, 12 Jan 2026).

In compressibility contexts, block differential entropy ranks innovation processes, with impulsive Poisson innovations exhibiting finite entropy rates, and heavy-tailed $\Sigma_{XX} = E[X_t X_t^\top]$5-stable processes showing divergent rates with decreasing stability $\Sigma_{XX} = E[X_t X_t^\top]$6 (i.e., being more compressible as $\Sigma_{XX} = E[X_t X_t^\top]$7 decreases) (Ghourchian et al., 2017).

5. Localization, Truncation, and Rate Control

For signals where Novikov's criterion for change-of-measure is violated, block entropy can be localized using stopping times (Ustunel, 2013): $\Sigma_{XX} = E[X_t X_t^\top]$8 yielding localized entropy identities

$\Sigma_{XX} = E[X_t X_t^\top]$9

Taking $\Delta Z_t = \Sigma_{XX}^{+/2} \Delta X_t,$0 recovers the full-interval result via monotone convergence.

In trimmed innovation subspaces, the variance floor $\Delta Z_t = \Sigma_{XX}^{+/2} \Delta X_t,$1 bounds $\Delta Z_t = \Sigma_{XX}^{+/2} \Delta X_t,$2: $\Delta Z_t = \Sigma_{XX}^{+/2} \Delta X_t,$3 These bounds allow fine control over block entropy growth and distinguishable history packing.

6. Comparative Perspectives: Shannon, Rényi, and Knowledge Measures

Classical Shannon entropy is nonselective—it is sensitive to all probability-mass rearrangements, regardless of relevance to a reference challenge (Samid, 2010). Samid's MARK (Missing Acquirable Relevant Knowledge) localizes entropy measurement by incorporating "intervals of interest" (IOI, IOF), quantifying only knowledge relevant to narrowing solution uncertainty. The continuous analogue implements block entropy via the averaged maximal window-coverage function: $\Delta Z_t = \Sigma_{XX}^{+/2} \Delta X_t,$4 with $\Delta Z_t = \Sigma_{XX}^{+/2} \Delta X_t,$5 the maximal interval probability. MARK curves facilitate tracking knowledge acquisition in R&D, risk management, and opportunity exploitation, complementing block differential entropy by focusing on decision-relevant uncertainty.

7. Summary and Current Trends

Innovation-block differential entropy is now recognized as a central tool for quantifying information growth in blocks of innovation processes, closely tied to the innovation capacity, geometric structure of the underlying reservoir or signal space, and the compressibility properties of stochastic models. The extensive scaling of entropy in block length and innovation dimension underpins the sample complexity of learning, distinguishable history enumeration, and the identification of compressible processes. The linkage with operational filtering, adaptive rate control, and decision-based entropy metrics (e.g., MARK) emphasizes its foundational role across information theory, statistical mechanics, and learning systems (Ustunel, 2013, Ghourchian et al., 2017, Polloreno, 12 Jan 2026, Samid, 2010).