Directional-Fisher Rate Overview

• Directional-Fisher Rate is the projection of the asymptotic Fisher information rate matrix along a specified unit vector in parameter space, measuring learning efficiency.
• It is derived using closed-form expressions from unifilar presentations and belief-state metadynamics, applicable to both Markovian and non-Markovian processes.
• The concept links predictive state dynamics to precise Cramér–Rao bounds, thereby characterizing direction-dependent estimation variance scaling.

Directional-Fisher Rate is the directional projection of the Fisher information rate matrix for a parametrized stochastic process observed through a time series. In the framework of sequence probabilities $P_\theta(X_{1:L})$, it quantifies asymptotic learning efficiency along a specified unit direction $u$ in parameter space, after normalizing Fisher information by observation length. The concept is introduced within a general theory of learning unknown parameters of stochastic processes, including non-Markovian processes of infinite Markov order, where the Fisher information of observed sequence probabilities lower-bounds estimation variance and admits exact closed-form asymptotics through unifilar presentations and belief-state metadynamics (Riechers, 2023).

1. Formal setting and definition

Consider a parameter vector $\theta \in \Theta$ and an observed sequence $X_{1:L}=X_1,X_2,\ldots,X_L$ over a countable alphabet $\mathcal{X}$, with sequence probability

$$P_\theta(X_{1:L}=x_{1:L}) = P_\theta(x_1)\,P_\theta(x_2\mid x_1)\cdots P_\theta(x_L\mid x_{1:L-1}) \,.$$

For vector parameters, the Fisher information matrix for sequences of length $L$ is

$$I_L(\theta)_{m,n} = \Big\langle \big[\partial_{\theta_m}\ln P_\theta(X_{1:L})\big] \big[\partial_{\theta_n}\ln P_\theta(X_{1:L})\big] \Big\rangle_{P_\theta(X_{1:L})} = -\Big\langle \partial_{\theta_m}\partial_{\theta_n}\ln P_\theta(X_{1:L}) \Big\rangle_{P_\theta(X_{1:L})} \,,$$

under standard regularity conditions, yielding a symmetric positive semidefinite matrix. In the scalar-parameter case,

$$I_L(\theta) = \Big\langle\big[\partial_\theta \ln P_\theta(X_{1:L})\big]^2\Big\rangle = -\Big\langle\partial_\theta^2 \ln P_\theta(X_{1:L})\Big\rangle \,.$$

The asymptotic per-symbol Fisher information rate is

$$I_F(\theta) = \lim_{L\to\infty} \frac{1}{L}\,I_L(\theta) \,.$$

For stationary processes with finite excess information $u$0, the Fisher information scales as

$u$1

The Directional-Fisher Rate is then defined for a unit direction $u$2 in parameter space by

$u$3

It is therefore the quadratic form of the Fisher information rate matrix along a chosen parameter-space direction. In one dimension, the directional quantity coincides with the scalar Fisher information rate (Riechers, 2023).

2. Closed-form expression and unifilar presentations

A central result is an exact closed-form expression for the Fisher information rate via any unifilar presentation, valid even for non-Markovian processes of infinite Markov order. Suppose a unifilar hidden-state model with countable recurrent states $u$4 and stationary distribution $u$5 over $u$6, and let $u$7 be the one-step symbol probabilities. Then

$u$8

and equivalently

$u$9

Applying the directional projection yields

$\theta \in \Theta$0

or, equivalently,

$\theta \in \Theta$1

These formulas are valid for any unifilar presentation that faithfully represents the parametrized process in a neighborhood of $\theta \in \Theta$2, including non-Markovian processes. The same framework applies when the unifilar presentation is obtained from the mixed-state presentation of a nonunifilar HMM. This suggests that the Directional-Fisher Rate is not restricted to finite-order Markov models, but is instead a property of the predictive state dynamics of the process (Riechers, 2023).

3. Belief-state metadynamics and finite-length convergence

The finite-length approach to the asymptotic rate proceeds through estimation states and the mixed-state presentation. Estimation states $\theta \in \Theta$3 are equivalence classes of histories that induce the same conditional distribution over futures uniformly in a neighborhood of $\theta \in \Theta$4. The mixed-state presentation constructs unifilar dynamics over belief distributions $\theta \in \Theta$5 over hidden HMM states, updated by Bayes under observed symbols.

The labeled transition operators over estimation states are

$\theta \in \Theta$6

with left stationary eigenvector $\theta \in \Theta$7 over recurrent estimation states and right eigenvector of ones.

The myopic Fisher information rate is the finite-length per-step increment

$\theta \in \Theta$8

with the matrix form

$\theta \in \Theta$9

where the information vector is $X_{1:L}=X_1,X_2,\ldots,X_L$0-independent and has components

$X_{1:L}=X_1,X_2,\ldots,X_L$1

If there is a single attractor, $X_{1:L}=X_1,X_2,\ldots,X_L$2 and

$X_{1:L}=X_1,X_2,\ldots,X_L$3

so the Fisher information rate is the stationary-average of the information vector. The Directional-Fisher Rate is then the corresponding stationary average after projection along $X_{1:L}=X_1,X_2,\ldots,X_L$4.

The spectral decomposition of powers of $X_{1:L}=X_1,X_2,\ldots,X_L$5 yields the convergence structure of $X_{1:L}=X_1,X_2,\ldots,X_L$6 toward $X_{1:L}=X_1,X_2,\ldots,X_L$7. Contributions with $X_{1:L}=X_1,X_2,\ldots,X_L$8 yield exponential decay, possibly modulated by polynomials; $X_{1:L}=X_1,X_2,\ldots,X_L$9 yields ephemeral terms up to $\mathcal{X}$0; and unit-modulus non-1 eigenvalues yield oscillatory modes. The same transient spectrum governs the convergence of the myopic Shannon entropy rate $\mathcal{X}$1 to the Shannon entropy rate $\mathcal{X}$2. Accordingly, $\mathcal{X}$3 and $\mathcal{X}$4 with exactly the same set of relaxation eigenvalues and timescales drawn from the transient spectrum of the mixed-state presentation (Riechers, 2023).

4. Excess information and asymptotic estimation limits

Finite-length deviations from the asymptotic linear scaling are captured by excess information: $\mathcal{X}$5 With $\mathcal{X}$6 invertible,

$\mathcal{X}$7

This parallels the corresponding formulas for excess entropy. For finite Markov-order $\mathcal{X}$8 processes, myopic excess information saturates at $\mathcal{X}$9, while in many cases saturation occurs at the cryptic order $$P_\theta(X_{1:L}=x_{1:L}) = P_\theta(x_1)\,P_\theta(x_2\mid x_1)\cdots P_\theta(x_L\mid x_{1:L-1}) \,.$$0. A plausible implication is that finite-length learning transients are controlled not only by Markov order but by the predictive state structure encoded in the cryptic organization of the process.

The estimation-theoretic significance of $$P_\theta(X_{1:L}=x_{1:L}) = P_\theta(x_1)\,P_\theta(x_2\mid x_1)\cdots P_\theta(x_L\mid x_{1:L-1}) \,.$$1 and its directional projection follows from the Cramér–Rao lower bound. For unbiased estimators based on a single length-$$P_\theta(X_{1:L}=x_{1:L}) = P_\theta(x_1)\,P_\theta(x_2\mid x_1)\cdots P_\theta(x_L\mid x_{1:L-1}) \,.$$2 sequence,

$$P_\theta(X_{1:L}=x_{1:L}) = P_\theta(x_1)\,P_\theta(x_2\mid x_1)\cdots P_\theta(x_L\mid x_{1:L-1}) \,.$$3

Using

$$P_\theta(X_{1:L}=x_{1:L}) = P_\theta(x_1)\,P_\theta(x_2\mid x_1)\cdots P_\theta(x_L\mid x_{1:L-1}) \,.$$4

the asymptotic bound becomes

$$P_\theta(X_{1:L}=x_{1:L}) = P_\theta(x_1)\,P_\theta(x_2\mid x_1)\cdots P_\theta(x_L\mid x_{1:L-1}) \,.$$5

For $$P_\theta(X_{1:L}=x_{1:L}) = P_\theta(x_1)\,P_\theta(x_2\mid x_1)\cdots P_\theta(x_L\mid x_{1:L-1}) \,.$$6 independent sequences of length $$P_\theta(X_{1:L}=x_{1:L}) = P_\theta(x_1)\,P_\theta(x_2\mid x_1)\cdots P_\theta(x_L\mid x_{1:L-1}) \,.$$7, $$P_\theta(X_{1:L}=x_{1:L}) = P_\theta(x_1)\,P_\theta(x_2\mid x_1)\cdots P_\theta(x_L\mid x_{1:L-1}) \,.$$8.

Since the Directional-Fisher Rate is $$P_\theta(X_{1:L}=x_{1:L}) = P_\theta(x_1)\,P_\theta(x_2\mid x_1)\cdots P_\theta(x_L\mid x_{1:L-1}) \,.$$9, it specifies the asymptotic learning efficiency in the direction $L$0. This suggests that anisotropy of the Fisher information rate matrix induces direction-dependent identifiability and direction-dependent variance scaling (Riechers, 2023).

5. Computation for Markov chains, HMMs, and infinite-state presentations

For finite-order Markov chains, the procedure is explicit. One takes the unifilar states to be the last $L$1 symbols, or a minimal equivalent, computes the stationary distribution $L$2 over recurrent unifilar states, evaluates $L$3 and their derivatives, forms the information vector components

$L$4

and then averages over $L$5: $L$6 For finite Markov order, the transient $L$7 has only zero eigenvalues with index $L$8, yielding exact convergence for $L$9. Often saturation occurs at cryptic order $$I_L(\theta)_{m,n} = \Big\langle \big[\partial_{\theta_m}\ln P_\theta(X_{1:L})\big] \big[\partial_{\theta_n}\ln P_\theta(X_{1:L})\big] \Big\rangle_{P_\theta(X_{1:L})} = -\Big\langle \partial_{\theta_m}\partial_{\theta_n}\ln P_\theta(X_{1:L}) \Big\rangle_{P_\theta(X_{1:L})} \,,$$0.

For HMMs, including infinite Markov order, one begins from an HMM $$I_L(\theta)_{m,n} = \Big\langle \big[\partial_{\theta_m}\ln P_\theta(X_{1:L})\big] \big[\partial_{\theta_n}\ln P_\theta(X_{1:L})\big] \Big\rangle_{P_\theta(X_{1:L})} = -\Big\langle \partial_{\theta_m}\partial_{\theta_n}\ln P_\theta(X_{1:L}) \Big\rangle_{P_\theta(X_{1:L})} \,,$$1, constructs mixed states

$$I_L(\theta)_{m,n} = \Big\langle \big[\partial_{\theta_m}\ln P_\theta(X_{1:L})\big] \big[\partial_{\theta_n}\ln P_\theta(X_{1:L})\big] \Big\rangle_{P_\theta(X_{1:L})} = -\Big\langle \partial_{\theta_m}\partial_{\theta_n}\ln P_\theta(X_{1:L}) \Big\rangle_{P_\theta(X_{1:L})} \,,$$2

groups them into estimation states $$I_L(\theta)_{m,n} = \Big\langle \big[\partial_{\theta_m}\ln P_\theta(X_{1:L})\big] \big[\partial_{\theta_n}\ln P_\theta(X_{1:L})\big] \Big\rangle_{P_\theta(X_{1:L})} = -\Big\langle \partial_{\theta_m}\partial_{\theta_n}\ln P_\theta(X_{1:L}) \Big\rangle_{P_\theta(X_{1:L})} \,,$$3, and computes

$$I_L(\theta)_{m,n} = \Big\langle \big[\partial_{\theta_m}\ln P_\theta(X_{1:L})\big] \big[\partial_{\theta_n}\ln P_\theta(X_{1:L})\big] \Big\rangle_{P_\theta(X_{1:L})} = -\Big\langle \partial_{\theta_m}\partial_{\theta_n}\ln P_\theta(X_{1:L}) \Big\rangle_{P_\theta(X_{1:L})} \,,$$4

The metadynamics matrices $$I_L(\theta)_{m,n} = \Big\langle \big[\partial_{\theta_m}\ln P_\theta(X_{1:L})\big] \big[\partial_{\theta_n}\ln P_\theta(X_{1:L})\big] \Big\rangle_{P_\theta(X_{1:L})} = -\Big\langle \partial_{\theta_m}\partial_{\theta_n}\ln P_\theta(X_{1:L}) \Big\rangle_{P_\theta(X_{1:L})} \,,$$5 are then built, the stationary distribution over recurrent estimation states is obtained, and the same stationary averaging formula yields $$I_L(\theta)_{m,n} = \Big\langle \big[\partial_{\theta_m}\ln P_\theta(X_{1:L})\big] \big[\partial_{\theta_n}\ln P_\theta(X_{1:L})\big] \Big\rangle_{P_\theta(X_{1:L})} = -\Big\langle \partial_{\theta_m}\partial_{\theta_n}\ln P_\theta(X_{1:L}) \Big\rangle_{P_\theta(X_{1:L})} \,,$$6 and $$I_L(\theta)_{m,n} = \Big\langle \big[\partial_{\theta_m}\ln P_\theta(X_{1:L})\big] \big[\partial_{\theta_n}\ln P_\theta(X_{1:L})\big] \Big\rangle_{P_\theta(X_{1:L})} = -\Big\langle \partial_{\theta_m}\partial_{\theta_n}\ln P_\theta(X_{1:L}) \Big\rangle_{P_\theta(X_{1:L})} \,,$$7.

For binary alphabets, the information vector simplifies to

$$I_L(\theta)_{m,n} = \Big\langle \big[\partial_{\theta_m}\ln P_\theta(X_{1:L})\big] \big[\partial_{\theta_n}\ln P_\theta(X_{1:L})\big] \Big\rangle_{P_\theta(X_{1:L})} = -\Big\langle \partial_{\theta_m}\partial_{\theta_n}\ln P_\theta(X_{1:L}) \Big\rangle_{P_\theta(X_{1:L})} \,,$$8

For processes with no finite unifilar HMM, such as Parentheses Matching, the mixed-state presentation is constructed directly over the possibly infinite set of mixed states generated by the HMM, and one proceeds on the recurrent component. For nonergodic mixtures with disconnected components, one computes $$I_L(\theta)_{m,n} = \Big\langle \big[\partial_{\theta_m}\ln P_\theta(X_{1:L})\big] \big[\partial_{\theta_n}\ln P_\theta(X_{1:L})\big] \Big\rangle_{P_\theta(X_{1:L})} = -\Big\langle \partial_{\theta_m}\partial_{\theta_n}\ln P_\theta(X_{1:L}) \Big\rangle_{P_\theta(X_{1:L})} \,,$$9 within each block and combines with mixing weights; directional projection again takes the form $$I_L(\theta) = \Big\langle\big[\partial_\theta \ln P_\theta(X_{1:L})\big]^2\Big\rangle = -\Big\langle\partial_\theta^2 \ln P_\theta(X_{1:L})\Big\rangle \,.$$0 (Riechers, 2023).

6. Representative processes and explicit forms

The framework admits exact expressions for a range of qualitatively distinct stochastic processes.

For the IID biased coin with scalar parameter $$I_L(\theta) = \Big\langle\big[\partial_\theta \ln P_\theta(X_{1:L})\big]^2\Big\rangle = -\Big\langle\partial_\theta^2 \ln P_\theta(X_{1:L})\Big\rangle \,.$$1,

$$I_L(\theta) = \Big\langle\big[\partial_\theta \ln P_\theta(X_{1:L})\big]^2\Big\rangle = -\Big\langle\partial_\theta^2 \ln P_\theta(X_{1:L})\Big\rangle \,.$$2

The myopic excess information is $$I_L(\theta) = \Big\langle\big[\partial_\theta \ln P_\theta(X_{1:L})\big]^2\Big\rangle = -\Big\langle\partial_\theta^2 \ln P_\theta(X_{1:L})\Big\rangle \,.$$3, and $$I_L(\theta) = \Big\langle\big[\partial_\theta \ln P_\theta(X_{1:L})\big]^2\Big\rangle = -\Big\langle\partial_\theta^2 \ln P_\theta(X_{1:L})\Big\rangle \,.$$4 for all $$I_L(\theta) = \Big\langle\big[\partial_\theta \ln P_\theta(X_{1:L})\big]^2\Big\rangle = -\Big\langle\partial_\theta^2 \ln P_\theta(X_{1:L})\Big\rangle \,.$$5.

For the General $$I_L(\theta) = \Big\langle\big[\partial_\theta \ln P_\theta(X_{1:L})\big]^2\Big\rangle = -\Big\langle\partial_\theta^2 \ln P_\theta(X_{1:L})\Big\rangle \,.$$6–$$I_L(\theta) = \Big\langle\big[\partial_\theta \ln P_\theta(X_{1:L})\big]^2\Big\rangle = -\Big\langle\partial_\theta^2 \ln P_\theta(X_{1:L})\Big\rangle \,.$$7 Golden Mean process with scalar parameter $$I_L(\theta) = \Big\langle\big[\partial_\theta \ln P_\theta(X_{1:L})\big]^2\Big\rangle = -\Big\langle\partial_\theta^2 \ln P_\theta(X_{1:L})\Big\rangle \,.$$8,

$$I_L(\theta) = \Big\langle\big[\partial_\theta \ln P_\theta(X_{1:L})\big]^2\Big\rangle = -\Big\langle\partial_\theta^2 \ln P_\theta(X_{1:L})\Big\rangle \,.$$9

which yields

$$I_F(\theta) = \lim_{L\to\infty} \frac{1}{L}\,I_L(\theta) \,.$$0

Convergence satisfies $$I_F(\theta) = \lim_{L\to\infty} \frac{1}{L}\,I_L(\theta) \,.$$1 for $$I_F(\theta) = \lim_{L\to\infty} \frac{1}{L}\,I_L(\theta) \,.$$2, showing cryptic-order saturation.

For the Even Process, which has infinite Markov order,

$$I_F(\theta) = \lim_{L\to\infty} \frac{1}{L}\,I_L(\theta) \,.$$3

The convergence is oscillatory with period $$I_F(\theta) = \lim_{L\to\infty} \frac{1}{L}\,I_L(\theta) \,.$$4: $$I_F(\theta) = \lim_{L\to\infty} \frac{1}{L}\,I_L(\theta) \,.$$5 Its excess information is

$$I_F(\theta) = \lim_{L\to\infty} \frac{1}{L}\,I_L(\theta) \,.$$6

For the Teddy Bear process with two parameters $$I_F(\theta) = \lim_{L\to\infty} \frac{1}{L}\,I_L(\theta) \,.$$7,

$$I_F(\theta) = \lim_{L\to\infty} \frac{1}{L}\,I_L(\theta) \,.$$8

and for a unit direction $$I_F(\theta) = \lim_{L\to\infty} \frac{1}{L}\,I_L(\theta) \,.$$9,

$u$00

Its convergence shows ephemeral cutoff by cryptic order $u$01, after which a period-3 decaying mode dominates.

For overparametrized, non-identifiable models, Fisher information matrices are singular, but the Fisher information rate can still be computed. An example based on the Even process with $u$02 yields

$u$03

which is rank-$u$04, and

$u$05

In such cases, the Drazin inverse or Moore–Penrose pseudoinverse can be used to assess variance bounds orthogonal to invariant subspaces (Riechers, 2023).

7. Assumptions, scope, and conceptual significance

The theory assumes a countable alphabet $u$06 and discrete-time observations, with continuous-time obtained via an appropriate limit. Standard regularity conditions are required for Fisher information, including exchange of derivative and expectation. Stationarity is assumed when asserting limits and stationary distributions over recurrent estimation states. Unifilar presentations must be faithful in a neighborhood of $u$07, and the mixed-state presentation provides a canonical unifilar presentation even for nonunifilar HMMs. The spectrum of $u$08 is assumed to comprise isolated eigenvalues. When using $u$09 invertibility, deterministic periodicities on the recurrent component are excluded; otherwise unit-modulus modes must be included explicitly in spectral sums (Riechers, 2023).

Within these assumptions, the Directional-Fisher Rate furnishes an exact asymptotic measure of inferential sensitivity along selected parameter-space directions. Its role is simultaneously geometric, through the quadratic form $u$10; dynamical, through the mixed-state metadynamics and transient spectrum; and statistical, through the asymptotic Cramér–Rao scaling. The central structural result is that both learning and synchronization are governed by the same transient spectral data: the myopic Fisher-information rate and the myopic entropy rate relax with identical eigenvalues and timescales. This establishes a direct connection between predictive-state relaxation and asymptotic parameter-learning efficiency in both Markovian and non-Markovian stochastic processes (Riechers, 2023).