Covariate Fisher Information Matrix (cFIM)

• cFIM is a computable, finite-dimensional representation of Fisher information that quantifies extractable information from models with covariate dependencies.
• It leverages orthogonal decomposition and score functions to restrict the Fisher–Rao metric, ensuring positive definiteness and efficient variance bounds.
• cFIM is applied in time series, hierarchical, and error-in-variable models to enable tractable inference, principled dimensionality estimation, and minimal estimator variance.

The Covariate Fisher Information Matrix (cFIM) generalizes and operationalizes Fisher information in statistical models involving covariates, latent variables, high-dimensional geometry, and measurement error. It arises in diverse contexts, from non-parametric information geometry to autoregressive time series and hierarchical models with latent covariate structures. The cFIM provides a finite-dimensional and computable representative of extractable information from complex or infinite-dimensional systems, enabling tractable inference, variance bounds, and principled dimensionality estimation.

1. Foundations: Orthogonal Decomposition and Finite Realization

In infinite-dimensional non-parametric information geometry, the set of all smooth, positive densities $f(x)$ on $\mathbb{R}^n$ forms a manifold $M$ with tangent space

$$T_fM = \left\{ h \in C^\infty(\mathbb{R}^n): \int h(x)\,dx = 0 \right\}$$

and Fisher–Rao metric

$$g_f(h_1, h_2) = \int_{\mathbb{R}^n} \frac{h_1(x) h_2(x)}{f(x)}\,dx.$$

The cFIM emerges by defining a finite-dimensional covariate subspace

$$S = \operatorname{span}\left\{ \frac{\partial f}{\partial x_i} : i=1,\dots,n \right\} \subset T_fM,$$

equivalently, in terms of score functions $s_i(x) = \partial_i \ln f(x)$,

$S = \operatorname{span}\{ s_i f : i=1,\dots,n \}.$

Via Hilbert space orthogonal decomposition,

$T_fM = S \oplus S^\perp,$

where $S^\perp$ is the residual subspace orthogonal to $S$. This construction allows restricting the Fisher–Rao metric to $S$, yielding a tractable $n$-dimensional matrix.

2. Definition and Properties of the cFIM

The Covariate Fisher Information Matrix $G_f$ is given by

$$(G_f)_{ij} = g_f(\partial_i f, \partial_j f) = \int \frac{\partial_i f(x) \partial_j f(x)}{f(x)}\,dx = \mathbb{E}_{X\sim f}[s_i(X) s_j(X)],$$

or

$$G_f = \left( \mathbb{E}_f[\partial_i \ln f \, \partial_j \ln f] \right)_{i,j=1}^n.$$

$G_f$ encapsulates all information available from the observed covariates. Under mild conditions, such as linear independence of the score functions in $L^2(f)$, $G_f$ is positive definite and invertible. The total explainable information in $f$ relative to the observed coordinates, termed G-entropy, is

$$H_G(f) := \mathbb{E}_f [\| \nabla \ln f(X) \|^2 ] = \operatorname{Tr}(G_f).$$

Hence, the trace of $G_f$ quantifies the statistical information captured by the distribution via the observable covariates (Cheng et al., 25 Dec 2025).

3. cFIM in Time Series and Conditional Inference

For logistic autoregressive (LARX) models with endogenous and exogenous covariates, the exact conditional Fisher information matrix (also labeled cFIM) corrects for autocorrelation and non-independence:

$$I_c(\vartheta | \mathcal{I}_0) = \sum_{t=p+1}^{T} \sum_{y_{-t}} \frac{e^{X_t^\top\alpha + Y_{-t}^\top\beta}}{[1+e^{X_t^\top\alpha + Y_{-t}^\top\beta}]^2} \,\phi_t \phi_t^\top \, Q_t(y_{t-1},\dots,y_{t-p}),$$

where $\vartheta = (\alpha, \beta)$ are parameters for exogenous and endogenous covariates, $\phi_t$ concatenates covariates and lagged responses, and $Q_t$ denotes the joint law of observed lag blocks. Recursive algorithms allow $O(T 2^p)$ computation, and the cFIM yields variance estimates that converge to asymptotic Fisher information as $T \to \infty$ (Gao et al., 2017).

4. cFIM for Hierarchical and Error-in-Variables Models

In settings where both coordinates $(X, Y)$ are measured with Gaussian error and arbitrary covariance, the “covariate Fisher-matrix” is constructed by marginalizing latent variables. For a model where $y = \mu(x, \theta)$ and observed covariances $C_{XX}, C_{YY}, C_{XY}$,

$$R = C_{YY} - C_{XY}^T T^T - T C_{XY} + T C_{XX} T^T,$$

with $T = \partial\mu/\partial x|_{x=X}$. The Fisher information is then computed via

$$F_{\alpha\beta} = \frac{1}{2} \operatorname{Tr}[ R^{-1} \partial_\alpha R R^{-1} \partial_\beta R ] + (\partial_\alpha \mu)^T R^{-1} (\partial_\beta \mu ),$$

enabling correct uncertainty quantification and propagation in hierarchical or measurement-error models (Heavens et al., 2014).

5. cFIM, KL-Divergence Curvature, and Covariate CRLB

The restricted Fisher–Rao metric corresponds to the curvature of the Kullback–Leibler divergence in covariate directions:

$$g_f(h, h) = \left.\frac{d^2}{dt^2}\right|_{t=0} D_{\mathrm{KL}}(f \| f_t)$$

with $h$ the tangent direction. The diagonal elements of $G_f$ are the second derivatives of $D_{\mathrm{KL}}$ along each coordinate. The Covariate Cramér–Rao Lower Bound (CRLB) asserts that, under regularity and alignment postulates,

$\mathrm{AsyCov}(\hat{\theta}) \succeq G_f^{-1}.$

Thus, cFIM establishes fundamental variance bounds for estimators in semi-parametric and nonparametric models (Cheng et al., 25 Dec 2025).

6. Semi-Parametric Efficiency and Geometric Congruence

In semi-parametric estimation with infinite-dimensional nuisance parameters, the efficient Fisher Information $I_{\mathrm{eff}}$ is the covariance of efficient scores, defined as projections onto the orthocomplement of the nuisance tangent space. Under the Geometric Alignment Postulate—that efficient scores coincide with covariate scores—

$G_f = I_{\mathrm{eff}}(\theta),$

which establishes congruence between cFIM and semi-parametric efficiency, dictating minimal estimator variance (Cheng et al., 25 Dec 2025).

7. Information Capture Ratio, Manifold Hypothesis, and Intrinsic Dimensionality

The Manifold Hypothesis posits data support on a $d$-dimensional submanifold ($d \ll n$). Under chain-rule and dominance assumptions, the signal subspace is

$$M_l = \operatorname{span}\{\partial_{y_j} \ln f : y \in \mathbb{R}^d\},$$

with $M_l \subset S$. Rank-deficiency of $G_f$ signals intrinsic dimensionality, and the Information Capture Ratio of the signal tangent space within $G_f$ provides a rigorous estimator of $d$, operationalizing the testability of the Manifold Hypothesis and facilitating intrinsic dimension estimation in high-dimensional data (Cheng et al., 25 Dec 2025).

Conclusion and Significance

The cFIM unifies multiple threads in modern statistics and information geometry, providing precise, computable measures of information for inference in situations ranging from non-parametric density estimation and model geometry to conditional time series and hierarchical models. It concretizes the link between geometric structures (such as the Fisher–Rao metric), regularization, and efficiency bounds, extending Fisher information to accommodate measurement error, endogenous autoregression, manifold structure, and latent variable uncertainty. Its implementation yields improved inference, narrower confidence intervals, and fundamental insights into dimensionality, signal representation, and statistical efficiency (Cheng et al., 25 Dec 2025, Gao et al., 2017, Heavens et al., 2014).