Strict positive definiteness of the Fisher information for general deep networks

Establish explicit sufficient conditions under which the population Fisher information matrix I(θ0) = E[∇θ fθ0(X) ∇θ fθ0(X)ᵀ] of multilayer (multi-hidden-layer) feedforward neural networks is strictly positive definite, extending the irreducibility-based criterion known for single-hidden-layer networks to general deep architectures.

Background

The paper contrasts identifiability properties of classical parametric models with those of neural networks. Local identifiability is linked to invertibility (strict positive definiteness) of the population Fisher information matrix I(θ0) = E[∇θ fθ0(X) ∇θ fθ0(X)ᵀ]. For one-hidden-layer networks, Fukumizu (1996) provided a sufficient condition (based on irreducibility) ensuring strict positive definiteness of the Fisher information for any positive continuous input density.

In the Discussion, the authors note that while such a condition exists for single-hidden-layer networks, the corresponding question for general deep networks with multiple hidden layers remains unresolved, motivating the need to determine conditions guaranteeing invertibility of the population Fisher information in deep architectures.