Spectral Relative Entropy in Gaussian Processes

• Spectral Relative Entropy is defined as the relative entropy rate of stationary Gaussian processes in the frequency domain, serving as a pseudo-distance between spectral densities.
• The approach leverages the equality of time-domain and spectral-domain relative entropy rates to align process-level discrepancies with spectral approximation.
• A globally convergent matricial Newton-type algorithm efficiently solves the dual optimization, yielding improved complexity and robust multivariate spectral estimation.

Searching arXiv for recent and foundational papers on spectral relative entropy and closely related usages. First, a targeted search for the exact phrase. Searching arXiv for: spectral relative entropy Spectral relative entropy, in the sense introduced for multivariate spectral estimation, is the relative entropy rate of jointly stationary Gaussian processes written directly in the frequency domain. In this formulation it is a pseudo-distance between spectral densities, not symmetric in general, and it provides the objective function for a constrained spectrum approximation problem that extends the THREE approach of Byrnes, Georgiou and Lindquist and follows the Burg-Jaynes Maximum Entropy Method. Its central theoretical feature is that, for stationary Gaussian processes, the time-domain relative entropy rate equals the spectral-domain relative entropy rate; its central algorithmic consequence is a multivariate spectral estimator based on the multivariate Itakura-Saito distance, with an improved complexity upper bound and a globally convergent matricial Newton-type solution method (Ferrante et al., 2011).

1. Definition and core properties

For two stationary Gaussian processes $y$ and $z$, with spectral densities $\Phi_y$ and $\Phi_z$ that are $m \times m$ positive-definite, coercive matrix functions, the spectral relative entropy, or relative entropy rate, is defined by

$$D_{\mathrm{RER}}(\Phi_y \| \Phi_z) = \frac{1}{4\pi}\int_{-\pi}^{\pi} \left\{ \log\det\!\left(\Phi_y^{-1}(e^{j\vartheta}) \Phi_z(e^{j\vartheta})\right) + \operatorname{tr}\!\left[ \Phi_z^{-1}(e^{j\vartheta})(\Phi_y(e^{j\vartheta})-\Phi_z(e^{j\vartheta})) \right] \right\} \,d\vartheta .$$

This quantity is non-negative and equals zero if and only if $\Phi_y=\Phi_z$ almost everywhere, but it is not symmetric in general (Ferrante et al., 2011).

In the scalar case, spectral relative entropy is directly tied to the Itakura-Saito distance: $$d_{IS}(\Phi,\Psi)=\frac{1}{2\pi}\int_{-\pi}^{\pi} \left\{ \frac{\Phi(e^{j\vartheta})}{\Psi(e^{j\vartheta})} - \log \frac{\Phi(e^{j\vartheta})}{\Psi(e^{j\vartheta})} - 1 \right\}d\vartheta,$$ with

$$D_{\mathrm{RER}}(\Phi,\Psi)=\frac{1}{2}d_{IS}(\Phi,\Psi).$$

This scalar relation is the mechanism by which the new multivariate estimator inherits the Itakura-Saito criterion while remaining information-theoretically interpretable (Ferrante et al., 2011).

The paper also places the construction against the background of classical relative entropy for zero-mean Gaussian laws,

$$D(p\|q)=\frac{1}{2}\left[\log\det(P^{-1}Q)+\operatorname{tr}(Q^{-1}P)-n\right],$$

and differential entropy,

$z$0

so that the frequency-domain expression is not an ad hoc divergence on spectra, but a rate quantity induced by the underlying process laws (Ferrante et al., 2011).

2. Equality of time-domain and spectral-domain relative entropy rates

A central theorem states that the time-domain relative entropy rate between two stationary Gaussian processes equals their spectral-domain relative entropy rate. If $z$1 and $z$2, the time-domain rate is defined by

$z$3

and the paper establishes

$z$4

with both sides given by the spectral integral above (Ferrante et al., 2011).

This identity is the decisive justification for using spectral relative entropy as the criterion in spectrum approximation. The distance being minimized in the frequency domain is exactly the relative entropy rate of the corresponding stationary Gaussian processes. In that sense, the spectrum approximation problem is also a process-level approximation problem, and the usual separation between time-domain statistical distinguishability and frequency-domain fitting disappears for this class of models (Ferrante et al., 2011).

A plausible implication is that the method gains its robustness from this equivalence: matching spectral data under the relative entropy criterion is simultaneously controlling a rate of discrepancy between entire stochastic laws, not only between pointwise spectral curves.

3. Spectral estimation as constrained spectrum approximation

The estimation problem is posed for data samples $z$5 from an unknown zero-mean $z$6-variate stationary Gaussian process, together with a prior spectral density $z$7, a bank of stable filters with transfer function $z$8, and the empirical state covariance $z$9 of the filter driven by the data. The estimator is defined by the constrained optimization problem

$\Phi_y$0

Among all spectra matching the filter-output covariance constraint, the selected $\Phi_y$1 is the one closest to the prior in spectral relative entropy (Ferrante et al., 2011).

This formulation is explicitly presented as an extension of THREE to the multivariate setting. The paper recasts spectral estimation as a constrained spectrum approximation problem in which the distance is equal to the processes relative entropy rate, and the solution satisfies the McMillan degree bound

$\Phi_y$2

where $\Phi_y$3 is the order of the filter $\Phi_y$4. The paper emphasizes that this improves the previous multichannel bound $\Phi_y$5, and that it coincides with the bound featured by THREE in the scalar case (Ferrante et al., 2011).

Finite-data feasibility is treated directly. Because the empirical covariance $\Phi_y$6 may fail to satisfy feasibility constraints, the method replaces it by the closest feasible positive definite $\Phi_y$7 in Kullback-Leibler distance, so that the filter constraint remains satisfiable (Ferrante et al., 2011).

A related later development, aimed at $\Phi_y$8-signals, uses the multidimensional Itakura-Saito distance as the optimization criterion for the covariance extension problem of stationary random vector fields. In that setting, a discrete formulation is adopted to avoid technicalities on the boundary of the feasible set, and the resulting problem has a unique solution that depends smoothly on the problem data (Zhu et al., 2019). This suggests a natural line of generalization from one-dimensional multivariate processes to multivariate multidimensional random fields.

4. Dual optimization and Newton-type computation

The computational method proceeds through the dual problem. The dual functional is strictly convex; it is twice differentiable and strongly convex; and the optimal spectrum is recovered from the dual solution. The paper’s main algorithmic contribution is a globally convergent matricial Newton-type algorithm for this dual problem (Ferrante et al., 2011).

At each iteration, the search direction is obtained by solving for a matrix increment through associated Lyapunov or Riccati equations, using a basis for the feasible Lagrange multipliers. The step length is adjusted by backtracking so as to preserve feasibility and positivity. The implementation relies on spectral factorization and Lyapunov equations to make the computation tractable in the multivariate case (Ferrante et al., 2011).

The significance of this algorithm is twofold. First, it closes the gap between the variational formulation and an implementable estimator; second, it aligns the improved degree bound with a practical numerical method rather than a merely existential characterization. The combination of a strictly convex dual problem and a globally convergent Newton-type scheme is the mechanism by which the estimator becomes computationally usable in multichannel settings (Ferrante et al., 2011).

5. Simulation evidence and short-record behavior

The simulation study is organized around spectral resolution, multivariate performance, model order, and short-data behavior. In the scalar case, the estimator can accurately resolve very close spectral lines buried in colored noise, even below the resolution limit of the periodogram, provided the filterbank poles are placed close to the unit circle and at the correct frequencies (Ferrante et al., 2011).

In the multivariate case, the paper reports that the relative-entropy-based method surpasses both classical maximum entropy and Hellinger-distance estimators. The estimation error in spectral norm is significantly reduced, and the resulting spectra have lower or equal order than those produced by previous multivariate methods, consistent with the improved complexity bound (Ferrante et al., 2011).

The short-record regime is highlighted as a particularly favorable use case. For sample size $\Phi_y$9, the relative-entropy-based estimates remain stable and artifact-free, whereas Matlab’s standard PEM and subspace N4SID methods exhibit spurious spikes or artifacts (Ferrante et al., 2011). The paper therefore presents the method as especially effective for short data records.

These numerical observations are empirical rather than asymptotic. A cautious reading is that the method’s performance depends not only on the divergence criterion, but also on the informative placement of filterbank poles and on the feasibility correction for estimated covariances. Even so, the reported evidence supports the claim that the time-spectral relative entropy formulation is operationally effective, not merely theoretically elegant (Ferrante et al., 2011).

6. Broader usages and terminological extensions

The phrase “spectral relative entropy” is not confined to multivariate spectral estimation. In the relative entropic uncertainty relation, the continuous-spectrum form is

$\Phi_z$0

and the framework is applied to observables with either discrete or continuous spectra by comparing outcome distributions with suitable reference distributions of maximum entropy. For position and momentum, the reference distributions are Gaussians with the same mean and variance as the measured distributions (Floerchinger et al., 2020). Here, “spectral” refers to measurement spectra rather than to power spectral densities of stationary processes.

In elliptic optical microcavities, relative entropy is used to compare the normalized intensity patterns of eigenmodes in Hermitian and non-Hermitian systems,

$\Phi_z$1

with the average value large in the range of the collective Lamb shift and small in the range of self-energy. In that context it is a mode-pattern distinguishability measure and is used to describe mixing and exchange of eigenmodes (Jeong et al., 2020).

In radio astronomy, spectral relative entropy is defined as the Kullback-Leibler divergence between the empirical distribution of quantized channelized voltage data from a frequency channel and a reference Gaussian distribution fitted with the same sample mean and variance. Both asymmetrical and symmetrical versions are used for radio-frequency interference detection, and the study reports significant signal-to-noise improvements through the application of symmetrical and asymmetrical SRE, alongside spectral entropy, spectral kurtosis, and the Shapiro-Wilks test (Cao et al., 2024).

In Gaussian statistical field theory, relative entropy between Gaussian measures admits a spectral expression in terms of the eigenvalues $\Phi_z$2 of the operator $\Phi_z$3,

$\Phi_z$4

and for different nonzero masses with the same classical boundary conditions it is finite if and only if $\Phi_z$5 (Schröfl et al., 2023). In this usage, “spectral” refers to the eigenvalue spectrum of covariance operators.

This suggests that spectral relative entropy is a context-dependent label rather than a single universally fixed object. The technically specific meaning introduced in multivariate spectral estimation remains the one tied to stationary Gaussian processes, frequency-domain power spectra, the Itakura-Saito distance, and the equality between time-domain and frequency-domain relative entropy rates (Ferrante et al., 2011).