Joint Identification of Linear Dynamics and Noise Covariance via Distributional Estimation

Abstract: In this paper, we propose a novel framework for the joint identification of system dynamics and noise covariance in linear systems, under general noise distributions beyond Gaussian. Specifically, we would like to simultaneously estimate the dynamical matrix $A$ and the noise covariance matrix $\varSigma$ using state transition data. The formulation builds upon a novel parameterization of the state-transition distribution, which enables more effective use of distributional "shape" information for improved identification accuracy. We introduce two practical estimators, namely the maximum likelihood estimator (MLE) and the score-matching estimator (SME), to solve the joint dynamics-covariance identification problem, and provide rigorous analysis of their statistical properties and sample complexity. Simulation results show that the proposed estimators outperform the ordinary least squares (OLS) baseline.

• The paper introduces a joint identification framework that simultaneously estimates the dynamics matrix and noise covariance in linear systems.
• It leverages a φ-density parameterization and develops MLE and SME estimators that outperform traditional OLS, especially under non-Gaussian noise conditions.
• Empirical and theoretical analyses demonstrate improved estimation accuracy, robustness, and scalability in high-dimensional settings.

Joint Identification of Linear Dynamics and Noise Covariance via Distributional Estimation

Overview

The paper "Joint Identification of Linear Dynamics and Noise Covariance via Distributional Estimation" (2604.14130) introduces a novel joint identification framework for simultaneously estimating the dynamic matrix $A$ and noise covariance matrix $\Sigma$ in linear systems under general noise distributions. The approach leverages parameterization of the state-transition distribution through $\phi$-density families, capturing not only first and second moments but also higher-order distributional "shape" information absent in conventional Gaussian-based models. The work formally develops and analyzes two estimators—maximum likelihood estimator (MLE) and score-matching estimator (SME)—and demonstrates their statistical superiority to the ordinary least squares (OLS) baseline through rigorous theoretical analysis and empirical results.

Problem Formulation and Parameterization

The identification problem is formulated for linear time-invariant (LTI) systems:

$x' = Ax + \Sigma^{1/2}w,$

with $w$ an i.i.d. base noise (zero mean, unit covariance), and both $A$ and $\Sigma$ unknown. Unlike traditional approaches, which typically treat either the dynamics or the covariance separately, the proposed framework unifies the identification task through a $\phi$-density family parameterization:

$$f_{A, \Sigma}(x'|x) = \det(\Sigma)^{-1/2} \phi(\Sigma^{-1/2}(x' - Ax)),$$

where $\phi$ denotes the base density (not restricted to Gaussian). This generalization allows for explicit modeling of non-Gaussian structural properties such as tail behavior and higher-order moments, fundamentally extending the scope of system identification.

Elliptical Families and Computational Tractability

The analysis highlights elliptical density families, wherein the base density $\Sigma$0 admits the form $\Sigma$1. This property affords simple expressions for the gradient and Hessian of the log-density:

$\Sigma$2

with $\Sigma$3 determined by derivatives of $\Sigma$4. Standard distributions such as Gaussian and Student-t fall within this class, enabling tractable re-weighted OLS-like forms in algorithmic characterization.

Estimator Construction and Theoretical Properties

Ordinary Least Squares (OLS) Limitations

The naive OLS estimator is shown to be optimal only for Gaussian base densities, failing to exploit non-Gaussian shape information. This leads to suboptimal identification performance when the underlying noise model deviates from Gaussian, as evidenced by analytic examples and comparative characterizations.

Maximum Likelihood Estimator (MLE)

MLE is constructed to maximize the empirical log-likelihood over the parameter space:

$\Sigma$5

with theoretical guarantees:

• Consistency: Unique maximization at the true parameters under standard regularity conditions.
• Asymptotic Normality: The estimator converges at rate $\Sigma$6 with covariance determined by Fisher information.
• Reweighted OLS Characterization (Elliptical Case): The MLE coincides with OLS only in the Gaussian case; otherwise, it employs a data-dependent reweighting based on $\Sigma$7.

Score-Matching Estimator (SME)

SME leverages the principle of matching score functions (gradients of log-densities), leading to the estimator:

$\Sigma$8

SME maintains unbiasedness and asymptotic normality (under additional regularity conditions), and, for elliptical families, produces a reweighted OLS form distinct from both naive OLS and MLE.

Unified Estimators Under Gaussian Noise

For Gaussian base density, MLE, SME, and OLS are mathematically equivalent, affirming classical results and providing a baseline for comparison.

Sample Complexity Analysis

Under the assumption of single-trajectory data with sub-Gaussian noise and system stability, both MLE and SME achieve

$\Sigma$9

matching standard non-asymptotic MLE rates. This demonstrates statistical efficiency of the estimators beyond classical conditions, with robustness to non-Gaussian noise.

Figure 1: Identification error of MLE, SME and OLS estimators as a function of sample size, highlighting convergence rates and confidence intervals in a 2-dimensional Student-t family.

Simulation Results

Sample Complexity and Comparative Performance

Empirical results confirm theoretical rates and show that MLE consistently achieves lower identification errors than OLS and SME, especially for $\phi$0. SME typically outperforms OLS, but lags behind MLE. These differences matter most with heavy-tailed (non-Gaussian) noise distributions.

Figure 2: Identification error of MLE and OLS as a function of system dimension (Student-t family), where MLE gains increase with higher dimensionality.

Scalability

As system dimensionality increases, the gap favoring MLE over OLS widens, attributable to increased deviation from Gaussianity with larger state spaces and heavier tails.

Computational Efficiency

Both MLE and SME demand higher computational resources due to nonlinear optimization, with iterative algorithms offering substantial time reductions for MLE in high-dimensional or large-sample settings.

Figure 3: Identification error for different MLE implementations, demonstrating efficient convergence of iterative MLE in two dimensions.

Figure 4: Computational time for MLE, SME, and OLS as a function of sample size and dimension, indicating a trade-off between statistical efficiency and computational cost.

Robustness to Base Density Misspecification

Experiments with perturbed Student-t base densities reveal that MLE's accuracy for $\phi$1 degrades gradually with increasing misspecification, while $\phi$2 estimation remains relatively robust, consistent with theoretical drift bounds.

Figure 5: Identification error of MLE under increasing base density misspecification, showing robustness and performance attenuation with higher error levels.

Implications and Future Directions

The joint identification framework substantially extends the applicability of data-driven system identification to settings with non-Gaussian noise, enabling improved estimation of both dynamics and noise structure. Practically, this capability is critical for robust control, adaptive Kalman filtering, and risk-sensitive applications where distributional properties beyond nominal parameters are vital. Theoretically, the generalization to $\phi$3-density families opens new avenues for semiparametric identification, minimum divergence estimators, and learning stability analysis under complex noise.

Computational trade-offs emerge as salient: higher sample efficiency and accuracy (especially for high dimensional or complex noise) require more optimization resources, motivating further research into algorithmic acceleration, convex relaxation, and scalable iterative schemes.

Prospective directions include expanding the estimator suite to additional distributional families, integrating model uncertainty, and adapting the framework for online and distributed settings.

Conclusion

The paper establishes a rigorous framework for joint system identification in linear models with arbitrary noise structure by leveraging distributional parameterization. The introduced MLE and SME estimators outperform OLS, particularly under non-Gaussian noise, achieving optimal sample complexity rates with formal theoretical and empirical validation. The results underscore the necessity to harness rich distributional information for improved identification accuracy and stimulate future research toward scalable and robust algorithms for complex dynamical systems.