Deep Subspace Encoders for Nonlinear System Identification

Abstract: Using Artificial Neural Networks (ANN) for nonlinear system identification has proven to be a promising approach, but despite of all recent research efforts, many practical and theoretical problems still remain open. Specifically, noise handling and models, issues of consistency and reliable estimation under minimisation of the prediction error are the most severe problems. The latter comes with numerous practical challenges such as explosion of the computational cost in terms of the number of data samples and the occurrence of instabilities during optimization. In this paper, we aim to overcome these issues by proposing a method which uses a truncated prediction loss and a subspace encoder for state estimation. The truncated prediction loss is computed by selecting multiple truncated subsections from the time series and computing the average prediction loss. To obtain a computationally efficient estimation method that minimizes the truncated prediction loss, a subspace encoder represented by an artificial neural network is introduced. This encoder aims to approximate the state reconstructability map of the estimated model to provide an initial state for each truncated subsection given past inputs and outputs. By theoretical analysis, we show that, under mild conditions, the proposed method is locally consistent, increases optimization stability, and achieves increased data efficiency by allowing for overlap between the subsections. Lastly, we provide practical insights and user guidelines employing a numerical example and state-of-the-art benchmark results.

• The paper introduces a subspace encoder that leverages truncated prediction loss to efficiently approximate nonlinear state-space models.
• It employs overlapping time series subsections with ANN-based estimation to improve data efficiency and mitigate noise effects.
• Benchmark tests, including the Wiener-Hammerstein model, demonstrate state-of-the-art performance and robustness.

Deep Subspace Encoders for Nonlinear System Identification

This essay provides a comprehensive analysis of the research paper titled "Deep Subspace Encoders for Nonlinear System Identification" (2210.14816). The paper introduces a novel approach to nonlinear system identification using a deep subspace encoder network (SUBNET), leveraging artificial neural networks (ANNs) for modeling challenging nonlinear dynamics commonly found in engineering systems.

Introduction

Nonlinear system identification remains a challenging domain due to the vast range of nonlinear behaviors exhibited by various systems, such as mechatronic and chemical systems. Classical approaches, such as linear state-space models, often fail to capture the complexity inherent in nonlinear dynamics. The paper addresses this gap by utilizing ANNs to develop a subspace encoder capable of efficiently estimating nonlinear system states and dynamics.

The central contribution is the formulation of a subspace encoder that approximates the state reconstructability map using ANNs. This approach facilitates efficient estimation under truncated prediction losses, enhancing data efficiency and optimization stability compared to traditional methods.

Subspace Encoder Method

Truncated Prediction Loss

The paper introduces a truncated prediction loss computed over selected subsections of the time series, reducing computational cost and improving optimization stability. The loss function minimizes the average prediction error over truncated subsections, allowing for parallel computation and overcoming the computational challenges associated with long time series.

Figure 1: Overall SUBNET structure: the subspace encoder $\psi_\eta$ estimates the initial state at time index $t$ based on past inputs and outputs, then the state is propagated through $f_\theta$ and $h_\theta$ multiple times until the truncation length $T$. The parts marked in blue constitute the innovation noise process.

Subspace Encoder

The subspace encoder, represented by $\psi_\eta$, estimates the initial state based on past inputs and outputs. It approximates the reconstructability map within the nonlinear state-space (NL-SS) model framework, offering improved computational scalability and data efficiency. This encoder is simultaneously estimated along with the state-transition and output functions, ensuring a smooth approximation of the underlying state dynamics.

Parameter Estimation and Guidelines

The paper outlines a detailed parameter estimation procedure, leveraging the Adam optimizer within a batch framework. It highlights essential user guidelines for hyperparameter selection, emphasizing the importance of truncation length, encoder window length, neural network architecture, and efficient optimization strategies.

Theoretical Analysis

The proposed method is theoretically analyzed to assess its consistency, smoothness, and data efficiency.

Consistency

The paper demonstrates that the SUBNET estimator is consistent, meaning that as the number of data points increases, the estimated model converges to an equivalent representation of the system that generated the data. This is crucial for ensuring statistical validity and unbiased model estimation.

Increased Cost Smoothness

The use of truncated prediction loss enhances the smoothness of the cost function, making the optimization process more stable and less prone to local minima. The Lipschitz constant of the cost function scales favorably with the truncation length, improving gradient stability.

Figure 2: Influence of the truncation length $T$ of the loss function on the test error during training.

Data Efficiency

The method is more data-efficient due to overlapping subsections. Overlapping increases the data coverage per estimation step, reducing variance and improving the reliability of the estimation process.

Simulation Study

The paper includes a thorough simulation study showcasing the effectiveness of the subspace encoder method. It compares the proposed method against classical ANN-based state-space identification techniques, demonstrating superior performance in terms of computational efficiency and accuracy.

Figure 3: Evolution of the NRMS simulation error of the estimated models by the considered approaches w.r.t. the test data. The keywords encoder init'' andparameter init'' indicate if either encoder-based prediction or parametric estimation is used to estimate the initial states, overlap'' andno-overlap'' indicate if the subsections can overlap, while ``OE'' stands for simulation based cost over the entire data sequence with no subsections.

Benchmark Results

The method achieves state-of-the-art results on the Wiener-Hammerstein benchmark, demonstrating its applicability and robustness in widely recognized nonlinear system identification challenges.

Conclusion

The paper introduces a powerful and efficient approach to nonlinear system identification using deep subspace encoders. It addresses practical challenges in noise handling and computational scalability, offering theoretical insights and empirical validation on complex benchmarks. The method represents a significant advancement in leveraging neural networks for modeling nonlinear systems, promising applications in engineering and beyond.