--- title: Data-Driven Finite-Dimensional Predictors url: https://www.emergentmind.com/topics/data-driven-construction-of-finite-dimensional-linear-predictors type: topic --- # Data-Driven Finite-Dimensional Predictors Data-driven construction of finite-dimensional linear predictors refers to a suite of methodologies that leverage observed data—typically in the form of state, input, and output trajectories of a dynamical or stochastic system—to synthesize linear models that optimally predict system evolution or outputs. These predictors are "finite-dimensional" in that they act on reduced (finite-dimensional) representations of potentially complex or nonlinear systems, enabling deployment of linear estimation and control tools even in highly nonlinear or infinite-dimensional contexts. The landscape includes operator-theoretic approaches such as lifting via Koopman operator theory, set membership and ARX paradigms, structured realization via interpolation of transfer function data, and low-rank regularized regression for systems with combinatorial interaction effects. ## 1. Koopman Operator and Lifted Linear Predictors Data-driven lifted linear predictors, prominently based on Koopman operator theory, address nonlinear controlled discrete-time systems of the form $x_{k+1} = f(x_k, u_k)$. The key concept is to construct a finite set of lifting functions or "observables" $\psi_1,\ldots,\psi_N$ mapping the state space $\mathbb R^n$ into a higher-dimensional space $\mathbb R^N$. Defining the lifted mapping $\psi(x) = [\psi_1(x), \ldots, \psi_N(x)]^\top$, one seeks a linear predictor $$ z_{k+1} = A z_k + B u_k, \qquad \hat y_k = C z_k, \qquad z_0 = \psi(x_0), $$ where $(A, B, C)$ are identified via least squares in the lifted space. The extended dynamic mode decomposition (EDMD) algorithm, or variants thereof, are used to find $A, B$ to minimize $$ \|Y_\text{lift} - A X_\text{lift} - B U\|_F^2, $$ with $X_\text{lift}$ and $Y_\text{lift}$ the collections of lifted current and next states, and $U$ the input matrix. Output mapping $C$ is solved similarly, often trivially if the identity coordinates are included in $\psi$ [1611.03537], [2408.01116]. Finite-dimensional approximation leverages the linearity of the (infinite-dimensional) Koopman operator, whose action on observables translates, after lifting, to a high-dimensional linear system that approximates the nonlinear flow. The accuracy over finite prediction horizons improves with richness of the lifting dictionary and abundance/diversity of training data, with cross-validation over held-out validation sets used to tune the dictionary (N), regularization, and other hyperparameters [2408.01116]. ## 2. Construction of Predictors: Algorithms and Methodologies ### EDMD-Based Lifting and Regression The canonical workflow involves: 1. **Dictionary Selection:** Choosing lifting functions tailored to the nonlinearities of interest—polynomial, trigonometric, or radial basis functions, as well as problem-informed custom features. 2. **Data Collection:** Recording tuples $(x_i,u_i, x_i^+)$, where $x_i^+ = f(x_i, u_i)$, over the relevant regions of state-input space. 3. **Lifted Regression:** Forming $X_\text{lift}$ and $Y_\text{lift}$, then solving for $(A,B)$ (and optionally $C$) via least squares or regularized regression. 4. **Validation:** Evaluating prediction errors with physically meaningful metrics, notably projected state error $\varepsilon_\text{proj}$ and multi-step prediction error $\varepsilon_\text{pred}$, which are more indicative for control purposes than lifted-state error. 5. **Iteration:** Updating dictionary size, regularization, or data coverage as dictated by validation metrics [2408.01116]. ### Set Membership and Multi-Step ARX Predictors An alternative paradigm is to fit independent $p$-step-ahead ARX predictors using set membership identification. Here, each predictor is characterized by a feasible parameter set (FPS) defined by polyhedral constraints induced by measurement noise bounds and observed regressor/output pairs. Online or offline (for reduced computation) algorithms intersect these parameter sets for prediction, yielding a filtered output and minimal worst-case error bound [2008.11994]. ### Structured Realization via Interpolatory Model Reduction For LTI and structured systems (including delay or second-order), data-driven structured realization leverages measured transfer function data at interpolation points. Generalized Loewner matrix pencils or Hermite-augmented systems enforce the desired dynamical structure, solving for the state-space realization through direct or least-squares methods. This framework accommodates prescribed structure via basis functions $h_k(s)$ in the transfer function and can exactly match interpolation data [1611.02072]. ### Low-Rank Latent Variable Linear Predictors In settings dominated by high-dimensional feature interactions, as in large-scale regression or factorization machine contexts, the LIT-LVM methodology introduces structured regularization for the interaction coefficient matrix $\Theta$ by parameterizing it as $UU^\top$ (latent variable model with $r \ll d$), reducing the number of free parameters from $O(d^2)$ to $O(rd)$. Optimization proceeds via block coordinate or proximal gradient descent, jointly minimizing prediction error and low-rank/sparsity penalties on $w$ and $U$ [2506.15492]. ## 3. Practical Implementation and Computational Aspects Key considerations in implementation of data-driven finite-dimensional linear predictors include: - **Numerical Linear Algebra:** Solving least squares via QR or SVD is recommended for stability (especially as $N$ increases). For massive $N$ or sample size $K$, kernel-based EDMD or randomized sketching circumvents explicit lifting [1611.03537], [2408.01116]. - **Regularization:** Tikhonov (ridge) or sparsity-inducing penalties address overfitting and ill-conditioning inherent in high-dimensional regression. - **Sample Complexity:** The effectiveness of the lifted predictor depends crucially on the ratio $K \gg N$ and the diversity (persistence of excitation) of sampling trajectories, particularly in control-relevant regions [1611.03537]. - **Scaling and Truncation:** In functional linear processes, the dimension $d$ of the retained basis determines the bias-variance tradeoff, with convergence rates optimized by balancing truncation and estimation error [1607.05874]. - **Offline/Online Tradeoffs:** Methods such as global set membership filtering shift complexity offline by precomputing conservative bounds at the expense of minimally looser error bounds, whereas local approaches yield tighter bounds through online optimization [2008.11994]. - **Interpretability:** For LIT-LVM and similar factors models, the learned low-dimensional embeddings provide interpretable structure in interaction effects [2506.15492]. ## 4. Applications in Control and Estimation A principal motivation for constructing such predictors is their direct utility in synthesis of linear controllers—most notably Model Predictive Control (MPC) and Linear Quadratic Regulation (LQR): - **Koopman-MPC:** Substituting the true nonlinear dynamics in MPC optimization with the finite-dimensional linear predictor yields a convex quadratic program whose computational burden scales with the input and prediction horizon, not the lift dimension. Linear and even nonlinear state/input constraints can be embedded via augmentation of the lifting dictionary [1611.03537]. - **Stochastic MPC and Chance Constraints:** Identified multi-step predictors with quantified parameter uncertainty can be incorporated into robust or stochastic MPC formulations, using ellipsoidal or tail bounds for constraint tightening [2409.10405]. - **Real-Time Feasibility and Closed-Loop Guarantees:** Across a range of low- and high-dimensional systems (including high-dimensional PDEs), lifted linear predictors have delivered order-of-magnitude reductions in prediction error and control computation time compared with local linearization or Carleman approximations [1611.03537], [2408.01116]. - **Functional Time Series:** The finite-dimensional innovations approach enables prediction of infinite-dimensional processes (e.g., curves or spatial-temporal fields) with explicit rates and error guarantees [1607.05874]. ## 5. Empirical Results, Theoretical Guarantees, and Performance Empirical benchmarks consistently demonstrate that data-driven finite-dimensional predictors, especially those exploiting operator-theoretic lifting or structured low-rank regularization, surpass conventional predictors (local linearization, Carleman) both in long-horizon accuracy and control performance. - **Koopman vs. Local/Carleman Linearization:** On canonical benchmarks (Van der Pol oscillator, bilinear DC motor, KdV PDE), the lifted "Koopman-MPC" approach reduced long-range RMSE by orders of magnitude and enabled feasible, reliable MPC tracking under constraints where other linearizations failed [1611.03537]. - **LIT-LVM in High-Dimensional Regression:** LIT-LVM maintained stable, low prediction error as the interaction-to-sample ratio $d^2/n$ increased, where unstructured elastic net devolved into severe overfitting [2506.15492]. - **Set Membership Multistep Predictors:** The set-membership ARX filtering approach provided the tightest certified worst-case prediction bounds among all possible estimators of the form proposed, outperforming classical Kalman filter-based schemes, especially under bounded measurement noise [2008.11994]. - **Convergence and Generalization Guarantees:** Statistical theory attests to consistency and generalization rates, e.g., EDMD predictors on Koopman-invariant subspaces approach vanishing error as the number of samples and dictionary size increase, subject to ergodicity and identifiability conditions [1611.03537], [2408.01116]. ## 6. Model Selection, Hyperparameters, and Best Practices Effective model construction depends on disciplined choices of hyperparameters and evaluation strategies: - **Dictionary Selection:** Tailoring the set of lifting functions to the known structure of the dynamics or the control objective often outperforms generic large bases. Heuristic rules suggest starting with the identity and key nonlinear terms, expanding until validation errors plateau [2408.01116]. - **Hyperparameter Tuning:** Grid or random search over lifting dimension $N$, regularization strengths, or latent dimension $r$ (in LIT-LVM) is standard, guided by validation on physically meaningful prediction errors rather than lifted-state loss [2506.15492], [2408.01116]. - **Outlier and Noise Handling:** Robust regression or outlier discarding before pseudoinversion improves conditioning, while normalization of observables prior to regression is essential for numerically stable fitting [2408.01116]. - **Interpretability and Diagnostics:** Inspection of identified matrices (eigenvalues, block structure) can reveal unphysical couplings or instability, motivating model refinement prior to downstream usage [2408.01116]. ## 7. Representative Algorithms and Model Classes | Approach | Model Class / Structure | Sample/Parameter Complexity | |-------------------------|-------------------------------------|----------------------------------------------------| | Koopman-EDMD | Nonlinear → lifted linear (MPC/LQR) | O(N × K), N = lift dim, K = samples | | Set Membership ARX | LTI/ARX with bounded noise | O(P̄ × N), P̄ = # predictors, N = samples | | Structured Realization | LTI/delays/second-order | O(n^2 K), n = realization order, K = basis size | | LIT-LVM | High-dim. regression w/ interactions| O(n d^2 + n r d), d = features, r = latent dim | | Functional Innovations | Hilbert-space time series | O(n^3) (eigendecomp), O(d^3 m) (recursion) | Optimally, the method is chosen to match system characteristics, data regime, and target application. --- The data-driven construction of finite-dimensional linear predictors has matured into a technically rigorous, computationally practical discipline, bridging nonlinear system identification, operator-theoretic analysis, robust control, and high-dimensional statistics. It enables principled reduction of complex dynamical phenomena into tractable linear surrogates, fostering advances in predictive control, estimation, and interpretable machine learning over wide-ranging domains [1611.03537], [2408.01116], [2506.15492], [2008.11994], [1611.02072], [1607.05874]. Source: https://www.emergentmind.com/topics/data-driven-construction-of-finite-dimensional-linear-predictors