Factor Augmented Tensor-on-Tensor Neural Networks (2405.19610v2)

Abstract: This paper studies the prediction task of tensor-on-tensor regression in which both covariates and responses are multi-dimensional arrays (a.k.a., tensors) across time with arbitrary tensor order and data dimension. Existing methods either focused on linear models without accounting for possibly nonlinear relationships between covariates and responses, or directly employed black-box deep learning algorithms that failed to utilize the inherent tensor structure. In this work, we propose a Factor Augmented Tensor-on-Tensor Neural Network (FATTNN) that integrates tensor factor models into deep neural networks. We begin with summarizing and extracting useful predictive information (represented by the ``factor tensor'') from the complex structured tensor covariates, and then proceed with the prediction task using the estimated factor tensor as input of a temporal convolutional neural network. The proposed methods effectively handle nonlinearity between complex data structures, and improve over traditional statistical models and conventional deep learning approaches in both prediction accuracy and computational cost. By leveraging tensor factor models, our proposed methods exploit the underlying latent factor structure to enhance the prediction, and in the meantime, drastically reduce the data dimensionality that speeds up the computation. The empirical performances of our proposed methods are demonstrated via simulation studies and real-world applications to three public datasets. Numerical results show that our proposed algorithms achieve substantial increases in prediction accuracy and significant reductions in computational time compared to benchmark methods.

• The paper introduces Factor Augmented Tensor-on-Tensor Neural Networks, significantly improving prediction accuracy by capturing tensor structures and nonlinear dependencies.
• The paper employs a two-step methodology integrating low-rank tensor factorization with temporal convolutional networks to extract essential features from multi-dimensional data.
• The paper demonstrates substantial MSE reductions and computational speedups across simulations and real-world datasets, proving its practical utility in predictive modeling.

Factor Augmented Tensor-on-Tensor Neural Networks

The paper entitled "Factor Augmented Tensor-on-Tensor Neural Networks" presents an advanced methodology to tackle the tensor-on-tensor regression problem, where both covariates and responses take the form of multi-dimensional arrays or tensors. This approach aims to enhance prediction tasks by effectively capturing nonlinear relationships while leveraging the intrinsic tensor structures in the data. The proposed solution integrates tensor factor models into neural networks and is shown to improve both prediction accuracy and computational efficiency.

Introduction and Background

Predictive modeling of tensors has increased in prevalence due to its applicability across various fields such as finance, meteorology, and neuroscience. Traditional methods primarily focused on tensor regression, where either the covariates or responses are tensors, leaving the tensor-on-tensor regression tasks less explored. Early approaches that have addressed tensor-on-tensor regression either employed linear models, which lacked the capacity to capture nonlinear dependencies, or used black-box deep learning algorithms which disregarded the internal structure of tensors. This paper addresses these limitations by proposing a robust framework that retains tensor structures while modeling complex dependencies between tensor covariates and responses.

Methodology

The Factor Augmented Tensor-on-Tensor Neural Network (FATTNN) encompasses two critical components: tensor factorization and temporal convolutional networks.

1. Tensor Factorization: The approach decomposes the covariate tensors into low-dimensional factor tensors and corresponding loading matrices. This reduction in dimensionality is accomplished without losing the spatial and temporal information inherent in the tensor structures. The tensor factor model captures essential features that contain predictive information, which are then used as inputs to the neural network.
2. Temporal Convolutional Network (TCN): The factor tensors derived from the previous step serve as inputs to a TCN designed to capture temporal dependencies in the data. This combination allows the model to efficiently handle nonlinearity and temporal dynamics. The training process involves fitting the TCN to the sequences of factor tensors and corresponding response tensors.

The methodological framework includes detailed processes for low-rank tensor factorization and the integration of these factors into the neural network architecture. The authors provide thorough mathematical formulations and theoretical guarantees for the tensor factorization step, ensuring the accuracy of the estimated factors.

Empirical Performance

The efficacy of the proposed FATTNN model is demonstrated through extensive simulation studies and real-world applications. Key findings from the numerical experiments are as follows:

• Simulation Studies: The simulations showcase the superiority of FATTNN in reducing mean squared error (MSE) compared to benchmark methods, including traditional multiway regression and standalone TCN. FATTNN achieves substantial improvements in predictive accuracy due to its ability to effectively capture and utilize tensor structures and nonlinear dependencies.
• Real-World Applications: The paper evaluates FATTNN on diverse datasets, namely the FAO agricultural data, NYC taxi trip data, and FMRI brain imaging data. Across these datasets, FATTNN consistently delivers lower prediction errors and reduced computational times. For instance, the model delivered a 46.92% and 33.45% reduction in MSE and significant computational speedups in agricultural predictions, thus illustrating its practical utility.

Discussion and Future Work

The integration of tensor factor models with TCN within FATTNN offers several advantages:

• Preservation of Tensor Structure: Unlike methods that flatten tensors, FATTNN preserves the essential spatial and temporal relationships within the data.
• Modeling Nonlinearity: The neural network component excels in capturing complex nonlinear associations between covariates and responses.
• Computational Efficiency: Factorization reduces dimensionality, leading to faster computations without sacrificing accuracy.

The findings hold significant implications for a wide range of applications in industrial and scientific domains, suggesting that FATTNN can be a valuable tool for time series forecasting and other predictive tasks involving tensor data.

Future research directions could explore the extension of FATTNN to different types of neural network architectures, tailoring factor models to specific dataset characteristics, and further enhancing the scalability of the approach. Additionally, more in-depth theoretical analyses could be conducted to explore the asymptotic properties and robustness of the proposed method under various data distributions and noise conditions.

Conclusion

The paper presents a comprehensive and effective approach to tensor-on-tensor regression by combining tensor factor models with deep learning techniques. The proposed FATTNN method demonstrates superior predictive accuracy and computational advantages, addressing key limitations of existing methods. The empirical results validate its applicability and efficiency across varied real-world datasets, heralding a notable contribution to the field of multidimensional data analysis and predictive modeling.