Tensorized Multi-modal Regression

• Tensorized multi-modal regression is a statistical framework that fuses structured data from multiple modalities into high-order tensors to capture complex interactions.
• It leverages low-rank tensor decompositions like CP, Tucker, and Tensor-Train to reduce parameter complexity while maintaining model accuracy.
• Advanced optimization methods, including ALS, gradient descent, and Bayesian strategies, enable scalable and interpretable solutions across diverse applications.

Tensorized multi-modal regression is a class of statistical learning models designed to fuse structured data from multiple modalities—such as neuroimaging, text, audio, video, and spatiotemporal sensors—through high-order tensor representations and structured low-rank regularization. By treating both predictors and/or responses as multiway arrays (tensors), these approaches efficiently capture complex intra- and inter-modal interactions and dependencies. The field encompasses numerous modeling frameworks, including CP, Tucker, Tensor-Train (TT) decompositions, graph-regularized constructions, and both frequentist and Bayesian estimation paradigms—each balancing expressive power with parameter parsimony and computational tractability (Liu et al., 2023, Wang et al., 2022, Xu et al., 2021, Barezi et al., 2018, Liu et al., 2023, Hu et al., 2019, Lock, 2017).

1. Tensorized Multi-modal Regression: Mathematical Formulation

Multi-modal regression problems often entail learning predictive mappings from a set of structured modalities $X_i^{(m)}$ to scalar, vector, or tensor-valued targets $y_i$. Commonly, each data instance is represented as a collection of order-$(L_m)$ tensors (one per modality), which are combined into a joint $(M+1)$-order tensor $X_i$ with modes indexing features and modalities (Xu et al., 2021). The most general tensorized regression models take the form

$Y = \langle X, B \rangle + E,$

where $Y$ and $X$ are response and predictor tensors (potentially of arbitrary order), $\langle X, B \rangle$ denotes a multi-mode contraction, $B$ is a high-order coefficient tensor parameterizing the regression mapping, and $y_i$0 represents noise (Lock, 2017, Wang et al., 2022).

Variants include:

• Multi-modal block-tensor structure: Multiple predictors $y_i$1, each mapped by modality-specific or shared coefficient tensors $y_i$2 (Liu et al., 2023).
• Joint fusion tensorization: Formation of an $y_i$3-way weight tensor $y_i$4 acting on the outer product or multi-way concatenation of all modalities, modeled via CP, Tucker, or TT forms (Barezi et al., 2018, Xu et al., 2021).

The high dimensionality of coefficient tensors makes direct estimation intractable, motivating the extensive use of structured low-rank tensor decompositions.

2. Low-Rank Tensor Decomposition Architectures

To achieve parameter efficiency and prevent overfitting, tensorized regression frameworks impose low-rank structure on $y_i$5 (or $y_i$6) via:

• CP (CANDECOMP/PARAFAC) decomposition: Expresses $y_i$7 as a sum of rank-1 tensors. For an $y_i$8-way $y_i$9, $(L_m)$0 (Lock, 2017, Liu et al., 2023).
• Tucker decomposition: Factorizes $(L_m)$1 as $(L_m)$2 for a core tensor $(L_m)$3 and factor matrices $(L_m)$4 (Wang et al., 2022, Barezi et al., 2018).
• Tensor-Train decomposition: Approximates $(L_m)$5 (or $(L_m)$6) using a chain of small core tensors, yielding linear scaling in mode dimension and exponential compression for high-order arrays. TT decomposition is central in the Multi-Graph Tensor Network (MGTN) for multi-modal regression with graph structure (Xu et al., 2021).

These low-rank parameterizations drastically reduce the effective number of free parameters:

• CP: $(L_m)$7 for CP-rank $(L_m)$8 and mode sizes $(L_m)$9.
• Tucker: $(M+1)$0 for rank tuple $(M+1)$1.
• TT: $(M+1)$2 for TT ranks $(M+1)$3 and mode sizes $(M+1)$4.

3. Optimization Algorithms and Estimation Schemes

All major frameworks employ block coordinate descent (alternating minimization) or gradient-based updates for parameter estimation (Liu et al., 2023, Barezi et al., 2018, Liu et al., 2023):

• Alternating Least Squares (ALS): Each set of factor matrices ($(M+1)$5, $(M+1)$6) or core is updated with all others fixed, solving a regularized linear or ridge regression subproblem at each step (Lock, 2017, Liu et al., 2023, Barezi et al., 2018).
• Gradient Descent/Autodiff: When embedded in neural network architectures (e.g., MRRF (Barezi et al., 2018)), automatic differentiation enables joint optimization of all factors via standard SGD/Adam.
• Gibbs Sampling and Bayesian MCMC: Bayesian tensor-on-tensor regression uses conjugate priors to enable blockwise Gibbs sampling and Metropolis-Hastings updates for joint rank and parameter learning (Wang et al., 2022).
• Simulated Annealing (SA): Ultra-fast non-MCMC estimation selects Tucker ranks via BIC-minimization and simulated-annealing search, then maximizes posterior probability over factor matrices (Wang et al., 2022).
• ADMM and Proximal Methods: Convex formulations (e.g., with nuclear-norm or trace-norm regularization) admit optimization via splitting algorithms (Liu et al., 2023).

Several frameworks (e.g., tLSSVM-MTL (Liu et al., 2023)) exploit the structure of the weight tensor to solve all subproblems as linear systems, boosting tractability.

4. Regularization, Model Selection, and Theoretical Guarantees

Regularization is essential for identifiability and generalization:

• Low-rank constraints: Controlled by CP, Tucker, or TT rank hyperparameters.
• Graph-smoothness penalties: Incorporated via Laplacian regularization on TT cores or factors (as in MGTN (Xu et al., 2021)), ensuring estimated parameters vary smoothly over graph-structured domains.
• Redundancy-reduction via modality-specific ranks: MRRF (Barezi et al., 2018) implements explicit low-mode-rank parameterization, removing modality-specific redundancies recoverable by other modalities.
• Elementwise or group sparsity, total variation, orthogonality constraints, and non-negative factors: These are optionally incorporated for additional structural parsimony (Liu et al., 2023, Hu et al., 2019, Lock, 2017).

Model selection uses cross-validation, BIC, or Bayesian model evidence for rank determination (Wang et al., 2022, Hu et al., 2019).

Representative theoretical results include:

Guarantee	Approach	Reference
Statistical consistency, minimax optimality	ALS, convex relax	(Hu et al., 2019)
Posterior contraction rates	Bayes CP/Tucker	(Wang et al., 2022)
ALS convergence	CP/Tucker ALS	(Lock, 2017)
Identifiability (CP, Tucker, TT)	Kruskal, orthogon	(Liu et al., 2023)

5. Empirical Applications and Performance Results

Tensorized multi-modal regression methods have demonstrated efficacy in a wide range of domains:

• Neuroimaging: Tensor-based regression with multi-modal predictors (e.g., fMRI, diffusion MRI, behavioral, genetic data) for association mapping and connectivity analysis (Niyogi et al., 2023, Hu et al., 2019, Liu et al., 2023).
• Spatio-temporal forecasting: MGTN yields state-of-the-art accuracy for climate and air-quality prediction, outperforming non-tensor approaches in parameter efficiency and RMSE by large margins (e.g., fMGTN achieves TRMSE/TEMSE = 0.0206/0.0186 with 1,894 parameters vs GRU = 0.0237/0.0226 with 36,740 parameters) (Xu et al., 2021).
• Sentiment analysis and emotion recognition: MRRF offers 1–4% absolute error improvements (over prior fusion baselines) on sentiment, personality, and emotion benchmarks while elucidating cross-modal redundancy and contribution (Barezi et al., 2018).
• Multitask learning on multi-indexed regression problems: tLSSVM-MTL achieves lowest RMSE and highest $(M+1)$7 predictions on restaurant-consumer, student performance, and comprehensive climate forecasting, surpassing matrix-based and convex tensor benchmarks (Liu et al., 2023).
• Structured response regression: Bayesian tensor-on-tensor regression attains lower relative prediction errors (RPE) and better empirical coverage on image/motion data compared to CP regression (e.g., RPE 0.375 with 1,154 parameters vs 0.477 with 3,840 parameters on LFW) (Wang et al., 2022).

6. Practical Considerations, Limitations, and Software

State-of-the-art tensorized multi-modal regression models are supported by efficient software libraries in Python (TensorLy, tntorch, scikit-tensor), MATLAB, and R (Liu et al., 2023). Key considerations include:

• Scalability: ALS, block coordinate, and TT-based formulations enable tractable learning in high dimensions. For very large tensors, randomized SVD, sketching, and online updates are necessary.
• Interpretability: Low-rank factors and modality-specific decompositions enable interrogation of modality contributions, feature importance, and cross-mode interactions (as in redundancy-reduction MRRF (Barezi et al., 2018)).
• Uncertainty quantification: Fully Bayesian approaches yield credible intervals and model-averaged predictions (Wang et al., 2022).
• Limitations: Combinatorial mode growth, nonconvexity (risk of local optima in CP/Tucker rank selection), prior/hyperparameter tuning sensitivity, and computational overhead for large-scale Bayesian inference remain open challenges (Wang et al., 2022, Liu et al., 2023).

7. Outlook and Research Directions

Tensorized multi-modal regression continues to evolve rapidly, with sustained theoretical and methodological advances:

• Unified frameworks for tensor-on-tensor and graph-structured data: Integration of non-Euclidean structures (graphs, manifolds) directly into tensorized predictors and regularization is increasingly common (Xu et al., 2021).
• Automatic rank selection and uncertainty quantification: Bayesian and BIC-based selection algorithms are prominent for balancing expressivity and overfitting (Wang et al., 2022, Hu et al., 2019).
• Deep learning integration: Embedding Tucker/CP/TT factorized tensor regression layers in neural architectures to fuse deep representations is a promising direction (Barezi et al., 2018, Liu et al., 2023).
• Application domains: Ongoing extensions include adaptive multi-modal fusion in medical imaging, spatiotemporal modeling, recommender systems, and chemometrics (Liu et al., 2023, Hu et al., 2019).

Tensorized multi-modal regression thus serves as a foundational paradigm for high-dimensional, data-rich modeling across modern scientific, engineering, and social applications.