Learning Efficient Tensor Representations with Ring Structure Networks

Abstract: Tensor train (TT) decomposition is a powerful representation for high-order tensors, which has been successfully applied to various machine learning tasks in recent years. However, since the tensor product is not commutative, permutation of data dimensions makes solutions and TT-ranks of TT decomposition inconsistent. To alleviate this problem, we propose a permutation symmetric network structure by employing circular multilinear products over a sequence of low-order core tensors. This network structure can be graphically interpreted as a cyclic interconnection of tensors, and thus we call it tensor ring (TR) representation. We develop several efficient algorithms to learn TR representation with adaptive TR-ranks by employing low-rank approximations. Furthermore, mathematical properties are investigated, which enables us to perform basic operations in a computationally efficiently way by using TR representations. Experimental results on synthetic signals and real-world datasets demonstrate that the proposed TR network is more expressive and consistently informative than existing TT networks.

• The paper introduces tensor ring decomposition, which overcomes limitations of tensor train models by using a ring structure that eliminates strict boundary rank constraints.
• The proposed methods, TR-SVD and Block-Wise ALS, efficiently compute tensor operations and adapt ranks for scalable high-dimensional data processing.
• Experimental results on datasets like COIL-100 and KTH video demonstrate that the TR model achieves superior accuracy, flexibility, and robustness in representing complex data patterns.

Learning Efficient Tensor Representations with Ring Structure Networks

Introduction

The paper "Learning Efficient Tensor Representations with Ring Structure Networks" introduces a novel approach to tensor decomposition, specifically focusing on high-order tensors commonly used in machine learning and signal processing. Tensor train (TT) decomposition has been widely used in these fields due to its ability to mitigate the curse of dimensionality. However, existing TT models exhibit limitations such as sensitivity to data permutation and rigid rank constraints. This paper proposes the tensor ring (TR) decomposition, an alternative that aims to enhance flexibility and representation power.

Tensor Ring Decomposition

TR decomposition builds upon the TT model by introducing a ring structure that connects the first and last core tensors, creating a circular interconnection. This structure alleviates the non-commutative nature of tensor products, thus addressing permutation sensitivity. Importantly, TR decomposition does not require the strict constraint $r_1=r_{d+1}=1$, enhancing the model's representational capacity.

Mathematically, a $d$-dimensional tensor is decomposed into a sequence of 3rd-order core tensors. Each element of the tensor is approximated by a circular trace operation over these core tensors. This design allows TR models to maintain permutation symmetry, ensuring consistent solutions regardless of tensor dimension order—an advantage over traditional TT models.

Algorithms for TR Representation

Two core algorithms are developed for efficient learning of TR representations: TR-SVD and Block-Wise Alternating Least Squares (ALS).

• TR-SVD: A non-iterative algorithm utilizing sequential singular value decompositions to construct TR representations. It is designed to be computationally efficient and scalable, suitable for large datasets with high-order tensors.
• Block-Wise ALS: An iterative algorithm enhancing the adaptation of TR-ranks by updating two adjacent cores simultaneously. This method leverages block optimization, employing truncated SVD to achieve the best approximation with rank adaptation.

Both algorithms efficiently perform tensor operations such as addition, multilinear products, and inner products directly on TR representations, thus supporting scalable processing of high-dimensional data.

Properties and Theoretical Insights

The paper investigates several mathematical properties of TR representation, demonstrating how operations like addition, Hadamard product, and inner product can be efficiently computed using the TR structure. These operations are crucial for practical applications in complex machine learning tasks. The theoretical framework ensures that compact and scalable tensor representation is achievable, providing robust tools for high-dimensional dataset processing.

Experimental Results

Empirical evaluations demonstrate the TR model's superior performance and robustness compared to traditional TT models. Experiments on synthetic data and real-world datasets, including the COIL-100 and KTH video datasets, showcase TR's higher expressiveness and accuracy in representing complex patterns, along with its efficient scalability due to reduced parameter requirements.

Conclusion

The introduction of tensor ring decomposition marks a significant step towards more adaptive and powerful tensor representations in machine learning. By alleviating the limitations of TT decomposition through a novel ring structure, TR models provide enhanced flexibility and computational efficiency. The paper's proposed algorithms and theoretical insights lay the foundation for future advancements in scalable high-order tensor computations, promising broader applications in AI and data processing.