Tensor Ring Decomposition

Abstract: Tensor networks have in recent years emerged as the powerful tools for solving the large-scale optimization problems. One of the most popular tensor network is tensor train (TT) decomposition that acts as the building blocks for the complicated tensor networks. However, the TT decomposition highly depends on permutations of tensor dimensions, due to its strictly sequential multilinear products over latent cores, which leads to difficulties in finding the optimal TT representation. In this paper, we introduce a fundamental tensor decomposition model to represent a large dimensional tensor by a circular multilinear products over a sequence of low dimensional cores, which can be graphically interpreted as a cyclic interconnection of 3rd-order tensors, and thus termed as tensor ring (TR) decomposition. The key advantage of TR model is the circular dimensional permutation invariance which is gained by employing the trace operation and treating the latent cores equivalently. TR model can be viewed as a linear combination of TT decompositions, thus obtaining the powerful and generalized representation abilities. For optimization of latent cores, we present four different algorithms based on the sequential SVDs, ALS scheme, and block-wise ALS techniques. Furthermore, the mathematical properties of TR model are investigated, which shows that the basic multilinear algebra can be performed efficiently by using TR representaions and the classical tensor decompositions can be conveniently transformed into the TR representation. Finally, the experiments on both synthetic signals and real-world datasets were conducted to evaluate the performance of different algorithms.

• The paper presents a flexible tensor decomposition method that represents high-dimensional data as a circular multilinear product, avoiding permutation dependency.
• It details four algorithms (TR-SVD, TR-ALS, TR-ALSAR, TR-BALS) for efficiently optimizing latent cores and dynamically adjusting TR-ranks.
• Experimental results on datasets such as COIL-100 and KTH videos show superior compression ability and classification accuracy compared to traditional methods.

Tensor Ring Decomposition

Introduction

The "Tensor Ring Decomposition" (1606.05535) introduces a novel approach to tensor decomposition, addressing the limitations of existing methods like Tensor Train (TT) decomposition. This paper presents the Tensor Ring (TR) decomposition as a more flexible and generalized model that can effectively handle high-dimensional tensor data by mitigating the dependence on permutations of tensor dimensions, a notable issue with TT decomposition.

The TR decomposition represents a tensor as a circular multilinear product over a sequence of lower-dimensional cores, offering circular dimensional permutation invariance. This innovative approach retains the representation power of TT decompositions while providing enhanced flexibility and efficiency.

Tensor Ring Model

The TR decomposition model represents a $d$th-order tensor with a circular sequence of 3rd-order tensors (cores). A key advantage of this model is its invariance to circular permutations of tensor dimensions, achieved by employing the trace operation and treating the latent cores equivalently. This property distinguishes TR decomposition from TT decomposition, allowing for more stable representations regardless of dimension order.

Mathematically, TR decomposition is expressed in several equivalent forms, providing robust and varied methods for implementing this model in practical applications. Additionally, by relaxing the constraints on TT-ranks, TR decomposition achieves increased representation ability, demonstrating its potential to handle complex tensor structures.

Algorithms for Tensor Ring Decomposition

The paper provides four algorithms to optimize the latent cores for TR decomposition:

1. TR-SVD Algorithm: This sequential algorithm uses SVD to perform decomposition, ensuring an efficient and stable non-recursive process.
2. TR-ALS Algorithm: Utilizes the Alternating Least Squares (ALS) method for precise optimization. It requires predefined TR-ranks, allowing direct comparison with baseline models.
3. TR-ALSAR Algorithm: An adaptive version of the ALS algorithm, it adjusts TR-ranks dynamically during optimization, suitable for datasets with unknown optimal ranks.
4. TR-BALS Algorithm: A block-wise ALS algorithm that employs truncated SVD for efficient rank adaptation, minimizing computational costs while maintaining accuracy.

These algorithms collectively offer flexibility, efficiency, and precision, making them suitable for various tensor decomposition tasks across different domains.

Properties and Mathematical Framework

The TR model allows efficient execution of fundamental tensor operations such as addition, multiplication, Hadamard product, and inner product directly on the decomposed core tensors. This capability significantly reduces computational complexity, making TR decomposition a powerful tool for processing large-scale tensor data.

The representation power of TR decomposition surpasses traditional models by allowing transformation from other decompositions such as CP and Tucker into the TR format while retaining essential properties like the ability to reduce dimensionality effectively.

Relation to Other Models

The TR decomposition model can be viewed as a generalization of TT decomposition, where TT is a special case with constraints on ranks. TR ranks are smaller due to their capability to represent the tensor as a linear combination of TT decompositions, leading to a more flexible representation model.

Additionally, the paper discusses how TR decomposition relates to and can transform classical tensor models such as CP and Tucker decompositions, making TR a versatile framework for tensor analysis.

Experimental Results

Extensive experiments on synthetic and real-world datasets, such as COIL-100 and KTH video dataset, demonstrate the effectiveness of TR decomposition. The results indicate that TR decomposition achieves competitive or superior results compared to TT and CP models, especially in terms of compression ability and classification accuracy.

Notably, TR decomposition retains discriminative features effectively even with higher approximation errors, emphasizing its potential for robust unsupervised feature extraction.

Conclusion

The "Tensor Ring Decomposition" paper presents a significant advancement in tensor analysis by introducing the TR model, which overcomes the limitations of traditional decompositions such as TT and CP. With its enhanced flexibility, efficiency, and representation power, TR decomposition has broad implications for handling complex, high-dimensional data in various scientific and engineering domains. The proposed algorithms and model properties provide a solid foundation for future research and practical applications in machine learning, signal processing, and beyond.