Rank-Adaptive HOOI: Efficient Tucker Decomposition

• Rank-adaptive HOOI is an iterative algorithm that computes the truncated Tucker decomposition using adaptive rank selection to meet a prescribed error tolerance.
• It employs mode-wise truncated SVD and a minimal rank selection procedure to achieve lower-rank representations without sacrificing accuracy.
• The approach guarantees local optimality and monotonic convergence, offering significant compression benefits in synthetic and real-world tensor applications.

Rank-adaptive higher-order orthogonal iteration (HOOI) is an iterative algorithm for computing the truncated Tucker decomposition of a tensor to within a prescribed error tolerance. It advances the classical HOOI approach by adaptively selecting mode-wise ranks, achieving minimal multilinear rank representation subject to accuracy constraints. This approach directly addresses longstanding inefficiencies associated with fixed-rank and non-orthogonal alternatives. Rank-adaptive HOOI is locally optimal within each iterative update and features monotonic convergence, providing strong theoretical guarantees and practical compression benefits across synthetic and real-world tensors (Xiao et al., 2021).

1. Problem Setup and Objective

Given an $N$th-order tensor

$$\mathcal{X} \in \mathbb{R}^{I_1 \times I_2 \times \cdots \times I_N}$$

and an error tolerance $0 < \varepsilon < 1$, the goal is the truncated Tucker decomposition: $$\mathcal{X} \approx \mathcal{G} \times_1 U^{(1)} \times_2 \cdots \times_N U^{(N)},$$ where each $U^{(n)} \in \mathbb{R}^{I_n \times R_n}$ is column-orthonormal and $$\mathcal{G} \in \mathbb{R}^{R_1 \times \cdots \times R_N}$$ is the core. The decomposition must satisfy: $$\| \mathcal{X} - \mathcal{G} \times_1 U^{(1)} \cdots \times_N U^{(N)} \|_F \leq \varepsilon \| \mathcal{X} \|_F.$$ Unlike standard HOOI, the multilinear ranks $R_n$ are adaptively determined during the iteration to be the smallest possible integers yielding the specified approximation accuracy (Xiao et al., 2021).

2. Rank-Adaptive HOOI Algorithm

The rank-adaptive HOOI algorithm integrates mode-wise orthogonal updates via truncated singular value decomposition (SVD) and a minimal rank selection procedure adhering to the prescribed error. The stepwise algorithm is as follows:

Pseudocode

1. Initialization:
   • Provide orthonormal $\{U_0^{(n)}\}$ and initial ranks $\{R_n^0\}$.
   • Compute initial core:

   $$\mathcal{X} \in \mathbb{R}^{I_1 \times I_2 \times \cdots \times I_N}$$0

• Set iteration $$\mathcal{X} \in \mathbb{R}^{I_1 \times I_2 \times \cdots \times I_N}$$1.

1. Iterative Update (repeat until $$\mathcal{X} \in \mathbb{R}^{I_1 \times I_2 \times \cdots \times I_N}$$2):

For $$\mathcal{X} \in \mathbb{R}^{I_1 \times I_2 \times \cdots \times I_N}$$3: - Form intermediate tensor

$$\mathcal{X} \in \mathbb{R}^{I_1 \times I_2 \times \cdots \times I_N}$$4

• Unfold $$\mathcal{X} \in \mathbb{R}^{I_1 \times I_2 \times \cdots \times I_N}$$5 into $$\mathcal{X} \in \mathbb{R}^{I_1 \times I_2 \times \cdots \times I_N}$$6 (mode-$$\mathcal{X} \in \mathbb{R}^{I_1 \times I_2 \times \cdots \times I_N}$$7 unfolding).
• Compute full SVD: $$\mathcal{X} \in \mathbb{R}^{I_1 \times I_2 \times \cdots \times I_N}$$8 with $$\mathcal{X} \in \mathbb{R}^{I_1 \times I_2 \times \cdots \times I_N}$$9.
• Determine minimal $0 < \varepsilon < 1$0 such that

  $0 < \varepsilon < 1$1

  Set $0 < \varepsilon < 1$2.

• Update $0 < \varepsilon < 1$3.

Set $0 < \varepsilon < 1$4.

By selecting the minimal $0 < \varepsilon < 1$5 per mode that maintains feasibility, the algorithm ensures locally optimal rank minimization at each step (Xiao et al., 2021).

3. Theoretical Guarantees: Local Optimality and Convergence

The procedure is supported by two central theorems:

• Local Optimality: For fixed factors except $0 < \varepsilon < 1$6, the update rule for $0 < \varepsilon < 1$7 provides the smallest rank such that the reconstructed error does not exceed $0 < \varepsilon < 1$8. The subproblem reduces to a best-rank approximation of $0 < \varepsilon < 1$9, where the truncation threshold is dictated by the tolerance constraint.
• Monotonicity: Each sequence of mode-wise ranks $$\mathcal{X} \approx \mathcal{G} \times_1 U^{(1)} \times_2 \cdots \times_N U^{(N)},$$0 is non-increasing and eventually stabilizes. At each update, a larger rank could always be retained without exceeding the error bound, but the algorithm explicitly selects the smallest feasible rank.

These results are direct consequences of the orthogonality-enforced SVD subproblem and orthogonal invariance of the Frobenius norm in HOOI (Xiao et al., 2021).

4. Computational Complexity

Let $$\mathcal{X} \approx \mathcal{G} \times_1 U^{(1)} \times_2 \cdots \times_N U^{(N)},$$1. For each mode-$$\mathcal{X} \approx \mathcal{G} \times_1 U^{(1)} \times_2 \cdots \times_N U^{(N)},$$2 update:

• Matricized product: $$\mathcal{X} \approx \mathcal{G} \times_1 U^{(1)} \times_2 \cdots \times_N U^{(N)},$$3, $$\mathcal{X} \approx \mathcal{G} \times_1 U^{(1)} \times_2 \cdots \times_N U^{(N)},$$4
• Full SVD: $$\mathcal{X} \approx \mathcal{G} \times_1 U^{(1)} \times_2 \cdots \times_N U^{(N)},$$5 for rank-$$\mathcal{X} \approx \mathcal{G} \times_1 U^{(1)} \times_2 \cdots \times_N U^{(N)},$$6 truncation

Summed over all modes, per-iteration cost is

$$\mathcal{X} \approx \mathcal{G} \times_1 U^{(1)} \times_2 \cdots \times_N U^{(N)},$$7

For $$\mathcal{X} \approx \mathcal{G} \times_1 U^{(1)} \times_2 \cdots \times_N U^{(N)},$$8, $$\mathcal{X} \approx \mathcal{G} \times_1 U^{(1)} \times_2 \cdots \times_N U^{(N)},$$9: $U^{(n)} \in \mathbb{R}^{I_n \times R_n}$0 Compared to classical HOOI, this approach requires a full SVD rather than (potentially less expensive) fixed-rank truncation, but manages lower effective ranks in practice (Xiao et al., 2021).

5. Comparative Numerical Results

Experiments highlight the advantages of rank-adaptive HOOI over fixed-rank and greedy strategies:

Problem Type	Fixed-Rank Methods	Greedy HOSVD	Rank-Adaptive HOOI
Synthetic $U^{(n)} \in \mathbb{R}^{I_n \times R_n}$1 tensor	$U^{(n)} \in \mathbb{R}^{I_n \times R_n}$2	$U^{(n)} \in \mathbb{R}^{I_n \times R_n}$3	$U^{(n)} \in \mathbb{R}^{I_n \times R_n}$4
Coulomb kernel, $U^{(n)} \in \mathbb{R}^{I_n \times R_n}$5, $U^{(n)} \in \mathbb{R}^{I_n \times R_n}$6	up to $U^{(n)} \in \mathbb{R}^{I_n \times R_n}$7 more parameters	$U^{(n)} \in \mathbb{R}^{I_n \times R_n}$8 more parameters	minimal parameter count
MNIST $U^{(n)} \in \mathbb{R}^{I_n \times R_n}$9	compression $$\mathcal{G} \in \mathbb{R}^{R_1 \times \cdots \times R_N}$$0	compression $$\mathcal{G} \in \mathbb{R}^{R_1 \times \cdots \times R_N}$$1	compression $$\mathcal{G} \in \mathbb{R}^{R_1 \times \cdots \times R_N}$$2

• On synthetic tensors with added noise, rank-adaptive HOOI exactly recovers the true rank and achieves an order of magnitude lower reconstruction error than fixed-rank strategies.
• For the regularized Coulomb kernel, rank-adaptive HOOI matches error thresholds with up to $$\mathcal{G} \in \mathbb{R}^{R_1 \times \cdots \times R_N}$$3 fewer parameters than $$\mathcal{G} \in \mathbb{R}^{R_1 \times \cdots \times R_N}$$4-HOSVD and $$\mathcal{G} \in \mathbb{R}^{R_1 \times \cdots \times R_N}$$5–$$\mathcal{G} \in \mathbb{R}^{R_1 \times \cdots \times R_N}$$6 improvements over greedy approaches, with comparable computation time—only at the tightest tolerances does SVD computation dominate.
• On MNIST digit tensors, rank-adaptive HOOI obtains much higher compression (up to $$\mathcal{G} \in \mathbb{R}^{R_1 \times \cdots \times R_N}$$7) at comparable classification accuracy ($$\mathcal{G} \in \mathbb{R}^{R_1 \times \cdots \times R_N}$$8–$$\mathcal{G} \in \mathbb{R}^{R_1 \times \cdots \times R_N}$$9), with testing time reduced from tens of seconds to $$\| \mathcal{X} - \mathcal{G} \times_1 U^{(1)} \cdots \times_N U^{(N)} \|_F \leq \varepsilon \| \mathcal{X} \|_F.$$0s (Xiao et al., 2021).

6. Rank Adaptivity in Context

Rationale for Rank Adaptivity: Fixed-rank Tucker decompositions require a priori specification of ranks $$\| \mathcal{X} - \mathcal{G} \times_1 U^{(1)} \cdots \times_N U^{(N)} \|_F \leq \varepsilon \| \mathcal{X} \|_F.$$1, typically set conservatively large to avoid exceeding error tolerance, resulting in redundancy and decreased computational efficiency. Greedy or uniform $$\| \mathcal{X} - \mathcal{G} \times_1 U^{(1)} \cdots \times_N U^{(N)} \|_F \leq \varepsilon \| \mathcal{X} \|_F.$$2-HOSVD variants improve upon this but still overestimate ranks. Rank-adaptive HOOI automatically determines minimal feasible ranks mode-by-mode, avoiding unnecessary storage and computation without compromising accuracy.

Distinction from Fixed-Rank HOOI and ALS: Fixed-rank HOOI maintains constant $$\| \mathcal{X} - \mathcal{G} \times_1 U^{(1)} \cdots \times_N U^{(N)} \|_F \leq \varepsilon \| \mathcal{X} \|_F.$$3, precluding rank minimization even if a lower rank suffices. Classical alternating least squares (ALS) solves unconstrained least-squares problems per mode, lacking an explicit rank constraint and mechanism for elimination of insignificant singular vectors. By enforcing orthonormal updates and truncated SVD solutions, HOOI is functionally a modified ALS (MALS) that uniquely admits natural rank adaptivity (Xiao et al., 2021).

The algorithm guarantees per-iteration feasibility ($$\| \mathcal{X} - \mathcal{G} \times_1 U^{(1)} \cdots \times_N U^{(N)} \|_F \leq \varepsilon \| \mathcal{X} \|_F.$$4), achieves local optimality in rank selection, converges as ranks are monotonically nonincreasing, and yields efficient representations that accelerate downstream tasks.

7. Practical Implications and Significance

Rank-adaptive HOOI provides an automated, accuracy-driven mechanism for compact multilinear tensor approximation. It combines the structural advantages of orthogonal tensor decompositions with dynamic rank minimization, resulting in storage-efficient and computationally competitive solutions. Empirical evidence corroborates the superiority of this approach over existing fixed-rank and greedy schemes in both synthetic and real-world application domains, such as scientific data compression and image classification tasks (Xiao et al., 2021).