ALS for Low-Rank Matrix Reconstruction

• Alternating Least Squares (ALS) is an iterative algorithm used to solve low-rank matrix and tensor reconstruction problems by alternating optimizations of variable subsets.
• It leverages quadratic, separable subproblems through factorization into L and R, achieving efficient convergence especially under Gaussian noise conditions.
• ALS adapts to structured constraints like Hankel and PSD projections, demonstrating near-optimal empirical performance relative to Cramér–Rao bounds in undersampled settings.

Alternating Least Squares (ALS) is a fundamental iterative method for solving low-rank matrix and tensor reconstruction problems from linear or structured observations under constraints such as low-rank, subspace membership, or semidefiniteness. Its core utility arises from efficiently alternating between optimizing subsets of variables, leveraging the separability of the least-squares objective in the chosen parametrization. ALS plays a central role in a wide range of applications, including compressed sensing, matrix completion, and multidimensional data analysis.

1. Optimization Framework and Objective Formulation

ALS addresses the recovery of an unknown $n \times p$ matrix $X$ (or higher-dimensional arrays) of rank $r \ll \min(n, p)$ from underdetermined, noisy linear measurements,

$$y = \mathcal{A}(X) + n, \quad y \in \mathbb{C}^m, \quad m < np,$$

where $$\mathcal{A}: \mathbb{C}^{n \times p} \to \mathbb{C}^m$$ is a known sensing operator and $n \sim \mathcal{N}(0, C)$. Equivalently, $y = A\,\mathrm{vec}(X) + n$ for some $A \in \mathbb{C}^{m \times np}$.

Under Gaussian noise, the maximum-likelihood estimator is

$$\hat{X} = \arg\min_{X: \operatorname{rank}(X) = r} \| C^{-1/2}[y-\mathcal{A}(X)] \|_2^2,$$

and for white noise ($C = \sigma^2 I$) this reduces to a least-squares cost

$X$0

Enforcing the rank constraint via a factorization $X$1, $X$2, $X$3, yields an objective

$X$4

This structure is especially advantageous for low-rank problems, as each subproblem is quadratic in $X$5 or $X$6 individually.

2. Alternating Minimization Algorithm

ALS exploits the fact that $X$7 is quadratic and separable in $X$8 and $X$9:

• Update $r \ll \min(n, p)$0 with $r \ll \min(n, p)$1 fixed:

$r \ll \min(n, p)$2

i.e., $r \ll \min(n, p)$3.

• Update $r \ll \min(n, p)$4 with $r \ll \min(n, p)$5 fixed:

$r \ll \min(n, p)$6

i.e., $r \ll \min(n, p)$7.

Here, $r \ll \min(n, p)$8 denotes the Moore–Penrose pseudoinverse. The linear systems involved are of manageable size when $r \ll \min(n, p)$9 is small, making ALS computationally efficient for moderate $$y = \mathcal{A}(X) + n, \quad y \in \mathbb{C}^m, \quad m < np,$$0, $$y = \mathcal{A}(X) + n, \quad y \in \mathbb{C}^m, \quad m < np,$$1, and $$y = \mathcal{A}(X) + n, \quad y \in \mathbb{C}^m, \quad m < np,$$2 (Zachariah et al., 2012).

Algorithmic Implementation (Pseudocode)

Initialization:

• Form $$y = \mathcal{A}(X) + n, \quad y \in \mathbb{C}^m, \quad m < np,$$3,
• Compute the top-$$y = \mathcal{A}(X) + n, \quad y \in \mathbb{C}^m, \quad m < np,$$4 SVD: $$y = \mathcal{A}(X) + n, \quad y \in \mathbb{C}^m, \quad m < np,$$5,
• Set $$y = \mathcal{A}(X) + n, \quad y \in \mathbb{C}^m, \quad m < np,$$6.

Main loop:

• Repeat until $$y = \mathcal{A}(X) + n, \quad y \in \mathbb{C}^m, \quad m < np,$$7 no longer decreases:
  1. $$y = \mathcal{A}(X) + n, \quad y \in \mathbb{C}^m, \quad m < np,$$8
  2. (If structure: $$y = \mathcal{A}(X) + n, \quad y \in \mathbb{C}^m, \quad m < np,$$9, $$\mathcal{A}: \mathbb{C}^{n \times p} \to \mathbb{C}^m$$0, $$\mathcal{A}: \mathbb{C}^{n \times p} \to \mathbb{C}^m$$1)
  3. $$\mathcal{A}: \mathbb{C}^{n \times p} \to \mathbb{C}^m$$2
  4. (If structure: $$\mathcal{A}: \mathbb{C}^{n \times p} \to \mathbb{C}^m$$3, $$\mathcal{A}: \mathbb{C}^{n \times p} \to \mathbb{C}^m$$4, $$\mathcal{A}: \mathbb{C}^{n \times p} \to \mathbb{C}^m$$5)

• Output $$\mathcal{A}: \mathbb{C}^{n \times p} \to \mathbb{C}^m$$6.

3. Incorporation of Structural Constraints

ALS is adaptable to situations where $$\mathcal{A}: \mathbb{C}^{n \times p} \to \mathbb{C}^m$$7 is known a priori to belong to a linear subspace $$\mathcal{A}: \mathbb{C}^{n \times p} \to \mathbb{C}^m$$8 (e.g., Hankel, Toeplitz) or to the positive semidefinite cone $$\mathcal{A}: \mathbb{C}^{n \times p} \to \mathbb{C}^m$$9.

• Linear Structure: After forming $n \sim \mathcal{N}(0, C)$0, project onto the subspace parameterized by $n \sim \mathcal{N}(0, C)$1:

$n \sim \mathcal{N}(0, C)$2

Re-solve for $n \sim \mathcal{N}(0, C)$3 (or $n \sim \mathcal{N}(0, C)$4) to minimize $n \sim \mathcal{N}(0, C)$5 via the closed-form update $n \sim \mathcal{N}(0, C)$6 (or $n \sim \mathcal{N}(0, C)$7).

• Positive Semidefiniteness: Symmetrize and project using spectral decomposition,

$n \sim \mathcal{N}(0, C)$8

with $n \sim \mathcal{N}(0, C)$9, $y = A\,\mathrm{vec}(X) + n$0 the eigenvectors and top $y = A\,\mathrm{vec}(X) + n$1 eigenvalues. Update $y = A\,\mathrm{vec}(X) + n$2, $y = A\,\mathrm{vec}(X) + n$3 (as above) onto $y = A\,\mathrm{vec}(X) + n$4 (Zachariah et al., 2012).

Structural steps add projection costs—$y = A\,\mathrm{vec}(X) + n$5 for linear subspaces, $y = A\,\mathrm{vec}(X) + n$6 for PSD projections—while preserving computational feasibility for moderate scale.

4. Convergence Properties and Computational Complexity

Each ALS iteration monotonically nonincreases the objective $y = A\,\mathrm{vec}(X) + n$7, ensuring convergence to a stationary point (which can be a local minimum rather than the global optimum). The per-iteration cost is dominated by solving two linear least-squares problems of size $y = A\,\mathrm{vec}(X) + n$8 and $y = A\,\mathrm{vec}(X) + n$9:

• Pseudoinverse computation for $A \in \mathbb{C}^{m \times np}$0: $A \in \mathbb{C}^{m \times np}$1,
• Pseudoinverse for $A \in \mathbb{C}^{m \times np}$2: $A \in \mathbb{C}^{m \times np}$3,

and, if needed, $A \in \mathbb{C}^{m \times np}$4 for subspace projection, or $A \in \mathbb{C}^{m \times np}$5 for PSD projection (Zachariah et al., 2012).

ALS thus remains tractable for moderate rank $A \in \mathbb{C}^{m \times np}$6 and moderate $A \in \mathbb{C}^{m \times np}$7.

5. Empirical Performance Relative to Cramér–Rao Bounds

Extensive simulations for $A \in \mathbb{C}^{m \times np}$8, true rank $A \in \mathbb{C}^{m \times np}$9, and varying sample complexity $$\hat{X} = \arg\min_{X: \operatorname{rank}(X) = r} \| C^{-1/2}[y-\mathcal{A}(X)] \|_2^2,$$0 with $$\hat{X} = \arg\min_{X: \operatorname{rank}(X) = r} \| C^{-1/2}[y-\mathcal{A}(X)] \|_2^2,$$1, demonstrate:

• Unstructured ALS achieves performance within $$\hat{X} = \arg\min_{X: \operatorname{rank}(X) = r} \| C^{-1/2}[y-\mathcal{A}(X)] \|_2^2,$$2 dB of the unstructured Cramér–Rao bound (CRB) across signal-to-noise ratios (SMNRs).
• Structured (Hankel, PSD) ALS yields significant gains, e.g., $$\hat{X} = \arg\min_{X: \operatorname{rank}(X) = r} \| C^{-1/2}[y-\mathcal{A}(X)] \|_2^2,$$3 dB (Hankel) and $$\hat{X} = \arg\min_{X: \operatorname{rank}(X) = r} \| C^{-1/2}[y-\mathcal{A}(X)] \|_2^2,$$4 dB (PSD) performance improvements at SMNR $$\hat{X} = \arg\min_{X: \operatorname{rank}(X) = r} \| C^{-1/2}[y-\mathcal{A}(X)] \|_2^2,$$5 dB.
• Sample complexity: For $$\hat{X} = \arg\min_{X: \operatorname{rank}(X) = r} \| C^{-1/2}[y-\mathcal{A}(X)] \|_2^2,$$6, ALS approaches the CRB; for $$\hat{X} = \arg\min_{X: \operatorname{rank}(X) = r} \| C^{-1/2}[y-\mathcal{A}(X)] \|_2^2,$$7, the unstructured CRB itself becomes loose.

This confirms that ALS is effective in general undersampled matrix reconstruction, with a-priori structure markedly narrowing the gap to the corresponding CRB (Zachariah et al., 2012).

6. Extensions and Practical Considerations

• Initialization: SVD-based spectral methods provide effective initializations that accelerate convergence.
• Applicability: The ALS paradigm generalizes directly to tensor decompositions (CP, Tucker, etc.) and can be adapted to handle more general nonconvex regularization or missing-data formulations.
• Limitation: Convergence is only to a stationary point; global optimality is not guaranteed except in specific cases (e.g. certain rank-one tensor approximations or highly overdetermined settings).
• Structural constraints: The insertion of projection steps allows ALS to exploit domain knowledge (e.g., harmonic or positive semidefinite structure) without compromising the alternating minimization efficiency.

7. Summary Table: ALS Methodology for Low-Rank Matrix Recovery

Aspect	Description	Complexity
Objective	$$\hat{X} = \arg\min_{X: \operatorname{rank}(X) = r} \| C^{-1/2}[y-\mathcal{A}(X)] \|_2^2,$$8 (white noise, rank constraint)	$$\hat{X} = \arg\min_{X: \operatorname{rank}(X) = r} \| C^{-1/2}[y-\mathcal{A}(X)] \|_2^2,$$9 (matrix size)
Parametrization	$C = \sigma^2 I$0, $C = \sigma^2 I$1, $C = \sigma^2 I$2	$C = \sigma^2 I$3 (parameter dims)
Update for $C = \sigma^2 I$4	$C = \sigma^2 I$5	$C = \sigma^2 I$6
Update for $C = \sigma^2 I$7	$C = \sigma^2 I$8	$C = \sigma^2 I$9
Structure projection (linear)	Project $X$00 onto $X$01, recompute $X$02, $X$03 by minimizing Frobenius norm error (closed-form updates)	$X$04
Structure projection (PSD)	Project $X$05 onto the leading-$X$06 eigenspace (symmetrization, eigen-decomposition), update $X$07, $X$08 by least-squares onto projected $X$09	$X$10

ALS thus provides a robust, extensible, and computationally efficient framework for low-rank matrix estimation in both unstructured and structured scenarios, with predictable convergence properties and favorable empirical performance against fundamental information-theoretic limits (Zachariah et al., 2012).