Alternating Low-Rank Updates

• Alternating low-rank updates are iterative algorithms that reconstruct low-rank matrices by alternately optimizing factor matrices using tractable subproblems.
• They integrate reweighted least-squares, structured projections, and block-coordinate updates to enhance robustness and convergence speed.
• These methods support applications like matrix completion, denoising, and distributed estimation, achieving performance near theoretical limits under noise and regularization.

Alternating low-rank updates refer to a broad class of iterative optimization algorithms that reconstruct or factorize matrices of fixed low rank by alternately solving tractable subproblems—typically least-squares, reweighted, or projected variants—on the factors, possibly subject to additional structure or regularization. These algorithms alternate between updates to left and right factor matrices, often within the context of matrix completion, sensing, regression, or adaptation settings. They generalize the classical Alternating Least Squares (ALS) paradigm and serve as the basis for many contemporary low-rank matrix and tensor reconstruction, completion, and estimation procedures.

1. Problem Formulation and Bilinear Factorization

The canonical alternating low-rank update framework seeks the reconstruction of an unknown rank-$r$ matrix $X\in\mathbb{C}^{n\times p}$ from noisy linear measurements $y = \mathcal{A}(X) + n \in \mathbb{C}^m$, where $m < np$ and the sensing operator $\mathcal{A}$ can be written as $\mathcal{A}(X)_i = \langle A_i, X \rangle$ or $y = A\,\mathrm{vec}(X) + n$ with $A\in\mathbb{C}^{m\times np}$ (Zachariah et al., 2012).

To impose the rank constraint, we use the bilinear factorization $X = U V^T$, $U\in\mathbb{C}^{n\times r}$, $V\in\mathbb{C}^{p\times r}$, and minimize the least-squares or generalized loss objective, for example,

$J(U, V) = \|\ y - A\,\mathrm{vec}(UV^T)\ \|^2,$

which is nonconvex in $(U,V)$ but convex in either factor holding the other fixed.

This formulation extends to structured matrices (e.g., Hankel, Toeplitz, PSD), matrix completion (observing entries on a subset $\Omega$), and regularized loss functions with column-sparsity or trace-norm penalties.

2. Iterative Alternating Updates and Variants

Alternating update algorithms operate by fixing one factor and performing an optimal update to the other. In the standard ALS setting, each subproblem is a linear least-squares solve:

• Fix $U$, update $V$: $$\mathrm{vec}(V^T) = [\,A (I_p \otimes U)\,]^\dagger\,y$$
• Fix $V$, update $U$: $$\mathrm{vec}(U) = [\,A (V \otimes I_n)\,]^\dagger\,y$$ (Zachariah et al., 2012)

Reweighted or regularized extensions incorporate promoted group-$\ell_2$ column sparsity through a joint penalty:

$$h(U,V) = \sum_{i=1}^d \sqrt{ \|u_i\|_2^2 + \|v_i\|_2^2 + \eta^2 },$$

with adaptive weights $w_i = (\|u_i\|^2 + \|v_i\|^2 + \eta^2)^{-1/2}$ updated at each iteration and Newton-type block coordinate updates (Giampouras et al., 2017).

Additional enhancements include:

• Block-diagonal Hessian approximations to enable efficient second-order steps (Newton or quasi-Newton).
• Structured projections after forming a bilinear iterate (e.g., project $L R^T$ onto a known subspace or PSD cone) (Zachariah et al., 2012, Li et al., 2014).
• Inner ADMM (Alternating Direction Method of Multipliers) subloops that solve quadratic minimization steps with directionality constraints to accelerate convergence and handle structure (Li et al., 2014).

3. Incorporation of Matrix Structure and Constraints

Alternating low-rank update algorithms can be generalized to handle structured matrices by embedding a "lift-and-project" at each iteration:

• For linear structure: $$\mathrm{vec}(\bar{X}) = S(S^\dagger \mathrm{vec}(L R^T))$$, projecting onto known subspaces, such as Hankel or Toeplitz (Zachariah et al., 2012).
• For positive semidefinite (PSD) constraints: symmetrize $M = (L R^T + (L R^T)^*)/2$, perform eigen-decomposition $M = V \Lambda V^*$, and retain the $r$ largest eigenpairs, $\bar{X} = V_r \Lambda_r V_r^*$.
• Rank-adaptive algorithms iteratively build the low-rank factorization by successive rank-1 updates and occasionally perform full or partial stage-$p$ solves to enhance accuracy (Lee et al., 2020).

Alternation can also integrate hard constraints such as entrywise nonnegativity or maximum norm, using alternating projections onto both non-convex constraint sets and the low-rank set—with guaranteed linear convergence under super-regularity and transversality (Budzinskiy, 2023).

4. Convergence, Optimality, and Cramér–Rao Bounds

The alternating low-rank update procedure yields a monotonic decrease in the cost function at every iteration; however, due to the inherent nonconvexity of bilinear factorizations, only local optimality can be guaranteed in general (Zachariah et al., 2012). Limit points are stationary for the block-coordinate descent, and ergodic convergence rates can reach $O(1/K)$ to stationarity (Hastie et al., 2014).

When side-information is available and exploited (e.g., matrix structure), the performance approaches the respective structured Cramér–Rao Bound (CRB): for unstructured rank-$r$,

$$\mathrm{CRB}(X) = \mathrm{tr}\{ (P^T\,A^T\,C^{-1}A\,P )^{-1} \},$$

where $P$ is the tangent basis to the rank-$r$ manifold. Structured CRBs (e.g., for Hankel parameterization or PSD matrices) generalize to appropriate tangent spaces with Moore–Penrose inverses for nonunique parametrizations (Zachariah et al., 2012).

Robustification via approximate least-squares solves and sketching maintains geometric convergence rates provided solve errors stay below theoretical tolerances (Gu et al., 2023). For matrix completion, alternating minimization can achieve sample complexity near the information-theoretic limit, provided the underlying matrix is sufficiently incoherent and the algorithm tolerates moderate update errors (Gu et al., 2023).

5. Computational Complexity, Scalability, and Distributed Variants

Each iteration of alternating low-rank algorithms is dominated by two LS solves of typical dimensions $m\times(nr)$ and $m\times(pr)$. For direct solvers, cost is $O(m (nr)^2 + (nr)^3)$; with structured $A$ or when $m \ll nr$, conjugate-gradient reduces per-step cost to $O(m n r)$, yielding $O(m n r + m p r)$ per overall iteration (Zachariah et al., 2012).

Rank-adaptive updates build the factorization one column at a time with linear cost in the ambient dimensions; enhanced full stage-$p$ solves can escalate to $(n_1 p)^\alpha$ per iteration for some $\alpha\in(1,2)$ depending on solver (Lee et al., 2020).

Alternating low-rank updates have been extended to fully distributed settings (e.g., in sensor networks or smart grids), where parameter transmission is bandwidth-limited. By alternating optimal updates in the local reduced-rank spaces, distributed algorithms like DRJIO-NLMS and DRJIO-RLS achieve convergence rates proportional to the reduced rank $D$ while minimizing communication cost compared to full-rank methods (Lamare, 2017).

Acceleration strategies such as warm-start, block orthogonalization, and strong initialization (e.g., SVD-based or random orthonormal factors) further improve practical scalability in large matrix completion and adaptation problems (Hastie et al., 2014).

6. Extensions and Applications Across Domains

Alternating low-rank update frameworks have found diverse application in large-scale matrix completion, denoising, nonnegative matrix factorization, adaptive distributed estimation, and continuous algebraic Riccati equation solvers.

• Matrix completion and SVD: Fast alternating least-squares enables scalable nuclear-norm minimization and regularized low-rank SVD, efficiently exploiting sparse+low-rank structure and distributed linear algebra (Hastie et al., 2014).
• Structured recovery (Hankel, PSD): Alternating updates with "lift-and-project" structure yield robust estimates, outperforming unconstrained methods at low measurement fractions, with performance gaps to CRB: $\sim2.7$dB (Hankel), $\sim0.8$dB (PSD) (Zachariah et al., 2012).
• Adaptive estimation: Alternating low-rank schemes in distributed networks minimize MSE with reduced communication, matching full-rank RLS performance at dramatically lower bandwidth (Lamare, 2017).
• Energy minimization for multi-term matrix equations: Successive rank-one update and refinement steps in alternating minimization enable high-fidelity low-rank solutions for discretized stochastic PDE systems (Lee et al., 2020).
• Chebyshev norm approximation: Accelerated alternating minimization achieves efficient low-rank matrix approximations in the max-norm, outperforming SVD in uniform error (Morozov et al., 7 Oct 2024).

7. Practical Insights, Limitations, and Empirical Performance

Extensive simulation studies across multiple setups reveal the following:

• Monotonic cost decrease and empirical robustness: Overall, alternating low-rank update algorithms guarantee non-increasing residuals (objective value) and converge to high-quality local minima (Zachariah et al., 2012, Giampouras et al., 2017, Hastie et al., 2014).
• Performance close to CRB: As signal-to-measurement-noise ratio (SMNR) or measurement fraction grows, unconstrained ALS approaches the CRB; exploiting structure delivers further improvements (Zachariah et al., 2012).
• Resilience under sub-sampling and noise: Structure-incorporating alternating updates remain robust even when unconstrained methods deteriorate due to rank condition failure at low sampling (Zachariah et al., 2012).
• Block-coordinate stationarity and sublinear rates: All limit points are near-stationary and block-residual decreases at sublinear rates (Giampouras et al., 2017).
• Efficient memory and computational footprint: By avoiding full matrix operations and leveraging sparsity and low-rank structure, alternating low-rank updates scale to extremely large dimensions and are amenable to parallelization and distributed implementation.

In summary, alternating low-rank update methodologies constitute a core toolbox for matrix and tensor reconstruction and optimization tasks with flexible incorporation of structure, regularization, and distributed computation. Their theoretical and empirical properties—monotonic descent, CRB-approaching performance, robustness to partial observation, and computational efficiency—make them fundamental in modern signal processing, statistical learning, and large-scale data analysis frameworks (Zachariah et al., 2012, Giampouras et al., 2017, Hastie et al., 2014, Li et al., 2014, Morozov et al., 7 Oct 2024, Lamare, 2017, Gu et al., 2023, Lee et al., 2020, Budzinskiy, 2023).