Randomized Subspace Nesterov Accelerated Gradient

Abstract: Randomized-subspace methods reduce the cost of first-order optimization by using only low-dimensional projected-gradient information, a feature that is attractive in forward-mode automatic differentiation and communication-limited settings. While Nesterov acceleration is well understood for full-gradient and coordinate-based methods, obtaining accelerated methods for general subspace sketches that use only projected-gradient information and can improve over full-dimensional Nesterov acceleration in oracle complexity is technically nontrivial. We develop randomized-subspace Nesterov accelerated gradient methods for smooth convex and smooth strongly convex optimization under matrix smoothness and generic sketch moment assumptions. The key technical ingredient is a three-sequence formulation tailored to matrix smoothness, which recovers the corresponding classical Nesterov methods in the full-dimensional case. The resulting theory establishes accelerated oracle-complexity guarantees and makes explicit how matrix smoothness and the sketch distribution enter the complexity. It also provides a unified basis for comparing sketch families and identifying when randomized-subspace acceleration improves over full-dimensional Nesterov acceleration in oracle complexity.

• The paper develops RS-NAG, an accelerated first-order method that integrates Nesterov acceleration with low-dimensional subspace projections.
• It provides explicit convergence and oracle-complexity bounds for both convex and strongly convex objectives, highlighting improvements over full-gradient methods.
• The work analyzes various sketching strategies (Haar, block-coordinate, Gaussian) to guide optimal subspace selection and reduce computational cost.

Randomized Subspace Nesterov Accelerated Gradient: Accelerated First-order Optimization with Subspace Oracle Complexity

Introduction and Context

The landscape of large-scale optimization is increasingly dominated by scenarios where either high-dimensionality or distributed computation imposes a bottleneck on the ability to use full-gradient information in each iteration. Randomized coordinate and block methods have mitigated some of the cost by focusing updates on smaller subspaces or blocks, but integrating the acceleration effects of Nesterov’s optimal first-order schemes into generic subspace methods has remained elusive, especially in regimes governed by distributed computation, forward-mode automatic differentiation (AD), or communication constraints.

The paper “Randomized Subspace Nesterov Accelerated Gradient” (2605.00740) addresses this gap by formulating, analyzing, and evaluating accelerated first-order methods that leverage only low-dimensional projected-gradient (subspace) information, generalizing both classical Nesterov acceleration and prior randomized subspace methods. The results provide oracle-complexity guarantees under matrix smoothness, characterize when subspace acceleration improves over classical full-gradient Nesterov acceleration, and offer a unified analysis of different sketching strategies, including Haar, coordinate, and Gaussian projections.

Methodological Advances

Oracle Model and Matrix Smoothness

The foundational premise is an oracle model where only directional derivatives can be queried. Projecting the gradient onto a random $r$-dimensional subspace incurs an oracle cost proportional to $r$ rather than the ambient dimension $d$. The setting assumes a matrix smoothness condition:

$$f(y)\le f(x)+\langle\nabla f(x), y-x\rangle + \frac{1}{2}(y-x)^\top L (y-x),$$

for a symmetric $L \succeq 0$, so that oracle complexity may explicitly depend on both the smoothness matrix and the sketch family.

Randomized Subspace Nesterov Accelerated Gradient (RS-NAG)

Classical Nesterov acceleration exploits two-sequence or estimate-sequence arguments for optimal convergence rates ($\mathcal{O}(1/N^2)$ for convex, $\mathcal{O}(\sqrt{L/\mu})$ for strongly convex). Porting these arguments to settings with random subspace projections has proven nontrivial, as the standard two-sequence template fails outside near-full-dimensional regimes.

To overcome this, the authors develop a three-sequence construction. RS-NAG maintains iterates $(x_k, y_k, z_k)$ as follows, using a random sketch $P_k \in \mathbb{R}^{d \times r}$:

For the convex case:

$$\begin{aligned} y_k &= \frac{A_k}{A_{k+1}}x_k + \frac{a_{k+1}}{A_{k+1}}z_k \ x_{k+1} &= y_k - \frac{1}{L\ell}P_k P_k^\top \nabla f(y_k) \ z_{k+1} &= z_k - a_{k+1} P_k P_k^\top \nabla f(y_k) \end{aligned}$$

with $r$0 defined by the matrix smoothness and sketch moment assumptions.

The strongly convex case introduces corresponding modifications to match standard accelerated convergence in the presence of strong convexity.

Crucially, with $r$1 and appropriate $r$2, these schemes exactly reduce to standard Nesterov (accelerated) gradient descent, ensuring theoretical and practical consistency.

Complexity and Convergence Results

The paper’s central theoretical contributions are explicit convergence and oracle-complexity bounds that account for matrix smoothness and the subspace sketch distribution:

Convex Case

Given $r$3, the RS-NAG-C method achieves:

$r$4

with oracle (directional derivative) complexity to $r$5-accuracy:

$r$6

Strongly Convex Case

Letting $r$7, the RS-NAG-SC method has:

$r$8

where $r$9.

Thus, the oracle complexity bound to $d$0-accuracy is:

$d$1

These bounds recover the classical accelerated rates for full-dimension ($d$2) but, importantly, can offer strictly better complexity for carefully chosen subspace distributions when $d$3.

Analysis of Sketching Strategies

The behavior of RS-NAG methods is dictated by the constants $d$4 and $d$5, which encapsulate the second-moment behavior and matrix–smoothness interaction for a given sketch. The three canonical sketches analyzed are:

• Haar projections: uniformly random $d$6-dimensional orthonormal subspaces
• Block-coordinate projections: uniformly sampled coordinate blocks of size $d$7
• Gaussian projections: i.i.d. Gaussian random matrices with normalized variance

Explicit expressions for $d$8, $d$9, and the sketch-dependent complexity factor $$f(y)\le f(x)+\langle\nabla f(x), y-x\rangle + \frac{1}{2}(y-x)^\top L (y-x),$$0 are derived (see Table below).

Sketch	$$f(y)\le f(x)+\langle\nabla f(x), y-x\rangle + \frac{1}{2}(y-x)^\top L (y-x),$$1	$$f(y)\le f(x)+\langle\nabla f(x), y-x\rangle + \frac{1}{2}(y-x)^\top L (y-x),$$2	$$f(y)\le f(x)+\langle\nabla f(x), y-x\rangle + \frac{1}{2}(y-x)^\top L (y-x),$$3
Haar	$$f(y)\le f(x)+\langle\nabla f(x), y-x\rangle + \frac{1}{2}(y-x)^\top L (y-x),$$4	$$f(y)\le f(x)+\langle\nabla f(x), y-x\rangle + \frac{1}{2}(y-x)^\top L (y-x),$$5	$$f(y)\le f(x)+\langle\nabla f(x), y-x\rangle + \frac{1}{2}(y-x)^\top L (y-x),$$6
Block-coord.	$$f(y)\le f(x)+\langle\nabla f(x), y-x\rangle + \frac{1}{2}(y-x)^\top L (y-x),$$7	$$f(y)\le f(x)+\langle\nabla f(x), y-x\rangle + \frac{1}{2}(y-x)^\top L (y-x),$$8	$$f(y)\le f(x)+\langle\nabla f(x), y-x\rangle + \frac{1}{2}(y-x)^\top L (y-x),$$9
Gaussian	$L \succeq 0$0	$L \succeq 0$1	$L \succeq 0$2

Here $L \succeq 0$3 and $L \succeq 0$4.

Through analysis and minimization, it is shown that the optimal sketch dimension is typically $L \succeq 0$5, and the improvement over the full-dimension ($L \succeq 0$6) scaling can be up to a factor of $L \succeq 0$7 in favorable regimes (e.g., for diagonally dominant $L \succeq 0$8 and small $L \succeq 0$9).

Numerical Experiments

Extensive experiments substantiate the theoretical perspective. The empirical study includes:

• Quadratic Problems: Four quadratic forms isolating effective rank and diagonal dominance confirm that block-coordinate and Haar-type sketches outperform full-gradient Nesterov acceleration in oracle cost, with the gap depending on $\mathcal{O}(1/N^2)$0 and $\mathcal{O}(1/N^2)$1.

Figure 1: Oracle-axis convergence on the four quadratic problems, demonstrating the superior oracle-efficiency of the RS-NAG methods over full-gradient accelerated schemes as predicted by theory.

• Logistic Regression: On real datasets, including colon-cancer, bioresponse, hiva_agnostic, and others, RS-NAG-SC methods with Haar and Gaussian sketches achieve superior or competitive convergence compared to both their full-gradient and non-accelerated subspace counterparts, with the empirical ordering closely following the predicted $\mathcal{O}(1/N^2)$2-constants for each dataset.

Figure 2: Oracle-axis convergence results for the colon-cancer dataset, highlighting the empirical match to complexity predictions among sketch families.

Figure 3: Convergence for convex diagonal/Haar settings, further illustrating subspace acceleration superiority when $\mathcal{O}(1/N^2)$3 is small.

Figure 4: Results for the phishing dataset, exhibiting strong performance of subspace-accelerated methods in varied data regimes.

Across all settings, optimality is achieved at the theoretically predicted $\mathcal{O}(1/N^2)$4, validating the oracle-based theoretical framework.

Theoretical and Practical Implications

• Acceleration in the Subspace Oracle Model: This work establishes, for the first time, that rigorous Nesterov acceleration is possible in generic subspace-projection regimes with explicit dependence on distributional sketch properties, not just for coordinate-aligned or full-gradient cases.
• When Can Subspace Methods Outperform Full-Gradient Nesterov?: The results make explicit when a subspace-accelerated method provides strictly lower oracle complexity than classical Nesterov, quantifying the improvement in terms of model-dependent quantities $\mathcal{O}(1/N^2)$5 and $\mathcal{O}(1/N^2)$6.
• Guidance for Sketching Choices: The $\mathcal{O}(1/N^2)$7-constants encapsulate the tradeoff between sketch type, problem structure, and oracle efficiency, enabling informed selection before deployment in high-dimensional or communication-limited settings.
• Broader Applicability: The model aligns naturally with modern computational modalities, including distributed optimization, federated learning, and forward-mode AD in deep learning, where full gradients are either too expensive or outright impractical to obtain.
• Generalizable Template for Accelerated Subspace Optimization: The three-sequence recursion used in RS-NAG opens the way for future generalizations, possibly employing sketch distributions tailored to problem structure (e.g., non-uniform or $\mathcal{O}(1/N^2)$8-aware sampling).

Future Directions

Principal avenues for development include:

• Custom Sketch Distributions: Inspired by non-uniform acceleration in coordinate methods, constructing distributions that exploit $\mathcal{O}(1/N^2)$9-matrix structure could provide sharper oracle bounds, especially in highly anisotropic settings.
• Non-smooth or Composite Objectives: Extension to proximal or composite settings would broaden applicability to a wider class of machine learning tasks.
• Adaptive and Momentum Variants: Investigating integration with adaptive stepsize procedures or momentum-rescaled update dynamics could further reduce empirical oracle cost.
• Federated, Decentralized, and Streaming Regimes: The methods’ compatibility with constrained-communication paradigms underpins relevance for federated and decentralized optimization in practical deployments.

Conclusion

This work closes a critical gap in first-order convex optimization by demonstrating that Nesterov-type acceleration is achievable in general randomized subspace settings. The use of a three-sequence method, thorough theoretical analysis under matrix smoothness, and comprehensive comparison of sketching strategies collectively chart a new frontier for accelerated large-scale optimization in both theory and practice. The ideas may serve as a basis for a new class of communication and memory-efficient learning algorithms, especially valuable in modern distributed and resource-constrained environments.