---
title: Random Subspace Cubic-Regularization (R-ARC)
url: https://www.emergentmind.com/topics/random-subspace-cubic-regularization-r-arc
type: topic
---

# Random Subspace Cubic-Regularization (R-ARC)

Random Subspace Cubic-Regularization (R-ARC) refers to a family of adaptive second-order optimization algorithms that apply the principle of cubic regularization, but restrict each major iteration to a random low-dimensional subspace of the parameter space. Originally motivated by the prohibitive cost of full-space Hessian evaluation and subproblem solving in high dimensions, R-ARC techniques exploit randomization and/or sub-sampling to preserve the theoretical and empirical efficiency of Adaptive Regularization by Cubics (ARC) while reducing per-iteration computation. Recent research demonstrates that under appropriate sampling and embedding schemes, R-ARC achieves global convergence rates and second-order optimality guarantees equivalent to full-dimensional ARC, with substantial speedup—especially for low-rank or structurally sparse objectives [1802.05426][2501.03718][2501.09734][2406.16666][2002.09526].

## 1. Mathematical Framework and Subspace Cubic Model

R-ARC methods address unconstrained optimization problems
\[
\min_{x\in\mathbb{R}^d} f(x),
\]
where \(f\) is smooth (typically \(C^2\)), and possibly of finite-sum structure (\(f(x) = \frac{1}{n}\sum_{i=1}^n f_i(x)\)). At each iteration \(x_k\), the standard (full-space) cubic model is
\[
m_k(s) = f(x_k) + \nabla f(x_k)^\top s + \frac{1}{2} s^\top \nabla^2 f(x_k) s + \frac{\sigma_k}{6}\|s\|^3,
\]
with regularization parameter \(\sigma_k\). R-ARC constructs and minimizes an analogous cubic model on a random subspace or based on a random sub-sample:
- **Random subspace:** A sketching matrix \(S_k \in \mathbb{R}^{r\times d}\) or orthonormal matrix \(U_k \in \mathbb{R}^{d\times r}\) defines a subspace \(\mathcal{U}_k = \text{span}(U_k)\).
- **Gradient and Hessian projections:** Compute \(\tilde{g}_k = U_k^\top \nabla f(x_k)\), \(\tilde{H}_k = U_k^\top \nabla^2 f(x_k) U_k\).
- **Subspace cubic model:** 
  \[
  m_k^r(z) = f(x_k) + \tilde{g}_k^\top z + \frac{1}{2} z^\top \tilde{H}_k z + \frac{\sigma_k}{6}\|z\|^3, \quad z \in \mathbb{R}^r.
  \]
- **Step and acceptance:** \(s_k = U_k z_k\), where \(z_k\) is the (approximate) minimizer of \(m_k^r\). Step acceptance uses a ratio of actual to predicted reduction, \( \rho_k \).

Alternatively, for finite-sum problems, the Hessian itself may be subsampled: for \(S_k \subset \{1,\ldots,n\}\), compute
\[
\tilde{H}_k = \frac{1}{|S_k|} \sum_{i\in S_k} \nabla^2 f_i(x_k)
\]
or use importance sampling proportional to curvature [1802.05426].

## 2. Hessian Approximation and Subspace Selection Strategies

R-ARC encompasses several strategies for subspace or Hessian reduction:
- **Uniform coordinate subspace sampling:** Sample a random block or coordinate subset to define \(S_k\). The projected gradient and Hessian are then restricted to these coordinates.
- **Random Gaussian or oblivious subspace embeddings:** Use dense random projections (e.g., scaled Gaussian) to guarantee embedding properties, especially when seeking dimension-independence or low-rank adaptivity [2501.03718][2501.09734].
- **Finite-sum subsampling:** For sum-structured \(f\), directly subsample Hessian terms (uniformly or by curvature importance) [1802.05426].

Probabilistic embedding properties (Oblivious Subspace Embedding, OSE) guarantee that subspace projections preserve relevant curvature, provided the subspace dimension is sufficient (typically \(\mathcal{O}(r+1)\) for target rank \(r\)).

Subspace size (or sample size) can be fixed or made adaptive. Adaptive selection involves monitoring the observed rank or error in the subspace model (e.g., updating \(r_k\) if negative curvature directions are found).

## 3. Algorithmic Structure and Pseudocode

A prototypical R-ARC algorithm is as follows [2501.09734][2406.16666]:

```
Input: x₀, σ₀, θ, subspace size r or adaptive rule, other algorithmic constants.

For k = 0, 1, 2, ...
    1. Draw random subspace U_k ∈ ℝ^{d×r}
    2. Compute g^r_k = U_k^T ∇f(x_k), H^r_k = U_k^T ∇^2f(x_k) U_k
    3. Approximately minimize
         m^r_k(z) = f(x_k) + (g^r_k)^T z + ½ z^T H^r_k z + (σ_k/6) ||z||^3
       to obtain z_k; set s_k = U_k z_k
    4. Set ρ_k = (f(x_k) - f(x_k + s_k)) / (m^r_k(0) - m^r_k(z_k))
    5. If ρ_k ≥ θ
           x_{k+1} = x_k + s_k; decrease σ_k
       else
           x_{k+1} = x_k; increase σ_k
    6. For adaptive variants: update subspace size r_{k+1} based on subspace Hessian rank or specified rule.
```

Minimum requirements for the subproblem solution are standard: model decrease and gradient norm in the subspace below specified thresholds.

For finite-sum settings, a two-phase algorithm is employed in [1802.05426]: Phase I uses subsampled adaptive cubic regularization (SSAS) for a moderate-accuracy solution, Phase II applies a Nesterov-style acceleration (ASAS) for optimal complexity.

## 4. Convergence Theory and Global Complexity

R-ARC achieves convergence rates that interpolate between those of coordinate descent and full Newton/ARC, depending on the subspace size and sampling [2501.09734][2406.16666][2002.09526][1802.05426]:

- **First-order complexity:** 
  - With random subspaces of dimension \(r\), matching OSE conditions, R-ARC finds \(\|\nabla f(x)\| \leq \varepsilon\) in \(\mathcal{O}(\varepsilon^{-3/2})\) iterations, replicating full ARC rates while per-iteration cost is reduced to \(\mathcal{O}(r)\) gradient/Hessian-vector products and \(\mathcal{O}(r^3)\) linear algebra [2501.09734][2501.03718].
  - For random coordinate or minibatch selection of size \(m\) in a \(d\)-dimension problem, R-ARC interpolates: O\((d/m)\) coordinate descent rate for first-order methods, O\((\varepsilon^{-2})\) for CD, up to O\((\varepsilon^{-3/2})\) as \(m\rightarrow d\) (full cubic regularization) [2406.16666][2002.09526].
- **Second-order complexity:** To guarantee \(\lambda_{\min}(\nabla^2 f(x)) \geq -\varepsilon\), the iteration count is \(\mathcal{O}(\varepsilon^{-3})\), mirroring ARC [2501.09734].
- **High-probability and worst-case bounds:** In finite-sum or inexact Hessian settings (e.g., uniform or importance sampling), global iteration complexity is O\((\varepsilon^{-1/3})\) (high-probability) and O\((\varepsilon^{-5/6}\log\varepsilon^{-1})\) in the worst case for accelerated variants with inexact Hessians [1802.05426].

## 5. Adaptivity, Low-Rank Structure, and Scalability

R-ARC is particularly effective for functions with low-rank structure:
- **Low-rank objectives:** If \(f(x) = h(Ax)\) for \(A \in \mathbb{R}^{r^*\times d}\), so that the Hessian is always rank at most \(r^*\), then adaptive subspace variants (e.g. R-ARC-D) increase the subspace dimension only as needed, eventually matching the true rank plus one [2501.03718][2501.09734].
- **Computational cost:** For rank-\(r\) problems, per-iteration cost drops from \(\mathcal{O}(d^2)\) or \(\mathcal{O}(d^3)\) (full ARC) to \(\mathcal{O}(rd)\) for gradient/Hessian projections and \(\mathcal{O}(r^3)\) subproblem solves. R-ARC-D discovers the intrinsic rank online, requiring neither prior knowledge nor manual tuning.
- **Oblivious subspace embeddings:** Use of OSEs (e.g., Gaussian sketches) ensures that key step-size and decrease properties required for ARC convergence analysis translate to the projected domain.

Empirical results show clear advantages for R-ARC in scenarios with low effective rank, strong ill-conditioning, or a prohibitive Hessian cost [2501.03718][2501.09734].

## 6. Accelerated and Stochastic Variants

- **Nesterov-style acceleration:** For finite-sum objectives, phase II acceleration via mirror-descent-like auxiliary sequences achieves an O\((\varepsilon^{-1/3})\) rate (high probability) and clear acceleration over non-accelerated cubic regularization [1802.05426].
- **Stochastic subspace and coordinate implementations:** The stochastic subspace cubic Newton (SSCN) method directly applies R-ARC principles, building cubic models on randomly sampled minibatches of coordinates or dimensions, with convergence rates interpolating between first-order and full second-order methods [2002.09526][2406.16666].
- **Batch size and regularization parameter selection:** Empirically and theoretically, single-digit percent subspace sizes (2–5% of the coordinates) often optimize wall-clock time and trade-off between per-iteration cost and progress. The regularization parameter can be selected in an adaptive fashion to ensure sufficient model decrease and step acceptance.

## 7. Practical Implementation and Empirical Performance

- **Hessian-vector products:** The core computational cost is dominated by \(r\)–dimensional projected-gradient and Hessian-vector products, which are scalable for automatic differentiation and Hessian-free approaches.
- **Subproblem solvers:** The subspace cubic subproblem can be solved exactly (e.g., via Cholesky for small \(r\)), or inexactly via gradient-based or Krylov subspace methods (Lanczos).
- **Subspace/sample size heuristics:** Cap the subspace dimension between small and moderate fractions of \(d\) (e.g., 0.01\(d\)–0.2\(d\)) for best trade-off, and increase adaptively for low-rank detection. For finite-sum methods, sample complexities scale with the desired gradient accuracy \(ε_k\) as \(\mathcal{O}(\epsilon_k^{-2}\log d)\).
- **Numerical benchmarks:** Across high-dimensional logistic regression and low-rank synthetic tasks, R-ARC and adaptive variants demonstrate substantial speed-ups (often 3–10× faster than full ARC at moderate accuracy) and outperform first-order and classical quasi-Newton methods when Hessian curvature is nontrivial or ill-conditioning is present [1802.05426][2406.16666][2501.09734][2501.03718].

## Table: R-ARC Variants and Key Elements

| Variant         | Subspace Mechanism      | Complexity (First/Second Order)      |
|:----------------|:-----------------------|:-------------------------------------|
| ARC-D [2501.03718]        | Adaptive random subspace (rank-based) | O(ε^{-3/2}) / O(ε^{-3})                 |
| SSCN [2406.16666]         | Random coordinate/minibatch           | Interpolates O(ε^{-2}) to O(ε^{-3/2})   |
| Finite-sum (R-ARC) [1802.05426] | Hessian subsampling (uniform/importance) | O(ε^{-1/3}) (accel., high-prob.)        |
| R-ARC-D [2501.09734]      | Adaptive sketching (OSE)              | O(ε^{-3/2}) / O(ε^{-3})                 |

A plausible implication is that for many large-scale or structurally low-dimensional problems, R-ARC and its descendants will become the de facto strategy to leverage second-order information, interpolating efficiently between first-order simplicity and second-order acceleration.

## References

- Accelerating Adaptive Cubic Regularization of Newton’s Method via Random Sampling [1802.05426]
- Random Subspace Cubic-Regularization Methods, with Applications to Low-Rank Functions [2501.09734]
- Cubic Regularized Subspace Newton for Non-Convex Optimization [2406.16666]
- Scalable Second-Order Optimization Algorithms for Minimizing Low-rank Functions [2501.03718]
- Stochastic Subspace Cubic Newton Method [2002.09526]

Source: https://www.emergentmind.com/topics/random-subspace-cubic-regularization-r-arc