Ridge-Regularized Dual Ascent

• The paper introduces a dual ascent framework that leverages the quadratic dual induced by ridge regularization, yielding efficient algorithms with linear convergence guarantees.
• It details both coordinate-wise and block update methods that exploit full Hessian curvature, reducing iteration complexity especially as minibatch sizes increase.
• Distributed and accelerated variants, including SDNA and Quartz, demonstrate how explicit dual structure can facilitate rapid optimization in large-scale ridge regression.

Ridge-regularized dual ascent denotes a class of primal-dual optimization methods for $L_2$-regularized empirical risk minimization and ridge regression in which the computational work is shifted to a Fenchel dual objective whose smooth component is quadratic. In the canonical setting,

$$\min_{w\in\mathbb{R}^d} P(w):=\frac1n\sum_{i=1}^n \phi_i(a_i^\top w)+\lambda g(w),$$

the ridge specialization $g(w)=\frac12\|w\|^2$ or $g(w)=\frac{\lambda}{2}\|w\|_2^2$ induces a dual problem with a separable conjugate-loss term and a quadratic coupling term, which is the structural feature exploited by stochastic dual coordinate ascent, block Newton ascent, distributed dual ascent, and primal-dual fixed point variants (Qu et al., 2015, Shalev-Shwartz et al., 2013, Yang et al., 2013, Riberio et al., 2018, Raj et al., 2020).

1. Primal–dual structure induced by ridge regularization

A standard starting point is regularized empirical risk minimization with convex smooth losses and a strongly convex regularizer. One formulation is

$$\min_{w\in\mathbb{R}^d} P(w) := \frac1n \sum_{i=1}^n \phi_i(a_i^\top w) + \lambda g(w),$$

with Fenchel dual

$$\max_{\alpha\in\mathbb{R}^n} D(\alpha) := \frac1n \sum_{i=1}^n -\phi_i^*(-\alpha_i) -\lambda\, g^*\!\left(\frac{1}{\lambda n}\alpha\right).$$

Equivalently, minimizing the negative dual gives a composite problem

$$\min_{\alpha\in\mathbb{R}^n} F(\alpha)=f(\alpha)+\sum_{i=1}^n \psi_i(\alpha_i),$$

with

$$f(\alpha)=\lambda g^*\!\left(\frac{1}{\lambda n}\alpha\right), \qquad \psi_i(\alpha_i)=\frac1n \phi_i^*(-\alpha_i).$$

In the $L_2$ case, the decisive fact is that $g^*(s)=\tfrac12\|s\|^2$, so the dual is a concave quadratic maximization problem plus a separable term (Qu et al., 2015).

For ridge regression, the primal is written as

$$\min_{w\in\mathbb{R}^d} P(w):=\frac1n\sum_{i=1}^n \phi_i(a_i^\top w)+\lambda g(w),$$0

and the dual is

$$\min_{w\in\mathbb{R}^d} P(w):=\frac1n\sum_{i=1}^n \phi_i(a_i^\top w)+\lambda g(w),$$1

In this case, the dual is itself a ridge regression problem, and the primal-dual relations can be expressed as

$$\min_{w\in\mathbb{R}^d} P(w):=\frac1n\sum_{i=1}^n \phi_i(a_i^\top w)+\lambda g(w),$$2

These optimality conditions admit multiple equivalent linear reformulations, which later become the basis of relaxed fixed point and Gauss–Seidel-type methods (Riberio et al., 2018).

The significance of ridge regularization is therefore not merely strong convexity in the primal. It also produces a constant, positive semidefinite curvature model in the dual. This suggests that dual ascent methods can be organized around exact or approximate use of that curvature, ranging from scalar coordinate updates to full Hessian-block updates.

2. Coordinate-wise dual ascent and proximal variants

A basic ridge-regularized dual ascent scheme updates one dual coordinate at a time while maintaining the coupled primal quantity

$$\min_{w\in\mathbb{R}^d} P(w):=\frac1n\sum_{i=1}^n \phi_i(a_i^\top w)+\lambda g(w),$$3

In proximal stochastic dual coordinate ascent, the direct coordinate step is

$$\min_{w\in\mathbb{R}^d} P(w):=\frac1n\sum_{i=1}^n \phi_i(a_i^\top w)+\lambda g(w),$$4

and the practical algorithm instead uses a proximal lower bound,

$$\min_{w\in\mathbb{R}^d} P(w):=\frac1n\sum_{i=1}^n \phi_i(a_i^\top w)+\lambda g(w),$$5

followed by

$$\min_{w\in\mathbb{R}^d} P(w):=\frac1n\sum_{i=1}^n \phi_i(a_i^\top w)+\lambda g(w),$$6

For ridge regression with squared loss, the coordinate subproblem has a closed form: $$\min_{w\in\mathbb{R}^d} P(w):=\frac1n\sum_{i=1}^n \phi_i(a_i^\top w)+\lambda g(w),$$7 The corresponding non-accelerated expected runtime is

$$\min_{w\in\mathbb{R}^d} P(w):=\frac1n\sum_{i=1}^n \phi_i(a_i^\top w)+\lambda g(w),$$8

and for ridge regression, where $$\min_{w\in\mathbb{R}^d} P(w):=\frac1n\sum_{i=1}^n \phi_i(a_i^\top w)+\lambda g(w),$$9, this becomes

$g(w)=\frac12\|w\|^2$0

(Shalev-Shwartz et al., 2013).

A distinct but related line formulates explicit regularization in the interpolation regime as a minimum-Bregman-divergence feasibility problem,

$g(w)=\frac12\|w\|^2$1

or equivalently

$g(w)=\frac12\|w\|^2$2

With the Euclidean mirror map

$g(w)=\frac12\|w\|^2$3

the $g(w)=\frac12\|w\|^2$4 case is ridge regularization. The dual is

$g(w)=\frac12\|w\|^2$5

and the primal recovery map is

$g(w)=\frac12\|w\|^2$6

The paper’s ridge specialization in the appendix is

$g(w)=\frac12\|w\|^2$7

which is the classic quadratic dual for regularized empirical risk minimization (Raj et al., 2020).

3. Block updates and Newton-like dual ascent

The most explicit Newton-type ridge-regularized dual ascent method in this literature is Stochastic Dual Newton Ascent (SDNA). It maintains dual variables $g(w)=\frac12\|w\|^2$8 and the primal coupling

$g(w)=\frac12\|w\|^2$9

At iteration $g(w)=\frac{\lambda}{2}\|w\|_2^2$0, a random subset $g(w)=\frac{\lambda}{2}\|w\|_2^2$1 is sampled from a proper, nonvacuous sampling distribution, and the method solves exactly the sampled block subproblem

$g(w)=\frac{\lambda}{2}\|w\|_2^2$2

then updates

$g(w)=\frac{\lambda}{2}\|w\|_2^2$3

and

$g(w)=\frac{\lambda}{2}\|w\|_2^2$4

In the quadratic-loss case,

$g(w)=\frac{\lambda}{2}\|w\|_2^2$5

so the subproblem uses a random principal submatrix of the exact Hessian (Qu et al., 2015).

This distinguishes SDNA from coordinate-separable minibatch SDCA. In SDCA-like methods, minibatch updates rely on diagonal or separable curvature bounds through an expected separable overapproximation, whereas SDNA uses the full principal submatrix on the sampled block. The paper therefore emphasizes that SDNA “utilizes all curvature information available in the random subspace,” and when $g(w)=\frac{\lambda}{2}\|w\|_2^2$6 almost surely, it reduces to a proximal SDCA-like method (Qu et al., 2015).

The convergence theory reflects this curvature usage. For the general composite dual formulation, the paper proves linear convergence,

$g(w)=\frac{\lambda}{2}\|w\|_2^2$7

and for quadratic loss with quadratic regularization,

$g(w)=\frac{\lambda}{2}\|w\|_2^2$8

It also establishes

$g(w)=\frac{\lambda}{2}\|w\|_2^2$9

where $$\min_{w\in\mathbb{R}^d} P(w) := \frac1n \sum_{i=1}^n \phi_i(a_i^\top w) + \lambda g(w),$$0 is the diagonal-based rate, $$\min_{w\in\mathbb{R}^d} P(w) := \frac1n \sum_{i=1}^n \phi_i(a_i^\top w) + \lambda g(w),$$1 an intermediate rate, and $$\min_{w\in\mathbb{R}^d} P(w) := \frac1n \sum_{i=1}^n \phi_i(a_i^\top w) + \lambda g(w),$$2 the full-curvature SDNA rate. In the ridge case, this is the cleanest formal statement that full-curvature randomized block ascent is theoretically at least as good as diagonal-coordinate alternatives (Qu et al., 2015).

A central qualitative conclusion is that SDCA-like methods tend to get worse in iteration complexity as minibatch size increases, whereas SDNA behaves oppositely: larger blocks provide richer Hessian information, so the number of iterations decreases as minibatch size grows, even though each iteration is more expensive. This is a ridge-specific benefit of the dual quadratic structure.

4. Acceleration, fixed-point reformulations, and deterministic Quartz

Ridge-regularized dual ascent also admits acceleration through proximal outer loops and through alternative primal-dual linear-system reformulations. In accelerated Prox-SDCA, the algorithm repeatedly solves proximal subproblems

$$\min_{w\in\mathbb{R}^d} P(w) := \frac1n \sum_{i=1}^n \phi_i(a_i^\top w) + \lambda g(w),$$3

with extrapolated point

$$\min_{w\in\mathbb{R}^d} P(w) := \frac1n \sum_{i=1}^n \phi_i(a_i^\top w) + \lambda g(w),$$4

and parameters

$$\min_{w\in\mathbb{R}^d} P(w) := \frac1n \sum_{i=1}^n \phi_i(a_i^\top w) + \lambda g(w),$$5

The number of outer iterations is bounded by

$$\min_{w\in\mathbb{R}^d} P(w) := \frac1n \sum_{i=1}^n \phi_i(a_i^\top w) + \lambda g(w),$$6

and the total runtime is

$$\min_{w\in\mathbb{R}^d} P(w) := \frac1n \sum_{i=1}^n \phi_i(a_i^\top w) + \lambda g(w),$$7

which, for ridge regression with $$\min_{w\in\mathbb{R}^d} P(w) := \frac1n \sum_{i=1}^n \phi_i(a_i^\top w) + \lambda g(w),$$8, becomes essentially

$$\min_{w\in\mathbb{R}^d} P(w) := \frac1n \sum_{i=1}^n \phi_i(a_i^\top w) + \lambda g(w),$$9

(Shalev-Shwartz et al., 2013).

A different acceleration viewpoint starts from the primal-dual optimality systems of ridge regression. For the coupled system,

$$\max_{\alpha\in\mathbb{R}^n} D(\alpha) := \frac1n \sum_{i=1}^n -\phi_i^*(-\alpha_i) -\lambda\, g^*\!\left(\frac{1}{\lambda n}\alpha\right).$$0

the relaxed fixed point iteration is

$$\max_{\alpha\in\mathbb{R}^n} D(\alpha) := \frac1n \sum_{i=1}^n -\phi_i^*(-\alpha_i) -\lambda\, g^*\!\left(\frac{1}{\lambda n}\alpha\right).$$1

The paper studies several such reformulations and their spectra. One matrix has real nonpositive eigenvalues; another has purely imaginary eigenvalues with

$$\max_{\alpha\in\mathbb{R}^n} D(\alpha) := \frac1n \sum_{i=1}^n -\phi_i^*(-\alpha_i) -\lambda\, g^*\!\left(\frac{1}{\lambda n}\alpha\right).$$2

These spectral differences produce different optimal relaxation parameters and rates (Riberio et al., 2018).

The main deterministic accelerated scheme is Quartz, implemented as a block Gauss–Seidel update: $$\max_{\alpha\in\mathbb{R}^n} D(\alpha) := \frac1n \sum_{i=1}^n -\phi_i^*(-\alpha_i) -\lambda\, g^*\!\left(\frac{1}{\lambda n}\alpha\right).$$3 with

$$\max_{\alpha\in\mathbb{R}^n} D(\alpha) := \frac1n \sum_{i=1}^n -\phi_i^*(-\alpha_i) -\lambda\, g^*\!\left(\frac{1}{\lambda n}\alpha\right).$$4

Quartz converges for

$$\max_{\alpha\in\mathbb{R}^n} D(\alpha) := \frac1n \sum_{i=1}^n -\phi_i^*(-\alpha_i) -\lambda\, g^*\!\left(\frac{1}{\lambda n}\alpha\right).$$5

its optimal parameter is

$$\max_{\alpha\in\mathbb{R}^n} D(\alpha) := \frac1n \sum_{i=1}^n -\phi_i^*(-\alpha_i) -\lambda\, g^*\!\left(\frac{1}{\lambda n}\alpha\right).$$6

and in the important scaling $$\max_{\alpha\in\mathbb{R}^n} D(\alpha) := \frac1n \sum_{i=1}^n -\phi_i^*(-\alpha_i) -\lambda\, g^*\!\left(\frac{1}{\lambda n}\alpha\right).$$7 the paper derives

$$\max_{\alpha\in\mathbb{R}^n} D(\alpha) := \frac1n \sum_{i=1}^n -\phi_i^*(-\alpha_i) -\lambda\, g^*\!\left(\frac{1}{\lambda n}\alpha\right).$$8

which it explicitly identifies with the optimal accelerated Nesterov rate $$\max_{\alpha\in\mathbb{R}^n} D(\alpha) := \frac1n \sum_{i=1}^n -\phi_i^*(-\alpha_i) -\lambda\, g^*\!\left(\frac{1}{\lambda n}\alpha\right).$$9 (Riberio et al., 2018).

These results place ridge-regularized dual ascent in a broader class of accelerated primal-dual linear methods. A plausible implication is that the favorable dual quadratic structure is compatible with both stochastic and deterministic acceleration mechanisms, provided the primal-dual coupling is used directly rather than only through diagonal surrogate curvature.

5. Distributed ridge-regularized dual ascent

Distributed stochastic dual coordinate ascent extends the same ridge-aware dual structure to multi-machine settings. The underlying primal objective is

$$\min_{\alpha\in\mathbb{R}^n} F(\alpha)=f(\alpha)+\sum_{i=1}^n \psi_i(\alpha_i),$$0

and under ridge regularization

$$\min_{\alpha\in\mathbb{R}^n} F(\alpha)=f(\alpha)+\sum_{i=1}^n \psi_i(\alpha_i),$$1

The dual then becomes

$$\min_{\alpha\in\mathbb{R}^n} F(\alpha)=f(\alpha)+\sum_{i=1}^n \psi_i(\alpha_i),$$2

or equivalently

$$\min_{\alpha\in\mathbb{R}^n} F(\alpha)=f(\alpha)+\sum_{i=1}^n \psi_i(\alpha_i),$$3

with Gram matrix $$\min_{\alpha\in\mathbb{R}^n} F(\alpha)=f(\alpha)+\sum_{i=1}^n \psi_i(\alpha_i),$$4 (Yang et al., 2013).

The paper distinguishes a naive distributed update from a practical update. In the naive variant, each dual increment is computed using the stale global primal vector $$\min_{\alpha\in\mathbb{R}^n} F(\alpha)=f(\alpha)+\sum_{i=1}^n \psi_i(\alpha_i),$$5: $$\min_{\alpha\in\mathbb{R}^n} F(\alpha)=f(\alpha)+\sum_{i=1}^n \psi_i(\alpha_i),$$6 with $$\min_{\alpha\in\mathbb{R}^n} F(\alpha)=f(\alpha)+\sum_{i=1}^n \psi_i(\alpha_i),$$7. In the practical variant, the algorithm uses the current local primal vector $$\min_{\alpha\in\mathbb{R}^n} F(\alpha)=f(\alpha)+\sum_{i=1}^n \psi_i(\alpha_i),$$8: $$\min_{\alpha\in\mathbb{R}^n} F(\alpha)=f(\alpha)+\sum_{i=1}^n \psi_i(\alpha_i),$$9 and updates

$$f(\alpha)=\lambda g^*\!\left(\frac{1}{\lambda n}\alpha\right), \qquad \psi_i(\alpha_i)=\frac1n \phi_i^*(-\alpha_i).$$0

The practical update can be interpreted as SDCA on a local subproblem, rather than as an ad hoc heuristic (Yang et al., 2013).

The rate distinction is substantial. For the naive variant, the quoted bound is

$$f(\alpha)=\lambda g^*\!\left(\frac{1}{\lambda n}\alpha\right), \qquad \psi_i(\alpha_i)=\frac1n \phi_i^*(-\alpha_i).$$1

Under orthogonality across machine partitions, the practical variant yields

$$f(\alpha)=\lambda g^*\!\left(\frac{1}{\lambda n}\alpha\right), \qquad \psi_i(\alpha_i)=\frac1n \phi_i^*(-\alpha_i).$$2

The paper interprets this as an exponential speed-up in the number of local dual updates $$f(\alpha)=\lambda g^*\!\left(\frac{1}{\lambda n}\alpha\right), \qquad \psi_i(\alpha_i)=\frac1n \phi_i^*(-\alpha_i).$$3 and a linear speed-up in the number of machines $$f(\alpha)=\lambda g^*\!\left(\frac{1}{\lambda n}\alpha\right), \qquad \psi_i(\alpha_i)=\frac1n \phi_i^*(-\alpha_i).$$4 in the orthogonal case (Yang et al., 2013).

The distributed analysis also clarifies the one-communication regime. If $$f(\alpha)=\lambda g^*\!\left(\frac{1}{\lambda n}\alpha\right), \qquad \psi_i(\alpha_i)=\frac1n \phi_i^*(-\alpha_i).$$5 for $$f(\alpha)=\lambda g^*\!\left(\frac{1}{\lambda n}\alpha\right), \qquad \psi_i(\alpha_i)=\frac1n \phi_i^*(-\alpha_i).$$6, then the Gram matrix is block diagonal and the dual decomposes into independent subproblems. Communication is then unnecessary during local optimization; the models can be averaged once at the end. The paper’s interpretation is that the closer the partition is to orthogonality, the closer one-communication averaging is to the true optimum (Yang et al., 2013).

6. Error measures, interpretations, and recurrent themes

Across these methods, ridge-regularized dual ascent is characterized by strong primal-dual recovery guarantees. In the explicit-regularization framework, the key inequality is

$$f(\alpha)=\lambda g^*\!\left(\frac{1}{\lambda n}\alpha\right), \qquad \psi_i(\alpha_i)=\frac1n \phi_i^*(-\alpha_i).$$7

and the dual gap decomposes as

$$f(\alpha)=\lambda g^*\!\left(\frac{1}{\lambda n}\alpha\right), \qquad \psi_i(\alpha_i)=\frac1n \phi_i^*(-\alpha_i).$$8

For randomized coordinate descent,

$$f(\alpha)=\lambda g^*\!\left(\frac{1}{\lambda n}\alpha\right), \qquad \psi_i(\alpha_i)=\frac1n \phi_i^*(-\alpha_i).$$9

and for accelerated coordinate descent,

$L_2$0

In the ridge/SDCA appendix, the paper states linear convergence to the ridge-regularized primal solution: $L_2$1 with an accelerated analogue as well (Raj et al., 2020).

For SDNA, the duality-gap bound is

$L_2$2

showing that full-curvature block ascent controls not only dual progress but also primal suboptimality (Qu et al., 2015).

Several interpretive themes recur in the literature. First, ridge-regularized dual ascent is not limited to scalar coordinate updates: SDNA uses exact Hessian blocks, distributed variants aggregate many local updates, and fixed point methods operate on coupled primal-dual linear systems. Second, the dual viewpoint is not restricted to implicit bias arguments. In the interpolation regime, dual coordinate-ascent and Dykstra-style methods are used precisely to impose explicit regularization and to obtain convergence to a minimum norm solution (Raj et al., 2020). Third, in least squares with $L_2$3 regularization, SDNA’s primal iterate can be written as a randomized Hessian-sketch-type update, so the method can be interpreted as a variant of Iterative Hessian Sketch inside a dual-ascent framework (Qu et al., 2015).

Taken together, these results present ridge-regularized dual ascent as a technically coherent family of methods unified by one structural fact: the ridge term makes the dual smooth part quadratic. From that point onward, the main distinctions are algorithmic—single-coordinate versus block, proximal versus exact, randomized versus deterministic, local versus distributed, and first-order versus curvature-exploiting—rather than conceptual.