---
title: Sequential Line Search (SLS)
url: https://www.emergentmind.com/topics/sequential-line-search-sls
type: topic
---

# Sequential Line Search (SLS)

Sequential Line Search (SLS) refers collectively to a class of optimization algorithms that utilize one-dimensional line searches, either deterministically or stochastically, as a core strategy for updating variables or parameter blocks within larger iterative schemes. SLS methods facilitate efficient, targeted updates by identifying optimal (or near-optimal) step sizes along chosen search directions, typically to minimize a nonsmooth or composite objective. These approaches are widely deployed in modern convex and stochastic optimization, particularly within penalized regression (e.g., group lasso) and stochastic gradient descent-type algorithms. Central theoretical and empirical results demonstrate that SLS variants can outperform traditional coordinate descent, blockwise descent with inexact searches, and global projection methods, especially in high-dimensional and structured statistical learning contexts [1010.3320, 2308.06058].

## 1. SLS in Blockwise Optimization for Group Lasso

In the context of the group lasso—the regression problem where predictors are partitioned into groups and groupwise sparsity is induced using an $\ell_2$-norm penalty—SLS provides an exact blockwise optimization procedure. The classical group lasso estimator for $G$ groups is

\[
\min_{\beta\in\mathbb R^p}\;\frac12\|y - X\,\beta\|_2^2\;+\;\lambda\sum_{g=1}^G w_g\,\|\beta^{(g)}\|_2,
\]

with $X$ partitioned into blocks $X^{(g)}$, and blockwise coefficients $\beta^{(g)}$. For each group $g$, SLS solves the associated subproblem in $\alpha$:

\[
\min_{\alpha\in\mathbb R^{p_g}}\;Q(\alpha) = \tfrac12\|\,r - X^{(g)}\,\alpha\|_2^2\;+\;\lambda\,w_g\,\|\alpha\|_2,
\]

with $r$ the partial residual fixing other groups. By leveraging the spectral decomposition of the block Gram matrix and formulating a univariate equation in a radial variable $r$, the global minimum in the subspace can be computed via a one-dimensional line search, specifically solving

\[
f(r)\;=\;\sum_{j=1}^{p_g}\frac{v_j^2}{\bigl(d_j\,r+\lambda\,w_g\bigr)^2}\;=\;1,
\]

where $d_j$ are the eigenvalues of $X^{(g)T}X^{(g)}$ and $v$ is a rotated gradient vector. If $\|v\|_2\le\lambda\,w_g$, the update is zero; otherwise, the unique root $r^\star > 0$ yields the blockwise update [1010.3320].

## 2. Stochastic Line-Search SLS in SGD and Adaptive Algorithms

Within stochastic optimization, SLS strategies are employed to dynamically select step sizes in stochastic gradient methods. At each iteration, a minibatch $i_t$ is sampled, and a trial step $\gamma$ is selected via backtracking line search to satisfy an Armijo-type condition on the sampled function:

\[
f_{i_t}(x_t-\gamma\,\nabla f_{i_t}(x_t))\le f_{i_t}(x_t)-\rho\,\gamma\,\|\nabla f_{i_t}(x_t)\|^2,
\]

with $\rho\in(0,1)$ and a backtracking factor $\beta\in[\tfrac12,1)$ controlling the reduction in $\gamma$ if the condition fails. The procedure enforces monotonic decrease or other safeguards as needed, and the iterate is updated with the found $\eta_t$ [2308.06058].

Adaptive SLS variants such as AdaSLS further rescale accepted steps with an AdaGrad-norm-style denominator, facilitating parameter-free operation—no need for $L$-smoothness or strong convexity parameters. This yields theoretical $O(1/T)$ convergence rates in convex interpolation regimes and robust behavior in noisy or non-interpolated settings.

## 3. Convergence Properties and Theoretical Guarantees

For blockwise SLS in group lasso, convergence is established via block-coordinate descent theory. Each sweep over all blocks produces a non-increasing objective, and the iterates converge to the unique solution under standard convexity assumptions. Finite-time error bounds are given in terms of current subgradient norms, and global optimality is ensured by strict convexity within blocks [1010.3320].

In stochastic settings, SLS-type procedures admit the following behaviors under standard smoothness and convexity:

- Interpolated, strongly convex: Linear convergence of $\|x_T-x^*\|^2= O(\exp(-cT))$.
- Interpolated, convex: $O(1/T)$ convergence in $f(\bar x_T)-f^*$.
- Non-interpolated (noisy): Converges to a neighborhood depending on the variance $\Omega(\sigma_{f,B}^2)$. AdaSLS matches these rates while eliminating the need for tuning problem-dependent parameters [2308.06058].

Variance-reduced SLS schemes (e.g., AdaSVRLS) achieve optimal $O(n\log(1/\varepsilon)+1/\varepsilon)$ gradient evaluation complexity via loopless SVRG-style estimators.

## 4. Extensions: Sparse Group Lasso and Signed SLS

The SLS methodology generalizes seamlessly to sparse group lasso, where both group-level ($\ell_2$) and elementwise ($\ell_1$) penalties are imposed:

\[
\min_\beta\;\tfrac12\|y-X\beta\|_2^2 + \lambda_1\sum_g\|\beta^{(g)}\|_2 + \lambda_2\|\beta\|_1.
\]

In this context, the Signed Single Line Search (SSLS) algorithm is employed: the subproblem for each block $\alpha$ requires identifying the active sign pattern and then solving a corresponding univariate equation conditional on the support. If the soft-thresholded block gradient norm is below $\lambda_1$, the solution is zero; otherwise, a search over sign patterns and corresponding line searches produces the exact solution. Theoretical global convergence is established [1010.3320].

## 5. Computational Considerations and Comparative Performance

Blockwise SLS for group lasso achieves per-group update cost $O(n p_g + p_g^2)$ after a one-time $O(p_g^3)$ spectral decomposition, with per-sweep residual calculations scaling linearly in $n$. By comparison:

- Inexact group-wise descent incurs higher line-search cost per block.
- Coordinate descent may erroneously stall at non-unique blockwise solutions.
- Gradient projection updates the entire parameter vector and may require expensive projections.
- Active-set methods offer speed for extremely sparse solutions but have higher active-set update costs.

SLS and its signed variant SSLS yield substantial empirical speed-ups—orders of magnitude in simulations where group sizes are moderate and within-group correlations high—over alternative methods [1010.3320]. In stochastic optimization, the line-search overhead is minimal, adding $O(\log(L \gamma_{\max} / (1-\rho)))$ function evaluations per iteration, while ensuring robust adaptation to unknown smoothness or variance structure [2308.06058].

| SLS Variant       | Application Domain        | Notable Properties/Results                                  |
|-------------------|--------------------------|-------------------------------------------------------------|
| Blockwise SLS     | (Sparse) Group Lasso     | Exact univariate update per block, global convergence       |
| SSLS              | Sparse Group Lasso       | Signed pattern search, global convergence, robust sparsity  |
| Stochastic SLS    | SGD and AdaSLS           | Armijo line search per minibatch, adaptive scaling          |
| AdaSVRLS          | Variance-Reduced SGD     | Loopless SVRG, optimal $O(n\log(1/\epsilon)+1/\epsilon)$   |

## 6. Practical Implementation and Empirical Observations

Empirical studies indicate that SLS methodologies deliver significant efficiency gains in high-dimensional penalized regression—particularly for groupwise-structured problems where groups are large or predictors are highly correlated within groups. SSLS enables practical sparse group lasso solutions for moderate group sizes (e.g., $p_g \le 5$, $G=40$) in seconds on standard hardware [1010.3320]. In stochastic optimization, AdaSLS self-tunes between constant and decaying step sizes, matching or outperforming classic adaptive methods (SPS, AdaGrad), while variance-reduced extensions attain optimal complexity-class guarantees [2308.06058].

Practical considerations include the setup of line-search parameters, numerically stable computation of blockwise spectral decompositions, and low additional overhead due to efficient inner-loop line search procedures.

## 7. Broader Impact, Variants, and Related Methods

SLS serves as a foundational approach within both deterministic and stochastic optimization for structured regularization and smooth convex learning. Key variants span blockwise search in composite-regularized models, stochastic Armijo step selection in SGD, and AdaGrad-style adaptivity for learning rate schedules. All maintain guarantees of monotonic descent, global convergence, and computational tractability in relevant regimes [1010.3320, 2308.06058].

Potential avenues for further development include scaling SSLS to large $p_g$ via structured search strategies, extending adaptive SLS theory to nonconvex settings, and developing parallel and GPU-optimized implementations for large-scale machine learning tasks. A plausible implication is that SLS-based strategies can act as unifying schemes bridging deterministic blockwise and stochastic mini-batch optimization principles.

Source: https://www.emergentmind.com/topics/sequential-line-search-sls