AdpSplit: Adaptive Three-Operator Splitting

• AdpSplit is an adaptive three-operator splitting algorithm for composite convex optimization that employs a backtracking line-search based on local quadratic upper bounds of the smooth function.
• It separates smooth and nonsmooth components, handling them via gradient and proximal operators respectively, and naturally extends to problems with multiple nonsmooth terms.
• Under strong convexity and smoothness assumptions, AdpSplit achieves linear convergence, offering practical speedups over fixed step-size methods.

AdpSplit denotes the adaptive three-operator splitting method introduced in "Adaptive Three Operator Splitting" (Pedregosa et al., 2018). It is an adaptive step-size variant of the Davis–Yin three-operator splitting for composite convex optimization problems of the form

$\min_{x\in\mathbb{R}^p} F(x)=f(x)+g(x)+h(x),$

where $f:\mathbb{R}^p\to\mathbb{R}$ is convex and $L_f$-smooth, and $g,h:\mathbb{R}^p\to\mathbb{R}\cup\{+\infty\}$ are proper, lower-semicontinuous, convex, and proximable. The method replaces the fixed step size of classical three-operator splitting by a backtracking rule based on local quadratic upper bounds of $f$, thereby allowing larger step sizes while preserving the known iteration complexity of the non-adaptive method and extending naturally to problems with an arbitrary number of proximable nonsmooth terms (Pedregosa et al., 2018).

1. Problem class and operator-splitting setting

AdpSplit is formulated for composite convex problems in which the smooth and nonsmooth components are separated according to oracle access: $f$ is handled through its gradient, while $g$ and $h$ are handled through proximal operators (Pedregosa et al., 2018). For any convex function $\varphi$ and $\gamma>0$, the proximal operator is

$f:\mathbb{R}^p\to\mathbb{R}$0

A standard qualification assumption is that the relative interiors of $f:\mathbb{R}^p\to\mathbb{R}$1 and $f:\mathbb{R}^p\to\mathbb{R}$2 intersect. The Euclidean norm is denoted by $f:\mathbb{R}^p\to\mathbb{R}$3, and indicator functions are written $f:\mathbb{R}^p\to\mathbb{R}$4.

The non-adaptive baseline is the Davis–Yin three-operator splitting. In one common explicit form, with $f:\mathbb{R}^p\to\mathbb{R}$5 and fixed $f:\mathbb{R}^p\to\mathbb{R}$6, the iteration is

$f:\mathbb{R}^p\to\mathbb{R}$7

An equivalent form, in variables $f:\mathbb{R}^p\to\mathbb{R}$8, is

$f:\mathbb{R}^p\to\mathbb{R}$9

AdpSplit is obtained by replacing the constant $L_f$0 by an adaptive sequence $L_f$1. When the accepted step sizes satisfy $L_f$2 always, Variant 1 reduces to standard TOS with constant step size.

2. Adaptive step-size mechanism

The defining feature of AdpSplit is an inexpensive backtracking line-search driven by local quadratic upper bounds of the smooth term $L_f$3 (Pedregosa et al., 2018). For current $L_f$4 and trial step size $L_f$5, the quadratic model is

$L_f$6

Given $L_f$7, $L_f$8, and $L_f$9, the candidate primal update is

$g,h:\mathbb{R}^p\to\mathbb{R}\cup\{+\infty\}$0

The trial step is accepted whenever the sufficient decrease condition

$g,h:\mathbb{R}^p\to\mathbb{R}\cup\{+\infty\}$1

holds. If it fails, the algorithm shrinks $g,h:\mathbb{R}^p\to\mathbb{R}\cup\{+\infty\}$2 with fixed $g,h:\mathbb{R}^p\to\mathbb{R}\cup\{+\infty\}$3, recomputes $g,h:\mathbb{R}^p\to\mathbb{R}\cup\{+\infty\}$4, and repeats until acceptance. The backtracking loop terminates in finite time, and the accepted step size satisfies

$g,h:\mathbb{R}^p\to\mathbb{R}\cup\{+\infty\}$5

After acceptance, the remaining updates are

$g,h:\mathbb{R}^p\to\mathbb{R}\cup\{+\infty\}$6

The method has two step-propagation variants. Variant 1 is non-increasing: $g,h:\mathbb{R}^p\to\mathbb{R}\cup\{+\infty\}$7 Variant 2 permits growth when $g,h:\mathbb{R}^p\to\mathbb{R}\cup\{+\infty\}$8 is $g,h:\mathbb{R}^p\to\mathbb{R}\cup\{+\infty\}$9-Lipschitz. Defining

$f$0

the next step size may be chosen in

$f$1

In practice, one may cap growth to avoid frequent backtracking, for example by limiting $f$2.

The initialization is similarly designed to avoid additional hyperparameters. Typical choices are $f$3. For $f$4, the heuristic starts from $f$5, sets $f$6, decreases $f$7 by $f$8 until $f$9, and then takes

$f$0

A common shrink factor is $f$1.

3. Convergence guarantees

The convergence theory is stated through a saddle formulation and ergodic averages (Pedregosa et al., 2018). The saddle objective is

$f$2

with primal and dual objectives

$f$3

Under the qualification condition, saddle points $f$4 satisfy

$f$5

and correspond to minimizers of $f$6 and $f$7.

Let

$f$8

For general convex problems, the method satisfies

$f$9

for all $g$0 and all $g$1. If $g$2 is $g$3-Lipschitz, one obtains the primal suboptimality bound

$g$4

hence an ergodic rate of $g$5.

Under stronger assumptions, AdpSplit also admits linear convergence. Suppose $g$6 is $g$7-strongly convex with $g$8 and $g$9 is $h$0-smooth with $h$1, so $h$2 is $h$3-strongly convex. Define

$h$4

Then Variant 1 satisfies

$h$5

with

$h$6

Variant 2 satisfies

$h$7

with

$h$8

For $h$9 fixed at $\varphi$0, that is, $\varphi$1 and Variant 1, this yields a contraction factor of order $\varphi$2, improving over some earlier analyses of fixed-step TOS that gave rates proportional to $\varphi$3.

4. Extension to multiple proximable terms

AdpSplit extends to an arbitrary number $\varphi$4 of proximable convex terms through a product-space reformulation (Pedregosa et al., 2018). The target problem is

$\varphi$5

where $\varphi$6 is $\varphi$7-smooth and each $\varphi$8 is proximable.

Introduce $\varphi$9 and define

$\gamma>0$0

where

$\gamma>0$1

The proximal operator of $\gamma>0$2 is the projection onto consensus: $\gamma>0$3 Applying AdpSplit to $\gamma>0$4 in the product space produces a line-search model

$\gamma>0$5

This construction preserves the structural appeal of three-operator splitting while accommodating more than two nonsmooth terms. A plausible implication is that the method is especially natural for consensus-form problems in which the nonsmooth terms are separable but the smooth component acts on the average variable. The row-wise prox evaluations of the $\gamma>0$6 remain separable, and the consensus prox is explicit.

5. Computational profile and implementation

Per accepted iteration, AdpSplit requires one gradient of $\gamma>0$7 at $\gamma>0$8, one prox of $\gamma>0$9, one prox of $f:\mathbb{R}^p\to\mathbb{R}$00, and two function evaluations of $f:\mathbb{R}^p\to\mathbb{R}$01, namely $f:\mathbb{R}^p\to\mathbb{R}$02 and $f:\mathbb{R}^p\to\mathbb{R}$03 (Pedregosa et al., 2018). These are the only overhead relative to non-adaptive TOS. In most applications, the gradient and proximal operators dominate runtime, so the two scalar evaluations tend to be negligible.

The method’s stopping criteria can be based on the saddle gap $f:\mathbb{R}^p\to\mathbb{R}$04 with a reference $f:\mathbb{R}^p\to\mathbb{R}$05, the relative change in iterates, the residual $f:\mathbb{R}^p\to\mathbb{R}$06, or primal suboptimality when evaluable. Although the theory is expressed in terms of ergodic averages, the last iterate often performs better in practice; one may return the better of the two as measured by $f:\mathbb{R}^p\to\mathbb{R}$07.

The implementation also admits straightforward specializations. When $f:\mathbb{R}^p\to\mathbb{R}$08, the acceptance condition follows from $f:\mathbb{R}^p\to\mathbb{R}$09-smoothness, and Variant 1 coincides with standard TOS. When $f:\mathbb{R}^p\to\mathbb{R}$10, one has $f:\mathbb{R}^p\to\mathbb{R}$11, and the method reduces to the classical proximal gradient method with backtracking. This places AdpSplit as a genuine three-term generalization of backtracked proximal-gradient schemes rather than a separate optimization paradigm.

In multi-term product-space formulations, the $f:\mathbb{R}^p\to\mathbb{R}$12 prox calls are separable over $f:\mathbb{R}^p\to\mathbb{R}$13 and amenable to parallelization. Memory scales with storing $f:\mathbb{R}^p\to\mathbb{R}$14, $f:\mathbb{R}^p\to\mathbb{R}$15, and $f:\mathbb{R}^p\to\mathbb{R}$16, and, in the product-space case, with matrices of size $f:\mathbb{R}^p\to\mathbb{R}$17. The sufficient decrease condition guarantees finite termination of the line search and maintains the lower bound $f:\mathbb{R}^p\to\mathbb{R}$18; for Variant 2, the growth rule is safeguarded by $f:\mathbb{R}^p\to\mathbb{R}$19 and the cap.

6. Empirical behavior and relation to neighboring methods

The empirical study compares AdpSplit on six settings: logistic regression with overlapping group lasso on the RCV1 and real-sim text datasets, and synthetic inverse problems with four penalties—overlapping group lasso with overlap, $f:\mathbb{R}^p\to\mathbb{R}$20D total variation, trace-norm plus $f:\mathbb{R}^p\to\mathbb{R}$21, and nearly isotonic penalty (Pedregosa et al., 2018). Baselines include non-adaptive TOS with $f:\mathbb{R}^p\to\mathbb{R}$22 and $f:\mathbb{R}^p\to\mathbb{R}$23, PDHG (Condat–Vũ) with tuned $f:\mathbb{R}^p\to\mathbb{R}$24, adaptive PDHG with one line-searched step size but still with a $f:\mathbb{R}^p\to\mathbb{R}$25 hyperparameter, and averaged-operator line-search combined with TOS.

AdpSplit Variant 2 is reported as the best performing method in $f:\mathbb{R}^p\to\mathbb{R}$26 experiments; in the remaining two cases it is roughly tied with the best baseline. In low-regularization regimes, adaptivity yields notably larger step sizes and sometimes order-of-magnitude speedups versus fixed $f:\mathbb{R}^p\to\mathbb{R}$27. When the smooth loss has non-uniform curvature, as in logistic loss, local step-size adaptation is especially beneficial; for quadratic losses, gains are smaller. The two extra evaluations of $f:\mathbb{R}^p\to\mathbb{R}$28 per iteration are therefore treated as a modest overhead relative to the empirical savings from larger admissible steps.

The method occupies a distinct position among related splitting schemes. Relative to non-adaptive TOS, it matches the known sublinear $f:\mathbb{R}^p\to\mathbb{R}$29 theory and attains a cleaner linear factor under strong convexity and smoothness. Relative to FISTA with backtracking, it reduces to proximal gradient with line search when $f:\mathbb{R}^p\to\mathbb{R}$30. Relative to PDHG and its adaptive variants, it uses a single step size with backtracking and requires only an initial guess $f:\mathbb{R}^p\to\mathbb{R}$31 and a shrink factor $f:\mathbb{R}^p\to\mathbb{R}$32, whereas adaptive PDHG still needs a $f:\mathbb{R}^p\to\mathbb{R}$33 hyperparameter. Relative to averaged-operator line-search on TOS, it provides $f:\mathbb{R}^p\to\mathbb{R}$34 guarantees in saddle gap and primal suboptimality, rather than $f:\mathbb{R}^p\to\mathbb{R}$35 operator-residual bounds.

A common misconception is that adaptivity here changes the operator-splitting structure itself. In fact, the splitting remains the Davis–Yin three-operator architecture; the innovation is the line-searched step-size policy and, in Variant 2, a controlled growth rule when $f:\mathbb{R}^p\to\mathbb{R}$36 is $f:\mathbb{R}^p\to\mathbb{R}$37-Lipschitz. This suggests that AdpSplit is best understood not as a new splitting family, but as a step-size-adaptive realization of three-operator splitting with the same asymptotic complexity and a more favorable practical operating regime.