Randomized ADMM Methods

• Randomized ADMM is a family of methods that introduce randomness in block order, selection, or data sampling to ensure convergence in multi-block optimization problems.
• Key mechanisms such as random permutation, block activation, and random block assembly offer strong convergence guarantees and improved spectral properties in various problem formulations.
• Randomized ADMM has practical applications in distributed consensus, machine learning, beamforming, and inverse problems, providing robustness and computational efficiency.

Randomized ADMM denotes a family of modifications of the alternating direction method of multipliers in which randomness enters the block order, block selection, data sampling, sketching operator, or inner linear algebra. The unifying motivation is that the direct multi-block generalization of ADMM is not necessarily convergent, whereas carefully chosen randomization can restore convergence in expectation, almost sure convergence, or ergodic rate guarantees in settings ranging from linear systems and convex quadratic programming to distributed consensus, stochastic learning, PDE-constrained inversion, and large-scale beamforming (Sun et al., 2015, Dang et al., 2014, Iutzeler et al., 2013, Frangella et al., 2023).

1. Problem classes and basic formulation

A canonical multi-block formulation is

$$\min_{x_1,\dots,x_n}\; f(x_1,\dots,x_n)\quad\text{s.t.}\quad \sum_{i=1}^n A_i x_i=b,$$

with augmented Lagrangian

$$\mathcal{L}(x_1,\dots,x_n;\mu)=f(x_1,\dots,x_n)-\mu^\top\Big(\sum_i A_i x_i-b\Big)+\frac{\beta}{2}\Big\|\sum_i A_i x_i-b\Big\|^2.$$

This model is the point of departure for the random-permutation analysis of multi-block ADMM and its close connection to randomized block coordinate descent (Sun et al., 2015).

A second foundational formulation is the bilinear saddle problem

$$\min_{x\in X}\;\Big\{h(x)+\max_{y\in Y}\;\langle A x,\;y\rangle - J(y)\Big\},$$

with block-product dual region $Y=Y_1\times\cdots\times Y_p$. In this setting, randomized primal-dual updates become exact randomized variants of multi-block ADMM for linearly constrained separable problems (Dang et al., 2014).

A third major setting is distributed consensus. For networked agents with private costs $f_v$, the global objective

$\inf_{x\in \mathsf X}\;\sum_{v\in V} f_v(x)$

can be written as a constrained consensus problem over local copies, and then attacked by asynchronous randomized ADMM through random component activations (Iutzeler et al., 2013).

Across these formulations, the same structural difficulty recurs: when the number of blocks exceeds two, a direct cyclic extension of ADMM may diverge. This is explicit in the randomized primal-dual literature and in the random-permutation analysis for multi-block ADMM (Dang et al., 2014, Sun et al., 2015).

2. Main randomization mechanisms

Randomized ADMM is not a single algorithmic template but a collection of mechanisms that intervene at different levels of the ADMM pipeline.

Mechanism	Representative update pattern	Representative guarantee or use
Random permutation	Uniformly sample a permutation and perform one Gauss–Seidel sweep without replacement	Expected convergence for linear systems; spectral tightening to $(-1/3,1)$ for the expected RP-BCD operator (Sun et al., 2015)
Random block activation	Update one component, edge, node, or block subset at each iteration	Almost sure convergence in asynchronous consensus; ergodic $O(1/N)$ and $O(1/N^2)$ rates in randomized primal-dual form (Iutzeler et al., 2013, Dang et al., 2014)
Random block assembly	Repartition variables into new blocks every epoch, then perform cyclic ADMM	Expected convergence for convex QP; almost sure convergence if $\rho(\mathbb E[M_\sigma\otimes M_\sigma])<1$ (Mihic et al., 2019)
Stochastic or sketched subproblems	Replace full losses, sources, or Hessians by random samples or sketches	$$\mathcal{L}(x_1,\dots,x_n;\mu)=f(x_1,\dots,x_n)-\mu^\top\Big(\sum_i A_i x_i-b\Big)+\frac{\beta}{2}\Big\|\sum_i A_i x_i-b\Big\|^2.$$0 stochastic rates, at least one order-of-magnitude PDE-solve reduction, and explicit inexact/sketched ADMM rates (Zhao et al., 2013, Aghazade et al., 2021, Frangella et al., 2023)

The best-known classical distinction is between sampling without replacement and sampling with replacement. In the multi-block linear-system analysis, random permutation is theoretically and empirically favorable, whereas with-replacement variants such as PD-RADMM and P-RADMM can diverge (Sun et al., 2015). This makes “randomized ADMM” a qualitative design choice rather than merely a stochastic perturbation of a deterministic sweep.

Other mechanisms are more specialized. In stochastic ADMM, one replaces the expected loss by a random loss associated with one uniformly drawn example plus a Bregman divergence, and then chooses the quadratic proximal metric adaptively (Zhao et al., 2013). In inverse problems, randomness may enter through a sketching matrix $$\mathcal{L}(x_1,\dots,x_n;\mu)=f(x_1,\dots,x_n)-\mu^\top\Big(\sum_i A_i x_i-b\Big)+\frac{\beta}{2}\Big\|\sum_i A_i x_i-b\Big\|^2.$$1 that compresses the source dimension while ADMM handles the extended-space constraints (Aghazade et al., 2021). In privacy-preserving decentralized consensus, randomization can act on the penalty parameter or on the primal variables along a Hamiltonian cycle, producing incremental ADMM variants with privacy guarantees against an external eavesdropper (Ye et al., 2020).

3. Operator-theoretic and spectral structure

The most detailed spectral analysis is available for randomly permuted ADMM on linear systems. For

$$\mathcal{L}(x_1,\dots,x_n;\mu)=f(x_1,\dots,x_n)-\mu^\top\Big(\sum_i A_i x_i-b\Big)+\frac{\beta}{2}\Big\|\sum_i A_i x_i-b\Big\|^2.$$2

with $$\mathcal{L}(x_1,\dots,x_n;\mu)=f(x_1,\dots,x_n)-\mu^\top\Big(\sum_i A_i x_i-b\Big)+\frac{\beta}{2}\Big\|\sum_i A_i x_i-b\Big\|^2.$$3 nonsingular and $$\mathcal{L}(x_1,\dots,x_n;\mu)=f(x_1,\dots,x_n)-\mu^\top\Big(\sum_i A_i x_i-b\Big)+\frac{\beta}{2}\Big\|\sum_i A_i x_i-b\Big\|^2.$$4, the stacked iterate $$\mathcal{L}(x_1,\dots,x_n;\mu)=f(x_1,\dots,x_n)-\mu^\top\Big(\sum_i A_i x_i-b\Big)+\frac{\beta}{2}\Big\|\sum_i A_i x_i-b\Big\|^2.$$5 satisfies

$$\mathcal{L}(x_1,\dots,x_n;\mu)=f(x_1,\dots,x_n)-\mu^\top\Big(\sum_i A_i x_i-b\Big)+\frac{\beta}{2}\Big\|\sum_i A_i x_i-b\Big\|^2.$$6

for a permutation-dependent linear operator $$\mathcal{L}(x_1,\dots,x_n;\mu)=f(x_1,\dots,x_n)-\mu^\top\Big(\sum_i A_i x_i-b\Big)+\frac{\beta}{2}\Big\|\sum_i A_i x_i-b\Big\|^2.$$7. Averaging over uniform random permutations yields

$$\mathcal{L}(x_1,\dots,x_n;\mu)=f(x_1,\dots,x_n)-\mu^\top\Big(\sum_i A_i x_i-b\Big)+\frac{\beta}{2}\Big\|\sum_i A_i x_i-b\Big\|^2.$$8

with $$\mathcal{L}(x_1,\dots,x_n;\mu)=f(x_1,\dots,x_n)-\mu^\top\Big(\sum_i A_i x_i-b\Big)+\frac{\beta}{2}\Big\|\sum_i A_i x_i-b\Big\|^2.$$9 symmetric (Sun et al., 2015).

The central spectral statement is

$$\min_{x\in X}\;\Big\{h(x)+\max_{y\in Y}\;\langle A x,\;y\rangle - J(y)\Big\},$$0

equivalently

$$\min_{x\in X}\;\Big\{h(x)+\max_{y\in Y}\;\langle A x,\;y\rangle - J(y)\Big\},$$1

The same analysis establishes the eigenvalue transform

$$\min_{x\in X}\;\Big\{h(x)+\max_{y\in Y}\;\langle A x,\;y\rangle - J(y)\Big\},$$2

so the bound on $$\min_{x\in X}\;\Big\{h(x)+\max_{y\in Y}\;\langle A x,\;y\rangle - J(y)\Big\},$$3 yields $$\min_{x\in X}\;\Big\{h(x)+\max_{y\in Y}\;\langle A x,\;y\rangle - J(y)\Big\},$$4 and hence expected convergence of RP-ADMM for nonsingular linear systems (Sun et al., 2015).

A second operator-theoretic strand interprets randomized ADMM as randomized Douglas–Rachford or averaged-operator iterations. In asynchronous distributed optimization, the method is a randomized Gauss–Seidel iteration of a Douglas–Rachford operator on the dual, and the induced block iteration

$$\min_{x\in X}\;\Big\{h(x)+\max_{y\in Y}\;\langle A x,\;y\rangle - J(y)\Big\},$$5

converges almost surely to a random point in $$\min_{x\in X}\;\Big\{h(x)+\max_{y\in Y}\;\langle A x,\;y\rangle - J(y)\Big\},$$6 under i.i.d. activations with positive probabilities (Iutzeler et al., 2013). Closely related work formulates ADMM+ as an $$\min_{x\in X}\;\Big\{h(x)+\max_{y\in Y}\;\langle A x,\;y\rangle - J(y)\Big\},$$7-averaged primal-dual operator and then applies randomized Krasnosel’skii–Mann updates to obtain node-asynchronous distributed methods; when $$\min_{x\in X}\;\Big\{h(x)+\max_{y\in Y}\;\langle A x,\;y\rangle - J(y)\Big\},$$8 and $$\min_{x\in X}\;\Big\{h(x)+\max_{y\in Y}\;\langle A x,\;y\rangle - J(y)\Big\},$$9, ADMM+ reduces to classical ADMM (Bianchi et al., 2014).

For randomly assembled cyclic ADMM, the expected iteration matrix retains the same block form as in the random-permutation case, with $Y=Y_1\times\cdots\times Y_p$0 now averaging over both random partitions and update orders. The almost sure convergence criterion

$Y=Y_1\times\cdots\times Y_p$1

controls second moments and separates mean stability from sample-path stability (Mihic et al., 2019).

4. Convergence rates and stability regimes

The guarantees available for randomized ADMM are heterogeneous because the randomization mechanisms solve different analytical problems.

For random permutation on linear systems, expected convergence is explicit. If $Y=Y_1\times\cdots\times Y_p$2 is nonsingular and $Y=Y_1\times\cdots\times Y_p$3 for all $Y=Y_1\times\cdots\times Y_p$4, then

$Y=Y_1\times\cdots\times Y_p$5

with $Y=Y_1\times\cdots\times Y_p$6. In the corresponding quadratic RP-BCD setting,

$Y=Y_1\times\cdots\times Y_p$7

and the expected RP-BCD rate is $Y=Y_1\times\cdots\times Y_p$8 times better than the worst-case rate of cyclic BCD (Sun et al., 2015).

In randomized primal-dual form, bounded-domain bilinear saddle problems admit ergodic $Y=Y_1\times\cdots\times Y_p$9 convergence without strong convexity, and smooth bilinear saddle problems admit $f_v$0 rates when $f_v$1 is strongly convex. The same framework yields randomized single-block ADMM updates for linearly constrained separable problems (Dang et al., 2014).

For general multi-block convex optimization with coupled objectives and linear constraints, randomized primal-dual proximal block coordinate updates establish $f_v$2 convergence in expectation for objective suboptimality and feasibility violation under mere convexity, and extend to $f_v$3 stochastic rates when only stochastic gradient approximations are available (Gao et al., 2016). At the opposite end of the approximation spectrum, GeNI-ADMM encompasses inexact first- and second-order ADMM schemes and yields the usual $f_v$4 rate under standard hypotheses, together with linear convergence under additional hypotheses such as strong convexity; this explicitly covers NysADMM and sketch-and-solve ADMM (Frangella et al., 2023).

Application-specific randomized ADMM can preserve these orders. In large-scale max–min beamforming for cell-free massive MIMO, the randomized method updates only a random subset of beamformer blocks each iteration yet retains an $f_v$5 convergence rate, matching the order of its deterministic counterpart (Wang et al., 25 Jul 2025).

A persistent theme is the difference between expectation guarantees and stronger notions of robustness. The ALM-based analysis of random multi-block ADMM for strongly convex QP emphasizes that convergence in expectation may not be a good indicator of robustness and efficiency, and shows that one randomized GS or RSSOR sweep is in general not accurate enough to guarantee convergence of the outer method; a constant number of inner CG, SOR, or randomly shuffled SOR iterations suffices to recover almost sure convergence in the inexact ALM sense (Cipolla et al., 2020).

5. Representative applications and implementations

Distributed consensus is one of the oldest application areas. Randomized asynchronous ADMM updates only one active network component at a time, requires local averaging within the active component, and converges almost surely under mild connectivity and activation assumptions (Iutzeler et al., 2013). Related node-asynchronous distributed primal-dual schemes update a random subset of agents at each iteration and include ADMM as a special case (Bianchi et al., 2014). In constrained multi-agent optimization with local polyhedral constraints, randomized proximal dual consensus ADMM handles polyhedra softly through slack variables and proximal terms, is robust against randomly ON/OFF agents and imperfect communication links, and has worst-case $f_v$6 convergence in expectation (Chang, 2014).

Machine learning has provided a second major testbed. Adaptive stochastic ADMM replaces the full expected loss by a random loss associated with one uniformly drawn example plus a Bregman divergence, and uses diagonal or full-matrix adaptive metrics that yield AdaGrad-type behavior with regret bounds matching the best proximal function chosen in hindsight up to constants (Zhao et al., 2013). Randomly assembled cyclic multi-block ADMM has also been applied to Linear Regression, LASSO, Elastic-Net, and SVM, with numerical tests reporting that it could significantly outperform other optimization algorithms or codes on many quadratic machine-learning instances and match the performance of specialized solvers such as Glmnet or LIBSVM (Zhu et al., 2019).

Inverse problems and scientific computing have motivated sketching-based variants. In frequency-domain full waveform inversion, randomized source sketching projects the source dimension into a smaller domain through a sketching matrix $f_v$7, and ADMM is then applied in an extended search-space formulation. The reported numerical examples show that the randomized sketching algorithm reduces the cost of large-scale problems by at least one order of magnitude compared to the original deterministic algorithm (Aghazade et al., 2021).

Randomized ADMM has also entered modern wireless optimization. In max–min beamforming for large-scale cell-free massive MIMO, a randomized ADMM is built on a reformulation of the feasibility check as a linearly constrained optimization problem; it updates only a small number of subproblems at each iteration, offers a significant complexity advantage over existing methods, and preserves the deterministic $f_v$8 rate order (Wang et al., 25 Jul 2025).

A different application thread concerns privacy-preserving decentralized consensus. Incremental ADMM along a Hamiltonian cycle is communication efficient but does not guarantee privacy against an external eavesdropper. Two privacy-preserving variants, PI-ADMM1 and PI-ADMM2, randomize either the step sizes or the primal variables; PI-ADMM1 is proved to preserve privacy and converge, and both algorithms are reported as communication efficient compared with state-of-the-art methods (Ye et al., 2020).

6. Limitations, controversies, and derandomized alternatives

The main controversy surrounding randomized ADMM is not whether randomization can help, but which randomization is structurally compatible with ADMM. The random-permutation analysis shows that sampling without replacement is qualitatively different from with-replacement schemes: RP-ADMM converges in expectation for nonsingular linear systems, while PD-RADMM and P-RADMM can diverge, and cyclic ADMM can diverge even when every random-permutation average is contractive (Sun et al., 2015).

Random block assembly adds another layer of subtlety. RAC-ADMM converges in expectation for convex QP under the stated blockwise positive-definiteness condition, yet almost sure convergence requires the stronger criterion

$f_v$9

The existence of examples with $\inf_{x\in \mathsf X}\;\sum_{v\in V} f_v(x)$0 but $\inf_{x\in \mathsf X}\;\sum_{v\in V} f_v(x)$1 shows that mean stability and sample-path stability need not coincide (Mihic et al., 2019).

Approximate and sketch-based variants have analogous failure modes. GeNI-ADMM proves $\inf_{x\in \mathsf X}\;\sum_{v\in V} f_v(x)$2 and linear convergence for NysADMM and sketch-and-solve ADMM only when the inexactness schedules are summable or geometrically decaying and, for the sketched Hessian case, when a correction term such as $\inf_{x\in \mathsf X}\;\sum_{v\in V} f_v(x)$3 restores the relative smoothness condition. The experiments reported there show that omitting the correction term can lead to divergence or oscillation (Frangella et al., 2023).

These limitations have motivated deterministic surrogates for randomization. A derandomized algorithm based on a block symmetric Gauss–Seidel sweep and a Richardson-type dual correction replaces random permutation by a forward–backward SGS pass and updates the dual by

$\inf_{x\in \mathsf X}\;\sum_{v\in V} f_v(x)$4

For linear systems, the paper proves linear convergence under this scheme and presents it as a deterministic alternative to RP-ADMM (Xu et al., 2017).

Several open questions remain explicit in the literature. The random-permutation analysis introduces a conjectured matrix AM-GM inequality for projectors, sharper rate constants for RP-ADMM remain unresolved, and the exact boundary between stable and unstable randomization schemes is left open (Sun et al., 2015). The ALM-based view of random multi-block ADMM similarly leaves open a broader characterization of when one randomized sweep is sufficient and when multiple inner iterations are necessary (Cipolla et al., 2020). In randomized primal-dual proximal block-coordinate methods, the deterministic cyclic counterpart with comparable guarantees is identified as an open problem (Gao et al., 2016).

This suggests that randomized ADMM is best understood as a design space whose successful instances are governed by operator averaging, spectral regularization, and controllable inexactness, rather than by randomness alone.