---
title: Randomized ADMM Methods
url: https://www.emergentmind.com/topics/randomized-admm
type: topic
---

# Randomized ADMM Methods

Randomized ADMM denotes a family of modifications of the alternating direction method of multipliers in which randomness enters the block order, block selection, data sampling, sketching operator, or inner linear algebra. The unifying motivation is that the direct multi-block generalization of ADMM is not necessarily convergent, whereas carefully chosen randomization can restore convergence in expectation, almost sure convergence, or ergodic rate guarantees in settings ranging from linear systems and convex quadratic programming to distributed consensus, stochastic learning, PDE-constrained inversion, and large-scale beamforming [1503.06387] [1409.8625] [1303.2837] [2302.03863].

## 1. Problem classes and basic formulation

A canonical multi-block formulation is
\[
\min_{x_1,\dots,x_n}\; f(x_1,\dots,x_n)\quad\text{s.t.}\quad \sum_{i=1}^n A_i x_i=b,
\]
with augmented Lagrangian
\[
\mathcal{L}(x_1,\dots,x_n;\mu)=f(x_1,\dots,x_n)-\mu^\top\Big(\sum_i A_i x_i-b\Big)+\frac{\beta}{2}\Big\|\sum_i A_i x_i-b\Big\|^2.
\]
This model is the point of departure for the random-permutation analysis of multi-block ADMM and its close connection to randomized block coordinate descent [1503.06387].

A second foundational formulation is the bilinear saddle problem
\[
\min_{x\in X}\;\Big\{h(x)+\max_{y\in Y}\;\langle A x,\;y\rangle - J(y)\Big\},
\]
with block-product dual region \(Y=Y_1\times\cdots\times Y_p\). In this setting, randomized primal-dual updates become exact randomized variants of multi-block ADMM for linearly constrained separable problems [1409.8625].

A third major setting is distributed consensus. For networked agents with private costs \(f_v\), the global objective
\[
\inf_{x\in \mathsf X}\;\sum_{v\in V} f_v(x)
\]
can be written as a constrained consensus problem over local copies, and then attacked by asynchronous randomized ADMM through random component activations [1303.2837].

Across these formulations, the same structural difficulty recurs: when the number of blocks exceeds two, a direct cyclic extension of ADMM may diverge. This is explicit in the randomized primal-dual literature and in the random-permutation analysis for multi-block ADMM [1409.8625] [1503.06387].

## 2. Main randomization mechanisms

Randomized ADMM is not a single algorithmic template but a collection of mechanisms that intervene at different levels of the ADMM pipeline.

| Mechanism | Representative update pattern | Representative guarantee or use |
|---|---|---|
| Random permutation | Uniformly sample a permutation and perform one Gauss–Seidel sweep without replacement | Expected convergence for linear systems; spectral tightening to \((-1/3,1)\) for the expected RP-BCD operator [1503.06387] |
| Random block activation | Update one component, edge, node, or block subset at each iteration | Almost sure convergence in asynchronous consensus; ergodic \(O(1/N)\) and \(O(1/N^2)\) rates in randomized primal-dual form [1303.2837] [1409.8625] |
| Random block assembly | Repartition variables into new blocks every epoch, then perform cyclic ADMM | Expected convergence for convex QP; almost sure convergence if \(\rho(\mathbb E[M_\sigma\otimes M_\sigma])<1\) [1903.01786] |
| Stochastic or sketched subproblems | Replace full losses, sources, or Hessians by random samples or sketches | \(O(1/\sqrt{t})\) stochastic rates, at least one order-of-magnitude PDE-solve reduction, and explicit inexact/sketched ADMM rates [1312.4564] [2108.03961] [2302.03863] |

The best-known classical distinction is between sampling without replacement and sampling with replacement. In the multi-block linear-system analysis, random permutation is theoretically and empirically favorable, whereas with-replacement variants such as PD-RADMM and P-RADMM can diverge [1503.06387]. This makes “randomized ADMM” a qualitative design choice rather than merely a stochastic perturbation of a deterministic sweep.

Other mechanisms are more specialized. In stochastic ADMM, one replaces the expected loss by a random loss associated with one uniformly drawn example plus a Bregman divergence, and then chooses the quadratic proximal metric adaptively [1312.4564]. In inverse problems, randomness may enter through a sketching matrix \(S\) that compresses the source dimension while ADMM handles the extended-space constraints [2108.03961]. In privacy-preserving decentralized consensus, randomization can act on the penalty parameter or on the primal variables along a Hamiltonian cycle, producing incremental ADMM variants with privacy guarantees against an external eavesdropper [2003.10615].

## 3. Operator-theoretic and spectral structure

The most detailed spectral analysis is available for randomly permuted ADMM on linear systems. For
\[
\min_x\;0\quad\text{s.t.}\quad Ax=b,
\]
with \(A=[A_1,\dots,A_n]\in\mathbb R^{N\times N}\) nonsingular and \(\beta=1\), the stacked iterate \(y^k=[(x^k)^\top,(\mu^k)^\top]^\top\) satisfies
\[
y^{k+1}=M_\sigma y^k+\bar L_\sigma^{-1}\bar b
\]
for a permutation-dependent linear operator \(M_\sigma\). Averaging over uniform random permutations yields
\[
M=\mathbb E[M_\sigma]=
\begin{bmatrix}
I-QA^\top A & QA^\top\\
-A+AQA^\top A & I-AQA^\top
\end{bmatrix},
\qquad
Q=\frac1{n!}\sum_{\sigma\in\Gamma}L_\sigma^{-1},
\]
with \(Q\) symmetric [1503.06387].

The central spectral statement is
\[
\mathrm{eig}(QA^\top A)\subset(0,\tfrac43),
\]
equivalently
\[
\mathrm{eig}(I-QA^\top A)\subset(-\tfrac13,1).
\]
The same analysis establishes the eigenvalue transform
\[
\lambda\in\mathrm{eig}(M)\quad\Longleftrightarrow\quad
\frac{(1-\lambda)^2}{1-2\lambda}\in\mathrm{eig}(QA^\top A),
\]
so the bound on \(QA^\top A\) yields \(\rho(M)<1\) and hence expected convergence of RP-ADMM for nonsingular linear systems [1503.06387].

A second operator-theoretic strand interprets randomized ADMM as randomized Douglas–Rachford or averaged-operator iterations. In asynchronous distributed optimization, the method is a randomized Gauss–Seidel iteration of a Douglas–Rachford operator on the dual, and the induced block iteration
\[
\zeta^{k+1}=\hat S_{\xi^{k+1}}(\zeta^k)
\]
converges almost surely to a random point in \(\mathrm{Fix}(S)\) under i.i.d. activations with positive probabilities [1303.2837]. Closely related work formulates ADMM+ as an \(\alpha\)-averaged primal-dual operator and then applies randomized Krasnosel’skii–Mann updates to obtain node-asynchronous distributed methods; when \(f\equiv 0\) and \(\tau=\rho\), ADMM+ reduces to classical ADMM [1407.0898].

For randomly assembled cyclic ADMM, the expected iteration matrix retains the same block form as in the random-permutation case, with \(Q=\mathbb E[L_\sigma^{-1}]\) now averaging over both random partitions and update orders. The almost sure convergence criterion
\[
\rho\!\left(\mathbb E[M_\sigma\otimes M_\sigma]\right)<1
\]
controls second moments and separates mean stability from sample-path stability [1903.01786].

## 4. Convergence rates and stability regimes

The guarantees available for randomized ADMM are heterogeneous because the randomization mechanisms solve different analytical problems.

For random permutation on linear systems, expected convergence is explicit. If \(A\) is nonsingular and \(A_i^\top A_i=I\) for all \(i\), then
\[
\| \mathbb{E}[y^k]-y^\ast \|
\le
\left(1-\frac{1}{2n}\lambda_{\min}(AA^\top)\right)^k
\|y^0-y^\ast\|,
\]
with \(y^\ast=[(A^{-1}b)^\top,0^\top]^\top\). In the corresponding quadratic RP-BCD setting,
\[
\| \mathbb{E}[x^k]-x^\ast \|
\le
\max\left\{1-\frac1n\lambda_{\min}(AA^\top),\frac13\right\}^k
\|x^0-x^\ast\|,
\]
and the expected RP-BCD rate is \(O(n)\) times better than the worst-case rate of cyclic BCD [1503.06387].

In randomized primal-dual form, bounded-domain bilinear saddle problems admit ergodic \(O(1/N)\) convergence without strong convexity, and smooth bilinear saddle problems admit \(O(1/N^2)\) rates when \(J\) is strongly convex. The same framework yields randomized single-block ADMM updates for linearly constrained separable problems [1409.8625].

For general multi-block convex optimization with coupled objectives and linear constraints, randomized primal-dual proximal block coordinate updates establish \(O(1/t)\) convergence in expectation for objective suboptimality and feasibility violation under mere convexity, and extend to \(O(1/\sqrt t)\) stochastic rates when only stochastic gradient approximations are available [1605.05969]. At the opposite end of the approximation spectrum, GeNI-ADMM encompasses inexact first- and second-order ADMM schemes and yields the usual \(O(1/t)\) rate under standard hypotheses, together with linear convergence under additional hypotheses such as strong convexity; this explicitly covers NysADMM and sketch-and-solve ADMM [2302.03863].

Application-specific randomized ADMM can preserve these orders. In large-scale max–min beamforming for cell-free massive MIMO, the randomized method updates only a random subset of beamformer blocks each iteration yet retains an \(O(1/\bar t)\) convergence rate, matching the order of its deterministic counterpart [2507.18980].

A persistent theme is the difference between expectation guarantees and stronger notions of robustness. The ALM-based analysis of random multi-block ADMM for strongly convex QP emphasizes that convergence in expectation may not be a good indicator of robustness and efficiency, and shows that one randomized GS or RSSOR sweep is in general not accurate enough to guarantee convergence of the outer method; a constant number of inner CG, SOR, or randomly shuffled SOR iterations suffices to recover almost sure convergence in the inexact ALM sense [2012.09230].

## 5. Representative applications and implementations

Distributed consensus is one of the oldest application areas. Randomized asynchronous ADMM updates only one active network component at a time, requires local averaging within the active component, and converges almost surely under mild connectivity and activation assumptions [1303.2837]. Related node-asynchronous distributed primal-dual schemes update a random subset of agents at each iteration and include ADMM as a special case [1407.0898]. In constrained multi-agent optimization with local polyhedral constraints, randomized proximal dual consensus ADMM handles polyhedra softly through slack variables and proximal terms, is robust against randomly ON/OFF agents and imperfect communication links, and has worst-case \(O(1/k)\) convergence in expectation [1409.3307].

Machine learning has provided a second major testbed. Adaptive stochastic ADMM replaces the full expected loss by a random loss associated with one uniformly drawn example plus a Bregman divergence, and uses diagonal or full-matrix adaptive metrics that yield AdaGrad-type behavior with regret bounds matching the best proximal function chosen in hindsight up to constants [1312.4564]. Randomly assembled cyclic multi-block ADMM has also been applied to Linear Regression, LASSO, Elastic-Net, and SVM, with numerical tests reporting that it could significantly outperform other optimization algorithms or codes on many quadratic machine-learning instances and match the performance of specialized solvers such as Glmnet or LIBSVM [1907.01995].

Inverse problems and scientific computing have motivated sketching-based variants. In frequency-domain full waveform inversion, randomized source sketching projects the source dimension into a smaller domain through a sketching matrix \(S\), and ADMM is then applied in an extended search-space formulation. The reported numerical examples show that the randomized sketching algorithm reduces the cost of large-scale problems by at least one order of magnitude compared to the original deterministic algorithm [2108.03961].

Randomized ADMM has also entered modern wireless optimization. In max–min beamforming for large-scale cell-free massive MIMO, a randomized ADMM is built on a reformulation of the feasibility check as a linearly constrained optimization problem; it updates only a small number of subproblems at each iteration, offers a significant complexity advantage over existing methods, and preserves the deterministic \(O(1/\bar t)\) rate order [2507.18980].

A different application thread concerns privacy-preserving decentralized consensus. Incremental ADMM along a Hamiltonian cycle is communication efficient but does not guarantee privacy against an external eavesdropper. Two privacy-preserving variants, PI-ADMM1 and PI-ADMM2, randomize either the step sizes or the primal variables; PI-ADMM1 is proved to preserve privacy and converge, and both algorithms are reported as communication efficient compared with state-of-the-art methods [2003.10615].

## 6. Limitations, controversies, and derandomized alternatives

The main controversy surrounding randomized ADMM is not whether randomization can help, but which randomization is structurally compatible with ADMM. The random-permutation analysis shows that sampling without replacement is qualitatively different from with-replacement schemes: RP-ADMM converges in expectation for nonsingular linear systems, while PD-RADMM and P-RADMM can diverge, and cyclic ADMM can diverge even when every random-permutation average is contractive [1503.06387].

Random block assembly adds another layer of subtlety. RAC-ADMM converges in expectation for convex QP under the stated blockwise positive-definiteness condition, yet almost sure convergence requires the stronger criterion
\[
\rho\!\left(\mathbb E[M_\sigma\otimes M_\sigma]\right)<1.
\]
The existence of examples with \(\rho(\mathbb E[M])<1\) but \(\rho(\mathbb E[M_\sigma\otimes M_\sigma])>1\) shows that mean stability and sample-path stability need not coincide [1903.01786].

Approximate and sketch-based variants have analogous failure modes. GeNI-ADMM proves \(O(1/t)\) and linear convergence for NysADMM and sketch-and-solve ADMM only when the inexactness schedules are summable or geometrically decaying and, for the sketched Hessian case, when a correction term such as \(\gamma^k\ge \|E^k\|\) restores the relative smoothness condition. The experiments reported there show that omitting the correction term can lead to divergence or oscillation [2302.03863].

These limitations have motivated deterministic surrogates for randomization. A derandomized algorithm based on a block symmetric Gauss–Seidel sweep and a Richardson-type dual correction replaces random permutation by a forward–backward SGS pass and updates the dual by
\[
\lambda^{k+1}=\lambda^k+\omega(Ax^{k+1}-b),\qquad 0<\omega<2\beta.
\]
For linear systems, the paper proves linear convergence under this scheme and presents it as a deterministic alternative to RP-ADMM [1705.08389].

Several open questions remain explicit in the literature. The random-permutation analysis introduces a conjectured matrix AM-GM inequality for projectors, sharper rate constants for RP-ADMM remain unresolved, and the exact boundary between stable and unstable randomization schemes is left open [1503.06387]. The ALM-based view of random multi-block ADMM similarly leaves open a broader characterization of when one randomized sweep is sufficient and when multiple inner iterations are necessary [2012.09230]. In randomized primal-dual proximal block-coordinate methods, the deterministic cyclic counterpart with comparable guarantees is identified as an open problem [1605.05969].

This suggests that randomized ADMM is best understood as a design space whose successful instances are governed by operator averaging, spectral regularization, and controllable inexactness, rather than by randomness alone.

Source: https://www.emergentmind.com/topics/randomized-admm