Stepwise Relaxation Algorithm

• Stepwise Relaxation (SR) is an iterative procedure that applies relaxation in successive steps to gradually approach the solution of linear systems.
• It encompasses variants such as fixed, adaptive (JBUA), and scheduled (SRJ) methods, optimizing relaxation factors based on spectral properties.
• SR techniques also inspire methods in Gaussian graphical models and oscillator systems, enabling structured model refinement and controlled energy dissipation.

Stepwise Relaxation Algorithm (SR) denotes, in the cited literature, a class of iterative procedures in which relaxation is applied step by step rather than through a single stationary update. In numerical linear algebra, SR most directly refers to Successive (or Stepwise) Relaxation for solving $Ax=b$, either with a fixed relaxation factor $\omega$, with self-adapted $\omega_i$ carried by an evolutionary population, or with a scheduled cycle $(\omega_1,\dots,\omega_M)$ of Jacobi weights. In adjacent literatures, related stepwise procedures include forward–backward graph construction in Gaussian graphical models and coarse-grained switching between trapped and high-dissipation regimes in dispersively coupled oscillator–particle systems (Jamali et al., 2013, Islam et al., 2021, Lafit et al., 2018, Rhen et al., 2017).

1. Terminological scope and core abstraction

In iterative numerical linear algebra, “SR” stands for Successive (or Stepwise) Relaxation. The basic idea is to replace a full Jacobi update by a relaxed step that moves only a fraction of the way from the current iterate toward the Jacobi iterate. For $A=D+L+U$, with $D$ diagonal, $L$ strictly lower triangular, and $U$ strictly upper triangular, the Jacobi fixed-point form is

$$x = Hx + V,\qquad H = D^{-1}(-L-U),\qquad V = D^{-1}b.$$

The plain Jacobi iteration is

$x^{(k+1)} = Hx^{(k)} + V,$

and the relaxed step is

$\omega$0

Componentwise, the Jacobi-SR update is

$\omega$1

The cases $\omega$2, $\omega$3, and $\omega$4 correspond respectively to Jacobi, under-relaxation, and over-relaxation (Jamali et al., 2013).

The Scheduled Relaxation Jacobi (SRJ) method is a non-stationary extension of the same principle. Instead of using one fixed $\omega$5, SRJ applies a prescribed sequence of relaxation factors $\omega$6 within a cycle and then repeats that cycle. The paper itself uses the name SRJ rather than “SR”, but conceptually SRJ is a stepwise relaxation algorithm: each iteration uses a possibly different relaxation factor $\omega$7, according to a precomputed schedule (Islam et al., 2021).

A broader use of “stepwise relaxation” appears outside classical solvers. In high-dimensional Gaussian graphical models, the graphical stepwise algorithm starts from an empty graph and iteratively adds or removes edges based on residual correlations; the paper does not explicitly use the label “Stepwise Relaxation (SR)”, but states that the method fits naturally the idea of a stepwise relaxation of the structural zeros in $\omega$8 (Lafit et al., 2018). In nonlinear oscillator–particle systems, stepwise relaxation refers instead to a physical relaxation pattern with alternating plateaus and rapid drops in energy dissipation (Rhen et al., 2017).

2. Classical SR for linear systems and its spectral properties

For the stationary Jacobi-SR method, the relaxed iteration can be written in matrix form as

$\omega$9

with

$\omega_i$0

Here $\omega_i$1 is the Jacobi-SR iteration matrix. The paper constrains $\omega_i$2 to a domain

$\omega_i$3

because a necessary condition for convergence of SR is $\omega_i$4. More generally, convergence of the fixed-point iteration requires

$\omega_i$5

so the asymptotic rate is governed by the spectral radius of $\omega_i$6 (Jamali et al., 2013).

The central practical issue is sensitivity to $\omega_i$7. The paper assumes an optimal relaxation factor $\omega_i$8 such that $\omega_i$9 is strictly decreasing for $(\omega_1,\dots,\omega_M)$0 and strictly increasing for $(\omega_1,\dots,\omega_M)$1. Under that assumption, SR has a single best relaxation factor but is highly problem-dependent. The experimental examples make this explicit. For the system

$(\omega_1,\dots,\omega_M)$2

the authors note that the optimal relaxation is near to $(\omega_1,\dots,\omega_M)$3, while classical Jacobi-SR with $(\omega_1,\dots,\omega_M)$4 or $(\omega_1,\dots,\omega_M)$5 failed to achieve error below $(\omega_1,\dots,\omega_M)$6 in 1000 iterations. For a second problem with

$(\omega_1,\dots,\omega_M)$7

Jacobi-SR converges in 18 iterations for $(\omega_1,\dots,\omega_M)$8, whereas many nearby choices diverge. The numerical message is that slight changes around $(\omega_1,\dots,\omega_M)$9 can shift the method from rapid convergence to divergence (Jamali et al., 2013).

This sensitivity is the main reason SR becomes a design problem rather than a mere scalar tuning problem. A fixed-$A=D+L+U$0 scheme is simple and fully parallelizable in the Jacobi case, but robust performance depends on spectral information that is often unavailable or expensive to estimate directly.

3. Adaptive and scheduled Jacobi variants

Two distinct stepwise generalizations of Jacobi-SR have been studied. The first is the Jacobi-SR Based Uniform Adaptive hybrid evolutionary algorithm (JBUA). Each individual $A=D+L+U$1 carries its own relaxation factor $A=D+L+U$2, and the population is

$A=D+L+U$3

Recombination uses a stochastic mixing matrix $A=D+L+U$4 to form

$A=D+L+U$5

Mutation is one Jacobi-SR step,

$A=D+L+U$6

Fitness is measured by $A=D+L+U$7. The uniform adaptation rule compares adjacent ranked individuals: the worse $A=D+L+U$8 is moved toward the better $A=D+L+U$9 through

$D$0

while the better factor is moved away from the worse one within $D$1 using $D$2. Selection keeps the best $D$3 individuals and duplicates them to form the next generation. A convergence theorem states that if there exists $D$4 such that $D$5 for all relevant $D$6, then every individual converges to the exact solution $D$7. An adaptation theorem further shows, under monotonicity assumptions on $D$8, that the update rules move relaxation factors toward lower spectral radius (Jamali et al., 2013).

The second generalization is SRJ, where stepwise relaxation is scheduled rather than learned online. For the Jacobi iteration matrix $D$9, one relaxed step uses

$L$0

A cycle of length $L$1 has iteration matrix

$L$2

and amplification polynomial

$L$3

The nonsymmetric extension constructs $L$4 by minimizing the maximum of $L$5 over an ellipse

$L$6

with

$L$7

Numerically, the scheme is obtained from the constrained problem

$L$8

subject to

$L$9

at test points on the ellipse boundary. The implementation described in the paper uses scipy.optimize.minimize with the Trust-Region Constrained algorithm. In 1D steady advection–diffusion with $U$0, $U$1, and $U$2, the $U$3 scheme offers almost no benefit compared to Jacobi, whereas the $U$4 scheme is significantly faster. In 2D, for $U$5 grids with $U$6, the $U$7 scheme is the best and robust, while $U$8 typically diverge then stagnate (Islam et al., 2021).

Variant	Stepwise mechanism	Design principle
JBUA	One Jacobi-SR step per generation with evolving $U$9	Uniform self-adaptation guided by $$x = Hx + V,\qquad H = D^{-1}(-L-U),\qquad V = D^{-1}b.$$0
SRJ	One cycle of $$x = Hx + V,\qquad H = D^{-1}(-L-U),\qquad V = D^{-1}b.$$1 relaxed Jacobi steps with $$x = Hx + V,\qquad H = D^{-1}(-L-U),\qquad V = D^{-1}b.$$2	Minimize $$x = Hx + V,\qquad H = D^{-1}(-L-U),\qquad V = D^{-1}b.$$3

A major structural distinction concerns parallelism. Because Jacobi-SR uses only the old iterate $$x = Hx + V,\qquad H = D^{-1}(-L-U),\qquad V = D^{-1}b.$$4, all components $$x = Hx + V,\qquad H = D^{-1}(-L-U),\qquad V = D^{-1}b.$$5 can be computed independently and simultaneously. The JBUA paper therefore argues that Jacobi-SR based hybrids inherently can be implemented in parallel processing environments efficiently, whereas Gauss-Seidel-SR based hybrids cannot (Jamali et al., 2013).

4. Stepwise relaxation of graph structure in Gaussian graphical models

In Gaussian graphical models, the stepwise procedure is not a relaxation parameter method but a forward–backward algorithm on graph support. Let $$x = Hx + V,\qquad H = D^{-1}(-L-U),\qquad V = D^{-1}b.$$6, with precision matrix $$x = Hx + V,\qquad H = D^{-1}(-L-U),\qquad V = D^{-1}b.$$7. Conditional independence is equivalent to $$x = Hx + V,\qquad H = D^{-1}(-L-U),\qquad V = D^{-1}b.$$8, and the partial correlation satisfies

$$x = Hx + V,\qquad H = D^{-1}(-L-U),\qquad V = D^{-1}b.$$9

The paper’s key parametrization expresses edges through Pearson correlations between prediction errors of nodewise best linear predictors. If

$x^{(k+1)} = Hx^{(k)} + V,$0

then

$x^{(k+1)} = Hx^{(k)} + V,$1

The graphical stepwise algorithm starts from empty neighborhoods $x^{(k+1)} = Hx^{(k)} + V,$2. In the forward step it computes empirical residual correlations

$x^{(k+1)} = Hx^{(k)} + V,$3

for non-neighbors and adds the edge with largest $x^{(k+1)} = Hx^{(k)} + V,$4 if it exceeds $x^{(k+1)} = Hx^{(k)} + V,$5. In the backward step it computes

$x^{(k+1)} = Hx^{(k)} + V,$6

for existing edges after leaving that edge out of the conditioning sets, and removes the edge with smallest $x^{(k+1)} = Hx^{(k)} + V,$7 if it is below $x^{(k+1)} = Hx^{(k)} + V,$8. The final precision estimate uses residual variances and covariances: $x^{(k+1)} = Hx^{(k)} + V,$9 for selected edges, and $\omega$00 otherwise. Thresholds $\omega$01 are chosen by $\omega$02-fold cross-validation minimizing

$\omega$03

The paper states that this procedure fits naturally the idea of a “stepwise relaxation”: the support of $\omega$04 starts empty and is gradually relaxed and refined, rather than being determined in one global optimization (Lafit et al., 2018).

The empirical comparisons emphasize graph recovery rather than convex optimality. Across AR(1), NN(2), and BG models with $\omega$05 and $\omega$06, the method substantially outperforms graphical lasso and CLIME in MCC. Representative values include AR(1), $\omega$07: GS 0.751 versus 0.433 for Glasso and 0.464 for CLIME; NN(2), $\omega$08: GS 0.802 versus 0.382 and 0.407; BG, $\omega$09: GS 0.857 versus 0.348 and 0.461. In the breast cancer gene expression example, GS attains MCC 0.520 (0.02), CLIME 0.516 (0.02), and Glasso 0.334 (0.02), with 54 selected edges versus 4823 and 2103. The paper also notes that no formal theorems about model selection consistency or rates of convergence are provided (Lafit et al., 2018).

5. Stepwise relaxation as a dynamical regime in dispersively coupled oscillators

A different usage appears in the study of degenerate harmonic oscillator modes dispersively coupled to particle positions. The system has Hamiltonian

$\omega$10

so the particle experiences an effective inertial potential

$\omega$11

The key observation is that the oscillator energy $\omega$12 does not decay smoothly. Instead, the dynamics shows plateaus, where the particle is trapped in a local minimum of $\omega$13 and dissipation is slow, alternating with rapid drops, where the particle traverses the domain in an irregular fashion and transfers energy efficiently from the oscillators to the dissipative particle bath (Rhen et al., 2017).

For the minimal 1D prototype with a degenerate doublet and one particle, the equations of motion are

$\omega$14

$\omega$15

$\omega$16

During plateau phases, a rotating-wave approximation yields slow amplitude dynamics,

$\omega$17

with slow mixing frequency

$\omega$18

In the high-dissipation regime, the paper reports

$\omega$19

which leads to an effective exponential relaxation picture through particle thermalization (Rhen et al., 2017).

In the membrane model, stepwise relaxation appears only in the presence of thermal noise. Noise enables escape and retrapping, and the degenerate doublet exhibits a slow-in-time stochastic precession of the mixing angle. The power spectral density of the angular velocity has a low-frequency Lorentzian whose width scales as

$\omega$20

The paper does not explicitly propose this as an algorithm, but the logic of its analysis naturally leads to a two-regime SR scheme: micro-integration of the coupled equations, trapping-state detection, regime-dependent dissipation laws, and stochastic evolution of the mixing angle in degenerate noisy plateaus (Rhen et al., 2017).

6. Comparative interpretation and recurrent misconceptions

Several recurrent distinctions are essential for interpreting “SR” correctly. First, stationary SR and non-stationary SR are different objects. A fixed-$\omega$21 Jacobi-SR or Gauss-Seidel-SR iteration modifies each step by the same relaxation factor, whereas SRJ uses a prescribed cycle of factors and JBUA adapts $\omega$22 online. The scheduled version is polynomially accelerated; the adaptive evolutionary version is population-based; the classical version is a single fixed-point iteration (Jamali et al., 2013, Islam et al., 2021).

Second, SRJ is not restricted to elliptic problems. Earlier SRJ work optimized amplification on a real interval and therefore targeted symmetric positive definite elliptic systems, but the nonsymmetric extension replaces the real interval by an ellipse in the complex plane and computes $\omega$23 from a constrained minimization problem. The paper explicitly shows that old Chebyshev-based schemes can have $\omega$24 inside $\omega$25, so SRJ can diverge while Jacobi still converges if the design ignores complex spectra (Islam et al., 2021).

Third, the graphical stepwise procedure is not a one-shot convex estimator of $\omega$26. Its logic is local and structural: add one edge whose residual correlation exceeds $\omega$27, then possibly remove one edge whose residual correlation falls below $\omega$28. This is why the paper contrasts it with graphical lasso and CLIME, which solve global penalized or constrained optimization problems (Lafit et al., 2018).

Fourth, in the oscillator–particle setting, “stepwise relaxation” names a relaxation pattern rather than a numerical solver. Plateaus and drops arise from inertial trapping, release, and retrapping of particles, with stochastic precession as a degenerate-mode effect in the noisy membrane problem (Rhen et al., 2017).

Taken together, these usages suggest that “Stepwise Relaxation Algorithm” is best understood as a procedural pattern rather than a single canonical algorithm. The common structure is step-by-step control of a system that is otherwise hard to optimize or stabilize globally: by adjusting $\omega$29, by scheduling $\omega$30, by relaxing the support of $\omega$31 through forward–backward edge updates, or by switching between coarse-grained dissipation regimes. The precise technical content of SR therefore depends on domain, but its most developed meaning remains the Jacobi-based relaxation family for linear systems and PDE discretizations (Jamali et al., 2013, Islam et al., 2021).