Parallel Fixed-Point Methods

• Parallel fixed-point methods are iterative algorithms that seek a point x* satisfying T(x*) = x* by updating multiple components concurrently.
• They employ strategies like Jacobi-type, block, and operator parallelism, with rigorous convergence theory ensuring robust performance in high-dimensional settings.
• Their applications span optimization, machine learning, PDE integration, and equilibrium problems, making them vital for large-scale and distributed computational challenges.

Parallel fixed-point methods are iterative numerical algorithms designed to find a fixed point of a (typically nonlinear) operator in a way that exploits modern parallel and distributed computing architectures. These methods are distinguished by their ability to update multiple components, coordinate blocks, or operator applications simultaneously, with the principal goals of reducing computation time and accommodating large-scale or high-dimensional problems. The development of parallel fixed-point schemes has played a pivotal role in optimization, variational inequalities, equilibrium problems, machine learning, dynamical systems, and scientific computing.

1. Mathematical Foundations and Key Principles

The central objective of parallel fixed-point methods is to find a point $x^*$ such that $T(x^*) = x^*$ for a mapping $T$, frequently under monotonicity, nonexpansiveness, or contractivity assumptions. Parallelization is generally configured in one of several forms:

• Jacobi-type parallelism: All coordinates or blocks are updated simultaneously, often using the previous iterate for all updates.
• Block-parallelism: Disjoint or overlapping blocks of variables or operators are updated in parallel, either cyclically or with randomized selection.
• Operator parallelism: Multiple operators (possibly with different structures, e.g., nonexpansive/quasinonexpansive) are applied in parallel compositions or averages.

Classic parallel fixed-point frameworks are grounded in the theory of averaged, nonexpansive, or contractive operators within Hilbert and Banach spaces. The general Jacobi scheme updates the iterate $x_k$ via:

$x_{k+1} = T(x_k)$

or, for averaged/relaxed operators,

$x_{k+1} = (1-\alpha_k)x_k + \alpha_k T(x_k)$

where $0 < \alpha_k < 1$. Parallelization emerges when $T$ is itself a block or operatorwise mapping applied independently to subsets of the variables or constraints.

2. Algorithmic Structures and Major Paradigms

Parallel fixed-point algorithms are realized in a variety of algorithmic structures, many with rigorous convergence properties:

• Parallel Jacobi-Type Methods: Simultaneously update all coordinates or all operator components. For example, in regularized kernel methods with loss function $\ell(y, \xi)$ and quadratic RKHS norm penalty, the fixed-point iteration

$c_{k+1} = -J_\alpha(a K c_k - c_k)$

has each step decomposed into matrix-vector products (parallelizable linear algebra), vector arithmetic, and per-coordinate resolvent solves, all suitable for parallel execution (Dinuzzo, 2010).

• Parallel Block and Block-Coordinate Iterations: Only a block of variables or operator subproblems is updated at a time, often scheduled according to a rule that ensures all variables/blocks are regularly refreshed. Block updates can be synchronous (all blocks per step) or asynchronous (different blocks updated per iteration, possibly with delays), and are often analyzed using "concentrating arrays" or quasi-Fejér monotonicity arguments (Combettes et al., 2020).
• Parallel String-Averaging and Operator Averaging Methods: Operator strings (ordered lists of operator compositions) are formed, and their application outcomes are averaged. String-averaging schemes like

$$x^{k+1} = \lambda_k u + (1 - \lambda_k) \sum_{t \in \Omega} w(t) T[t](x^k)$$

permit "intrinsic parallelism" in evaluating multiple operator strings and aggregating their results (Censor et al., 2021).

• Hybrid Parallel Schemes: In composite problems (e.g., seeking a common solution to variational inequalities, equilibrium problems, and fixed-point inclusions), parallelization incorporates solution steps for all components—such as projection-type steps for VIs, application of fixed-point mappings, and equilibrium resolvents—all computed in parallel and combined via selection (e.g., farthest element) or convex aggregation before a projection or correction step (Anh et al., 2015, Hieu, 2015, Hieu, 2016).
• Asynchronous and Inertial Accelerated Methods: Asynchronous parallel methods remove the need for global synchronization, allowing updates to be computed with delayed (stale) information (Hannah et al., 2016, Stathopoulos et al., 2017). Inertial (momentum) acceleration can be incorporated directly in the fixed-point iteration, further enhancing convergence speed, provided suitable assumptions (such as strong monotonicity or cocoercivity) are satisfied (Stathopoulos et al., 2017).

3. Convergence Theory and Analytical Guarantees

The convergence of parallel fixed-point schemes depends on various operator-theoretic properties:

• For (firmly) nonexpansive or contractive operators in Hilbert spaces, parallel Jacobi-type iterations exhibit global convergence under mild technical conditions.
• For regularized kernel methods, convergence is ensured if the relaxation parameter $a$ satisfies $0 \le a \cdot \lambda_i < 2$ for all eigenvalues $\lambda_i$ of the kernel matrix $K$. Under additional strict positive-definiteness or smoothness (e.g., differentiable loss with Lipschitz gradient), linear convergence rates can be proven (Dinuzzo, 2010).
• In equilibrium and variational inequality problems, the Lyapunov functional approach, projection properties, and demiclosedness/demicontractivity conditions are employed to establish strong or even linear convergence (Anh et al., 2015, Hieu, 2016).
• Asynchronous methods (e.g., ARock) employ Lyapunov functions that account for asynchronous errors to prove norm convergence, even under unbounded (stochastic or deterministic) delays, provided adaptive step-size selection tied to the delay distribution or recent delays (Hannah et al., 2016).
• Block-update schemes in composite models achieve convergence by ensuring that every block is activated regularly and that errors induced by stale or recycled operator evaluations diminish under a concentrating array framework (Combettes et al., 2020).

4. Implementation, Scalability, and Numerical Performance

Parallel fixed-point methods are expressly designed to be compatible with large-scale or distributed environments:

• Vectorized and Componentwise Operations: Steps such as matrix-vector multiplications and application of additively separable loss or operator terms naturally map to parallel processors or distributed architectures (Dinuzzo, 2010).
• Decentralized or Networked Computations: In network optimization problems, each agent or processor may handle its own local update, sharing iterates as needed; the parallel subgradient method (Iiduka, 2015) typifies this, balancing local autonomy and communication overhead.
• Block Splitting and Composite Operator Handling: Only a subset of expensive operators is refreshed at each iteration, with the rest recycled, reducing computational cost and enabling block-level parallelism (Combettes et al., 2020).
• Parallel-In-Time and Double Parallelism: For time-dependent PDEs, approaches such as α-circulant preconditioned Richardson iterations utilize FFT-based diagonalization and block partitioning in space and time, enabling efficient MPI-based execution and strong (order-100×) speedup relative to sequential time stepping (Caklovic et al., 2021).
• Low-Synchronization Acceleration: Accelerated fixed-point solvers (e.g., Anderson Acceleration) can become communication-bottlenecked on large clusters. Modern low-synchronization orthogonalization, such as ICWY–MGS and DCGS–2, reduces the number of required reductions per iteration to a constant, crucial for scalability on large CPU/GPU systems and achieving strong scaling (Lockhart et al., 2021).

A summary table of typical parallelization scopes in selected methods:

Method Type	Parallelism Scope	Notable Application
Regularized kernel FP iteration (Dinuzzo, 2010)	Matrix-vector, resolvent elements	Large-scale SVM, RLS, SVR
Parallel hybrid VI/EP/FP schemes (Anh et al., 2015)	Solution of multiple VIs, EPs	Common solution of VIs, EPs, and fixed-point sets
Block-update composite FP (Combettes et al., 2020)	Operator-blocks	Nonsmooth minimization, monotone inclusions
Parallel-in-time collocation (Caklovic et al., 2021)	Time steps, collocation nodes	All-at-once ODE/PDE solvers
Asynchronous ARock (Hannah et al., 2016)	Coordinates without synchronization	Distributed ML/optimization
Anderson Acceleration (parallel AA) (Saad, 15 Jul 2025)	QR, vector history, residuals	CFD, DFT, iterative solvers
Low-sync AA orthogonalization (Lockhart et al., 2021)	Gram–Schmidt/QR in AA	Large-scale, multi-GPU AA solvers

5. Applications and Representative Problem Domains

Parallel fixed-point methods have proven effective in varied domains:

• Kernel-based machine learning: Training of SVMs, RLS, SVR; resolving large-scale regularized problems with convex loss (Dinuzzo, 2010).
• Nonsmooth and composite convex optimization: Minimization in networks of users with complex constraints expressed via fixed-point sets of quasinonexpansive mappings (Iiduka, 2015).
• Variational inequalities and equilibrium problems: Simultaneously targeting multiple VIs, equilibria, and fixed points in Banach/Hilbert spaces (Anh et al., 2015, Hieu, 2015, Hieu, 2016, Hieu, 2016).
• Time-parallel PDE integration: All-at-once, parallel-in-time methods for PDE discretizations using collocation and circulant preconditioning (Caklovic et al., 2021).
• Power system stability: Parallel fixed-point eigensolvers for dominant poles in large-scale stability analysis (Bezerra, 2016).
• Boolean networks and discrete dynamical systems: Block-parallel update algorithms create exponentially many fixed points, underscoring the role of update scheduling in dynamical complexity (Perrot et al., 21 May 2025).
• Reinforcement learning and Markov decision processes: Fixed-point approaches (e.g., Halpern iteration with parallelizable BeLLMan operator applications) obtain optimal convergence rates for multichain MDPs (Zurek et al., 26 Jun 2025).

6. Comparative Analysis with Sequential and Coordinate Methods

Parallel fixed-point iterations exhibit inherently different performance characteristics compared to:

• Coordinate Descent (Gauss–Seidel variants): Sequential coordinate updates exploit most recent information and can converge rapidly for separable problems, but suffer from limited parallel scalability due to update dependencies (Dinuzzo, 2010).
• Sequential String/Block Procedures: While capable for moderate scale, sequential approaches are outperformed by parallel or block-update versions, especially in contexts with many operators or constraints (Anh et al., 2015, Combettes et al., 2020).
• Asynchronous vs. Synchronous Schemes: Asynchronous methods tolerate communication delays and processor heterogeneity, with convergence guarantees sustained via adapted step sizes and Lyapunov-based error analysis (Hannah et al., 2016, Stathopoulos et al., 2017).

Potential trade-offs include:

• Greater communication or synchronization overhead in broadcast-averaging or QR-based steps as the network size grows (Iiduka, 2015, Saad, 15 Jul 2025).
• Careful design of block-activation rules to balance load and ensure convergence in composite operator scenarios (Combettes et al., 2020).
• Possible requirement of additional error-tracking or parameter tuning to safeguard robustness and stability in large-scale or ill-conditioned settings (Lockhart et al., 2021).

7. Future Directions and Open Challenges

Ongoing research explores:

• Further reduction of communication in distributed environments, e.g., reducing synchronization in acceleration/orthogonalization or enlarging block sizes judiciously without harming convergence (Lockhart et al., 2021).
• Generalization to time-varying, random, or adversarial block activation, including adaptive and stochastic block selection in composite or inconsistent feasibility settings (Combettes et al., 2020).
• Extension to broader operator classes: Recent studies investigate quasinonexpansive mappings and hybrid parallel strategies in more general Banach space settings (Aoyama et al., 5 Sep 2024).
• Parallel fixed-point algorithms for emerging hardware, such as multi-GPU or neuromorphic systems, with new communication models and latency considerations (Lockhart et al., 2021).
• Applications in high-dimensional learning, scientific machine learning, and large-scale optimization, further leveraging the "embarrassingly parallel" nature of many operator applications and vector calculus steps (Jung, 2017, Zurek et al., 26 Jun 2025).

A plausible implication is that as parallel fixed-point methodology is further refined—balancing operator structure, parallel hardware, relaxation/adaptation strategies, and acceleration—the achievable scales and convergence efficiency of fixed-point based solvers will expand to new domains with previously prohibitive computational costs.