Stochastic Krasnoselskii-Mann Iterations: Convergence without Uniformly Bounded Variance

Abstract: We investigate the Stochastic Krasnoselskii-Mann iterations for expected nonexpansive fixed-point problems in a real Hilbert space. We establish convergence guarantees under significantly weaker assumptions on the variance than those typically used in the literature. In particular, instead of a uniform bound on the variance of the stochastic oracle, we only assume finite variance at a single fixed point. Under this assumption, we prove almost sure weak convergence of the iterates, derive convergence rates for the expected residual, and obtain almost sure convergence rates for the running minimum residual. Notably, we recover the best-known stochastic oracle complexity without imposing uniformly bounded variance. We illustrate the applicability of our results to Stochastic Gradient Descent, where we recover known guarantees, and to Stochastic Three-Operator Splitting, for which we obtain the first results that avoid uniform variance bounds.

• The paper introduces a variance-at-solution assumption, showing convergence of stochastic KM iterations even when variance is unbounded elsewhere.
• It employs nonexpansiveness and Lipschitz continuity to control variance propagation and recovers O(ε⁻⁴) oracle complexity for residual reduction.
• The results extend to SGD and stochastic three-operator splitting, enhancing theoretical guarantees in high-dimensional optimization.

Stochastic Krasnoselskii-Mann Iterations Under Weak Variance Assumptions

Introduction

The paper "Stochastic Krasnoselskii-Mann Iterations: Convergence without Uniformly Bounded Variance" (2604.22581) addresses the convergence properties of stochastic fixed-point algorithms under relaxed variance conditions. The Krasnoselskii-Mann (KM) iteration is a foundational fixed-point method for nonexpansive operators in Hilbert spaces, widely used in optimization and monotone operator theory. Traditional analyses of stochastic variants of KM iterations assume uniformly bounded variance of the stochastic oracle, a restriction that is often unjustified in high-dimensional statistical learning and variational analysis. The work in this paper notably weakens this assumption by requiring bounded variance only at a fixed point, thus significantly broadening the range of applicability of the convergence theory for stochastic fixed-point iterations.

Technical Contributions

The principal innovation is the introduction and exploitation of a variance-at-solution assumption: rather than requiring $$\sup_{x \in \mathcal H} \mathbb{E}\|T_\xi x - T x\|^2 < +\infty$$, it suffices to assume that there exists $p \in \mathrm{Fix}(T)$ such that $\mathbb{E}\|T_\xi p - T p\|^2$ is finite.

Key components of the analysis include:

• Propagation of Variance Bound: The nonexpansiveness of $T_\xi$ ensures that bounded variance at one fixed point implies bounded variance at all fixed points, a result formalized in Lemma 2.1.
• Variance Transfer: The variance at arbitrary points can be controlled by the variance at fixed points plus an explicit dependence on the distance to the fixed point, leveraging $2$-Lipschitz continuity of $T_\xi - T$.
• Convergence Analysis: Almost sure weak convergence of the iterates and convergence of the residuals is established, utilizing a combination of supermartingale techniques and Robbins-Siegmund-type arguments—without the need for uniform variance bounds.
• Oracle Complexity and Rates: The analysis recovers the $\mathcal{O}(\varepsilon^{-4})$ stochastic oracle complexity for expected residuals, matching best-known rates previously derived only under the uniform variance hypothesis and extending them to almost sure running minimum residuals.

Main Results

The primary theoretical contributions are as follows:

1. Variance-at-Solution Assumption:
   • The variance need only be bounded at a solution $p \in \mathrm{Fix}(T)$. This leads to global variance control via the nonexpansiveness property.
2. Convergence Guarantees:
   • The stochastic KM iterates converge weakly almost surely (a.s.) to a fixed point of $T$.
   • The expected fixed-point residual at a randomly selected iterate converges to zero at rate $O(1/\sqrt{K})$ under usual decaying step sizes and at $p \in \mathrm{Fix}(T)$0 with horizon-dependent constant step sizes (matching [bravo_stochastic_2024], [iiduka_minibatch_2026], but under weaker assumptions).
   • The almost sure convergence of the running minimum residual is also quantified.
3. Comparison with Literature:
   • Prior works such as [bravo_stochastic_2024] and [iiduka_minibatch_2026] imposed uniformly bounded variance and achieved $p \in \mathrm{Fix}(T)$1 complexity only for special metrics (e.g., random or running minimum residuals), and did not establish strong/weak convergence of the iterates under local variance assumptions.
4. Instantiations:
   • Stochastic Gradient Descent (SGD): The framework recovers known SGD convergence rates without uniform variance, with $p \in \mathrm{Fix}(T)$2 expected gradient-norm rate under step-size $p \in \mathrm{Fix}(T)$3, consistent with minimax information-theoretic lower bounds for non-strongly convex objectives.
   • Stochastic Three-Operator Splitting: This paper provides, for the first time, convergence guarantees for stochastic three-operator splitting methods without requiring uniformly bounded variance, covering settings previously inaccessible by fixed-point theory.

Practical and Theoretical Implications

The central implication is that the uniformly bounded variance assumption is unnecessarily strict for the convergence of stochastic fixed-point algorithms when the operator is nonexpansive. In practical stochastic optimization and variational settings—where only variance at the solution is often justified or accessible—guarantees at the same oracle complexity and with the same qualitative convergence properties remain valid. This advances the theory of stochastic methods in both convex and monotone settings (and by extension, convex-concave saddle-point and variational inequality problems), and brings theoretical analysis closer to typical statistical learning practice, where variance can be unbounded away from the solution.

Moreover, the methodology extends to settings involving multiple non-smooth and stochastic operators, such as stochastic three-operator splitting, which has not previously been covered without stringent stochastic assumptions.

On the theoretical side, this work clarifies the robustness of stochastic fixed-point methodology to oracle variability—a crucial consideration in large-scale and distributed optimization, Monte Carlo algorithms, and learning under heavy-tailed or non-i.i.d. noise.

Future Directions

The analysis opens several avenues:

• Inertial and Accelerated Methods: The current requirement of diminishing relaxation/stepsize parameters to control stochasticity may be incompatible with inertial (momentum-based) schemes, which often require step-sizes bounded away from zero for acceleration. Adapting the present variance-at-solution analysis to inertial frameworks is a natural area for further research [cortild_krasnoselskii_2025, maulen_inertial_2024].
• Non-Euclidean and Banach Spaces: Extending the analysis to stochastic fixed-point problems in Banach spaces or for Bregman/nonexpansive operators beyond the Hilbert framework.
• Variance Reduction: Developing adaptive or variance-reduced schemes compatible with local variance conditions.
• Differential Privacy and Robustness: Since privacy mechanisms (e.g., Gaussian mechanisms) or adversarial noise may only satisfy local variance-type constraints, the framework can directly inform privacy-preserving or robust algorithmic design.

Conclusion

The paper establishes that stochastic KM iterations converge under variance bounded only at the solution with no deterioration in oracle complexity rates or qualitative convergence guarantees, as compared to uniform variance settings (2604.22581). This significantly enlarges the class of problems—to include those encountered in large-scale learning and Monte Carlo computation—for which rigorous convergence of stochastic fixed-point methods is available. The results are directly transferable to classical and modern stochastic optimization frameworks, as well as to operator-splitting methods involving stochastic components. Future work will further develop inertial and non-Euclidean extensions under these relaxed stochastic conditions.