Robbins–Siegmund Theorem Overview

• Robbins–Siegmund theorem is a supermartingale convergence principle for nonnegative adapted processes, ensuring almost sure convergence under summability conditions.
• The theorem underpins stochastic approximation methods by transforming one-step Lyapunov recursions into forms applicable to Q-learning, TD learning, and other algorithms.
• Recent extensions relax summable additive noise with square-summable and metastable conditions, yielding explicit rates and convergence to bounded sets.

The Robbins–Siegmund theorem is a supermartingale convergence principle for adapted nonnegative stochastic processes. In its classical form, it treats recursions of the type

$E[X_{n+1}\mid \mathcal F_n]\le (1+A_n)X_n-B_n+C_n$

and concludes, under summability assumptions on the perturbation terms, that $X_n$ converges almost surely to a finite random limit and that the cumulative descent term $\sum_n B_n$ is finite almost surely. Recent work has developed three major directions around this theorem: quantitative finitary versions based on metastability, relaxed quantitative variants with explicit mean and almost-sure rates under structured regularity hypotheses, and extensions that replace summable additive noise by square-summable control plus an increment condition. The theorem has also become a central abstraction for formal verification of stochastic approximation proofs, including $Q$-learning and linear temporal-difference learning (Neri et al., 2024, Neri et al., 17 Apr 2025, Zhang, 5 Nov 2025, Liu et al., 30 Sep 2025).

1. Classical formulation and canonical special cases

A standard recent statement of the classical theorem considers nonnegative integrable processes $(X_n),(A_n),(B_n),(C_n)$ adapted to a filtration $(\mathcal F_n)$ and satisfying

$E[X_{n+1}\mid \mathcal F_n]\le (1+A_n)X_n-B_n+C_n$

almost surely for all $n$. If

$$\sum_{n=0}^\infty A_n<\infty,\qquad \sum_{n=0}^\infty C_n<\infty$$

almost surely, then almost surely $X_n\to X_\infty$ for some finite random variable $X_n$0, and

$X_n$1

Recent quantitative and application-oriented papers present this as the basic “almost supermartingale” convergence lemma underlying stochastic approximation, stochastic optimization, Robbins–Monro-type schemes, stochastic gradient methods, and reinforcement learning (Neri et al., 2024).

A particularly important special case arises when one has deterministic stepsizes $X_n$2 satisfying the Robbins–Monro conditions

$X_n$3

and a scalar nonnegative adapted process $X_n$4 obeying

$X_n$5

In that case the conclusion simplifies to

$X_n$6

This special form is exactly the version formalized in Lean 4 as ae_tendsto_zero_of_almost_supermartingale and used as the terminal convergence step for both $X_n$7-learning and linear TD learning (Zhang, 5 Nov 2025).

Conceptually, $X_n$8 controls multiplicative inflation, $X_n$9 is an additive disturbance, and $\sum_n B_n$0 is the dissipative term. The theorem asserts that summable multiplicative and additive errors do not destroy convergence, while the negative drift accumulates only finitely often.

2. Supermartingale mechanism and proof architecture

The standard proof strategy normalizes the recursion to isolate a genuine supermartingale. In the quantitative treatment, one introduces

$\sum_n B_n$1

together with correspondingly normalized descent and error terms, and then defines

$\sum_n B_n$2

The process $\sum_n B_n$3 is a supermartingale, so one can combine supermartingale convergence with boundedness estimates such as Ville’s inequality to recover convergence of the original process and summability of the drift term (Neri et al., 2024).

This normalization argument is the stochastic counterpart of deterministic “almost-monotone sequence” arguments. It explains why the theorem is so reusable: many stochastic algorithms first produce a one-step Lyapunov inequality, then rewrite that inequality into an almost-supermartingale form, and finally invoke Robbins–Siegmund as a black-box convergence engine.

A related but more flexible construction appears in the relaxed quantitative theorem of 2025. There, the key recursion is

$\sum_n B_n$4

and the proof builds a nonnegative supermartingale from $\sum_n B_n$5, the product terms $\sum_n B_n$6, and a conditional tail term involving $\sum_n B_n$7. Jensen’s inequality is then applied to a slowdown observable $\sum_n B_n$8, and Ville’s inequality yields almost-sure control. The main structural point is that the theorem no longer requires the descent term to be a direct metric on the target variable; it is enough that $\sum_n B_n$9 control the target through a regularity modulus (Neri et al., 17 Apr 2025).

3. Quantitative reformulations

Recent work distinguishes two qualitatively different quantitative directions. One retains the classical hypothesis structure and extracts finitary convergence information in the sense of Tao’s metastability. The other strengthens the hypotheses by adding regularity data and thereby obtains direct explicit rates.

A central observation is that a general computable direct modulus of convergence cannot be expected from the classical Robbins–Siegmund assumptions alone. The obstruction already appears for deterministic monotone bounded sequences, via Specker-type phenomena. For that reason, the 2024 quantitative theory replaces direct asymptotic rates by metastable bounds of the form

$Q$0

such that

$Q$1

In that framework, the classical theorem is “finitized” by combining a metastable version of Doob’s theorem for $Q$2-supermartingales with quantitative closure lemmas for sums and products (Neri et al., 2024).

By contrast, the 2025 relaxed supermartingale theorem introduces additional quantitative data: a rate of divergence for $Q$3, a tail modulus for $Q$4, and a regularity implication linking $Q$5 to the target quantity through a slowdown function $Q$6. The function $Q$7 must be s.i.c.c., meaning strictly increasing, concave, continuous, and $Q$8-supermultiplicative, with moduli $Q$9 and $(X_n),(A_n),(B_n),(C_n)$0; examples include $(X_n),(A_n),(B_n),(C_n)$1 for $(X_n),(A_n),(B_n),(C_n)$2 and $(X_n),(A_n),(B_n),(C_n)$3. Under these hypotheses one obtains both

$(X_n),(A_n),(B_n),(C_n)$4

with explicit rate

$(X_n),(A_n),(B_n),(C_n)$5

and almost sure convergence

$(X_n),(A_n),(B_n),(C_n)$6

with rate

$(X_n),(A_n),(B_n),(C_n)$7

The same work records the qualitative corollary that if $(X_n),(A_n),(B_n),(C_n)$8, $(X_n),(A_n),(B_n),(C_n)$9, and $(\mathcal F_n)$0, then $(\mathcal F_n)$1 and $(\mathcal F_n)$2 almost surely (Neri et al., 17 Apr 2025).

Variant	Core hypothesis pattern	Quantitative output
Classical Robbins–Siegmund	$(\mathcal F_n)$3, with $(\mathcal F_n)$4 a.s.	a.s. convergence of $(\mathcal F_n)$5; $(\mathcal F_n)$6
Metastable quantitative form	Classical recursion plus probabilistic bounds on $(\mathcal F_n)$7 and $(\mathcal F_n)$8	learnable and metastable bounds (Neri et al., 2024)
Relaxed effective form	$(\mathcal F_n)$9 plus $E[X_{n+1}\mid \mathcal F_n]\le (1+A_n)X_n-B_n+C_n$0	explicit rates for $E[X_{n+1}\mid \mathcal F_n]\le (1+A_n)X_n-B_n+C_n$1 and $E[X_{n+1}\mid \mathcal F_n]\le (1+A_n)X_n-B_n+C_n$2 a.s. (Neri et al., 17 Apr 2025)
Square-summable extension	additive term only $E[X_{n+1}\mid \mathcal F_n]\le (1+A_n)X_n-B_n+C_n$3 or general $E[X_{n+1}\mid \mathcal F_n]\le (1+A_n)X_n-B_n+C_n$4, plus increment control	convergence to a bounded set, plus a.s./high-probability/$E[X_{n+1}\mid \mathcal F_n]\le (1+A_n)X_n-B_n+C_n$5 rates (Liu et al., 30 Sep 2025)

The two quantitative paradigms answer different questions. Metastability is the finitary surrogate available at full classical generality, whereas direct asymptotic rates require stronger structural information.

4. Relaxations beyond summable additive noise

A major limitation of the classical theorem is the requirement that the additive perturbation be summable. Recent reinforcement-learning work emphasizes that many modern algorithms violate this assumption: instead of

$E[X_{n+1}\mid \mathcal F_n]\le (1+A_n)X_n-B_n+C_n$6

one encounters

$E[X_{n+1}\mid \mathcal F_n]\le (1+A_n)X_n-B_n+C_n$7

where $E[X_{n+1}\mid \mathcal F_n]\le (1+A_n)X_n-B_n+C_n$8 is Robbins–Monro and therefore not summable. The paper gives a counterexample showing that under this recursion alone the sequence can diverge to $E[X_{n+1}\mid \mathcal F_n]\le (1+A_n)X_n-B_n+C_n$9 almost surely, even though the deterministic analogue would converge to a bounded interval (Liu et al., 30 Sep 2025).

The proposed extension compensates for the loss of summability with an increment condition. For the special recursion

$n$0

if $n$1 is deterministic decreasing, $n$2, $n$3, and there exists $n$4 such that

$n$5

then for some constant $n$6,

$n$7

The conclusion is deliberately weaker than classical point convergence: the process converges almost surely to a bounded set rather than necessarily to a single limit point (Liu et al., 30 Sep 2025).

The general extension considers

$n$8

under three hypotheses: $n$9 a.s.; increment control

$$\sum_{n=0}^\infty A_n<\infty,\qquad \sum_{n=0}^\infty C_n<\infty$$0

and negative drift outside a bounded region, namely that whenever $$\sum_{n=0}^\infty A_n<\infty,\qquad \sum_{n=0}^\infty C_n<\infty$$1,

$$\sum_{n=0}^\infty A_n<\infty,\qquad \sum_{n=0}^\infty C_n<\infty$$2

The conclusion is

$$\sum_{n=0}^\infty A_n<\infty,\qquad \sum_{n=0}^\infty C_n<\infty$$3

The proof centers on the positive part

$$\sum_{n=0}^\infty A_n<\infty,\qquad \sum_{n=0}^\infty C_n<\infty$$4

for which a Robbins–Siegmund-type inequality is derived for $$\sum_{n=0}^\infty A_n<\infty,\qquad \sum_{n=0}^\infty C_n<\infty$$5. In the special case, one obtains

$$\sum_{n=0}^\infty A_n<\infty,\qquad \sum_{n=0}^\infty C_n<\infty$$6

so the new theorem effectively reuses the classical machinery after lifting the analysis to squared distance from the target set (Liu et al., 30 Sep 2025).

The same paper derives nonasymptotic refinements. Under additional conditions on $$\sum_{n=0}^\infty A_n<\infty,\qquad \sum_{n=0}^\infty C_n<\infty$$7, it proves almost sure polynomial rates such as

$$\sum_{n=0}^\infty A_n<\infty,\qquad \sum_{n=0}^\infty C_n<\infty$$8

uniform high-probability concentration bounds, and $$\sum_{n=0}^\infty A_n<\infty,\qquad \sum_{n=0}^\infty C_n<\infty$$9 convergence rates. This suggests a new convergence paradigm for algorithms whose perturbations are only square summable: the natural endpoint is set convergence with rates, rather than point convergence under classical summability.

5. Stochastic approximation, Robbins–Monro procedures, and reinforcement learning

The theorem’s influence is clearest in stochastic approximation. The relaxed quantitative theory uses the Robbins–Monro procedure as a template and isolates a regime in which the abstract rate becomes fast. When the regularity modulus is linear, $X_n\to X_\infty$0, and the coefficients satisfy

$X_n\to X_\infty$1

the theorem yields

$X_n\to X_\infty$2

hence explicit $X_n\to X_\infty$3 behavior for $X_n\to X_\infty$4. In the classical strongly monotone Robbins–Monro setting, this recovers familiar

$X_n\to X_\infty$5

type bounds for the root error, together with corresponding tail or probability estimates (Neri et al., 17 Apr 2025).

The formalization work on reinforcement learning uses Robbins–Siegmund as the common convergence engine for two distinct algorithms. Both linear TD and $X_n\to X_\infty$6-learning are rewritten in the form

$X_n\to X_\infty$7

and the proof establishes a Lyapunov inequality

$X_n\to X_\infty$8

After taking conditional expectations and using the martingale-difference property of the noise, the scalar Lyapunov process fits the special-case Robbins–Siegmund theorem. For linear TD this yields $X_n\to X_\infty$9 almost surely; for $X_n$00-learning it yields $X_n$01 almost surely. In the Markovian setting the proofs use a skeleton-iterate construction and block steps

$X_n$02

so that the theorem applies to a subsequence and the remaining iterates are handled by a Grönwall-style argument (Zhang, 5 Nov 2025).

The same RL-oriented line of work also motivates the square-summable extension. For stochastic approximation with time-inhomogeneous Markovian noise,

$X_n$03

the analysis combines the skeleton iterates technique of Qian et al. with a new error decomposition from Liu et al. After block aggregation, one obtains

$X_n$04

for $X_n$05, which lies outside the direct reach of the classical theorem but inside the reach of the extended square-summable framework. Under a negative drift condition

$X_n$06

the result is convergence to a bounded ball together with almost sure and nonasymptotic rates. The paper explicitly claims the first almost sure convergence rate, the first high probability concentration bound, and the first $X_n$07 convergence rate for $X_n$08-learning with linear function approximation (Liu et al., 30 Sep 2025).

6. Limits, misconceptions, and current research directions

A common misconception is that Robbins–Siegmund should always provide a direct rate of convergence once the recursion is identified. Recent quantitative work shows that this is false in general: at full classical generality, one should expect metastable bounds rather than a computable modulus $X_n$09. Direct rates become available only after adding further quantitative structure, such as explicit control of $X_n$10, tails of $X_n$11, and regularity of the descent observable (Neri et al., 2024, Neri et al., 17 Apr 2025).

A second misconception is that replacing a summable additive term by a square-summable one is a minor technical modification. The modern extension literature treats this as a genuinely different regime. Without extra increment control, point convergence can fail completely; with increment control, the appropriate conclusion is typically convergence to a bounded set. This is not merely a weakened proof technique but a change in the mathematical endpoint (Liu et al., 30 Sep 2025).

A third misconception is that the theorem itself dictates the step-size exponents seen in RL convergence results. In the Lean formalization, the restriction

$X_n$12

is stated to arise from the skeleton-iterate argument used to handle Markovian noise, specifically from the auxiliary relation $X_n$13, and not from Robbins–Siegmund itself. The theorem supplies the final convergence step; the technical burden lies in arranging the recursion so that its hypotheses hold (Zhang, 5 Nov 2025).

Current work therefore treats Robbins–Siegmund less as a single theorem than as a modular template. One branch develops proof-mined finitary bounds through metastability; another extracts explicit asymptotic rates in structured “relaxed supermartingale” settings; a third extends the theorem to square-summable perturbations and convergence-to-set statements motivated by reinforcement learning. A plausible implication is that future advances will continue to follow this modular pattern: derive a one-step Lyapunov inequality, normalize to a supermartingale, and then choose the Robbins–Siegmund variant whose conclusion matches the regularity and noise regime of the algorithm under study.