Asynchronous option Q-learning convergence

Analyze the asynchronous versions of average-reward inter-option and intra-option Q-learning with options, in which only a subset of state-option Q-estimates is updated at each iteration, and establish finite-time convergence guarantees.

Background

The paper establishes non-asymptotic convergence rates only for synchronous inter-option and intra-option Q-learning, where all state-option coordinates are updated simultaneously. The asynchronous setting is practically important because updating every coordinate at every iteration can be computationally restrictive, but its analysis must address uneven update frequencies and the resulting stochastic dependence.

References

The natural next step is to analyse the asynchronous setting, where only a subset of the Q-estimates are updated at each iteration.

— Non-asymptotic Convergence of Average-reward Q-learning with Options  (2610.20347 - Saha et al., 14 Oct 2026) in Section Discussions

Another direction is the analysis of two-timescale option Q-learning ~{manenti2025convergencestabilityqlearninghierarchical}.

— Non-asymptotic Convergence of Average-reward Q-learning with Options  (2610.20347 - Saha et al., 14 Oct 2026) in Section Discussions

Second, an important open question is whether the unichain assumptions can be relaxed to more general multichain settings for options, in line with recent developments in average-reward RL~{lee2025optimal, Lee2026PolicyGA}.

— Non-asymptotic Convergence of Average-reward Q-learning with Options  (2610.20347 - Saha et al., 14 Oct 2026) in Section Discussions