Convergence of Markovian Stochastic Approximation with discontinuous dynamics

Abstract: This paper is devoted to the convergence analysis of stochastic approximation algorithms of the form $\theta_{n+1} = \theta_n + \gamma_{n+1} H_{\theta_n}(X_{n+1})$ where ${\theta_nn, n \geq 0}$ is a $R^d$-valued sequence, ${\gamma, n \geq 0}$ is a deterministic step-size sequence and ${X_n, n \geq 0}$ is a controlled Markov chain. We study the convergence under weak assumptions on smoothness-in-$\theta$ of the function $\theta \mapsto H_{\theta}(x)$. It is usually assumed that this function is continuous for any $x$; in this work, we relax this condition. Our results are illustrated by considering stochastic approximation algorithms for (adaptive) quantile estimation and a penalized version of the vector quantization.