Stochastic Zeroth-order Functional Constrained Optimization: Oracle Complexity and Applications

Abstract: Functionally constrained stochastic optimization problems, where neither the objective function nor the constraint functions are analytically available, arise frequently in machine learning applications. In this work, assuming we only have access to the noisy evaluations of the objective and constraint functions, we propose and analyze stochastic zeroth-order algorithms for solving the above class of stochastic optimization problem. When the domain of the functions is $\mathbb{R}^n$, assuming there are $m$ constraint functions, we establish oracle complexities of order $\mathcal{O}((m+1)n/\epsilon^2)$ and $\mathcal{O}((m+1)n/\epsilon^3)$ respectively in the convex and nonconvex setting, where $\epsilon$ represents the accuracy of the solutions required in appropriately defined metrics. The established oracle complexities are, to our knowledge, the first such results in the literature for functionally constrained stochastic zeroth-order optimization problems. We demonstrate the applicability of our algorithms by illustrating its superior performance on the problem of hyperparameter tuning for sampling algorithms and neural network training.