Hinge-Proximal Stochastic Gradient Methods for Convex Optimization with Functional Constraints

Abstract: This paper considers stochastic convex optimization problems with smooth functional constraints arising in constrained estimation and robust signal recovery. We operate in the high-dimensional and highly-constrained setting, where oracle access is restricted to one or a few objective and constraint gradients per-iteration, as in streaming or online estimation. Existing approaches to solve such problems are based on either the stochastic primal-dual or stochastic subgradient methods, and require globally Lipschitz continuous constraint functions. In this work, we develop a hinge-proximal framework that utilizes an exact penalty reformulation to yield updates involving only one linearized constraint (and hence accessing one constraint gradient) per-iteration. The updates also admit a novel hinge-proximal three-point inequality relying on smoothness rather than global Lipschitz continuity of the constraint functions. The framework leads to three algorithms: a baseline hinge-proximal SGD (HPS), a variance-reduced HPS version for finite-sum settings, and a nested HPS version whose performance depends on a geometric regularity constant of the constraint region rather than explicitly on the number of constraints, while achieving near-SGD sample complexity. The superior empirical performance of the proposed algorithms is demonstrated on a robust regression problem with noisy features, representative of errors-in-variables estimation.