Embedding generalization within the learning dynamics: An approach based-on sample path large deviation theory

Abstract: We consider a typical learning problem of point estimations for modeling of nonlinear functions or dynamical systems in which generalization, i.e., verifying a given learned model, can be embedded as an integral part of the learning process or dynamics. In particular, we consider an empirical risk minimization based learning problem that exploits gradient methods from continuous-time perspective with small random perturbations, which is guided by the training dataset loss. Here, we provide an asymptotic probability estimate in the small noise limit based-on the Freidlin-Wentzell theory of large deviations, when the sample path of the random process corresponding to the randomly perturbed gradient dynamical system hits a certain target set, i.e., a rare event, when the latter is specified by the testing dataset loss landscape. Interestingly, the proposed framework can be viewed as one way of improving generalization and robustness in learning problems that provides new insights leading to optimal point estimates which is guided by training data loss, while, at the same time, the learning dynamics has an access to the testing dataset loss landscape in some form of future achievable or anticipated target goal. Moreover, as a by-product, we establish a connection with optimal control problem, where the target set, i.e., the rare event, is considered as the desired outcome or achievable target goal for a certain optimal control problem, for which we also provide a verification result reinforcing the rationale behind the proposed framework. Finally, we present a computational algorithm that solves the corresponding variational problem leading to an optimal point estimates and, as part of this work, we also present some numerical results for a typical case of nonlinear regression problem.