Replica Theory of Spherical Boltzmann Machine Ensembles

Abstract: Training in machine learning generally consists in finding one model, whose parameters minimize a data-dependent loss. Yet, empirical work shows that ensemble learning, an approach in which multiple models are sampled, can improve performance. Here, we provide an analytical framework to understand these observations in the case of Boltzmann machines, exploiting a duality between ensemble learning and large deviations of the free energy in spin-glass models. Replica calculations allow us to fully solve the case of spherical Boltzmann machine ensembles, and clarify when ensemble learning improves over standard loss minimization. Our findings are corroborated by numerical simulations on deep networks. Special care is brought to the case of nearly finite-dimensional data, for which we show that replica predictions are valid for arbitrarily large number of data points compared to the (large) embedding dimension.