Training-time theory for the serial vs. parallel contrast

Establish rigorous theorems that characterize a parallel-versus-serial computation contrast during neural network training, analogous to the inference-time contrast analyzed in the paper, thereby providing a formal learning-theoretic account of how serial computation requirements manifest in the training phase.

Background

The paper focuses its formal results on inference-time computation, arguing that many important problems require serial computation and that widely used parallelizable architectures cannot solve such inherently serial tasks in general.

In discussing training, the authors note that they attempted to obtain theoretical results analogous to their inference-time analysis but were unsuccessful, citing the difficulty of learning theory. They express belief that such a theory should exist and leave it to future work.