Optimal pre-training initialization for PMMs

Determine the optimal pre-training initialization of parametric matrix model parameters in order to reduce the computational cost of training.

Background

The thesis identifies parameter initialization as an important practical component of training parametric matrix models (PMMs). Although simple heuristics—small initial magnitudes and symmetry-breaking initial values—have been successful, no generally optimal initialization strategy has been established.

An improved initialization method could substantially reduce the number of optimization steps and the computational resources required to train PMMs, particularly for large or complicated models.