Importance of hyper-parameter optimization during training of physics-informed deep learning networks

Abstract: Incorporating scientific knowledge into deep learning (DL) models for materials-based simulations can constrain the network's predictions to be within the boundaries of the material system. Altering loss functions or adding physics-based regularization (PBR) terms to reflect material properties informs a network about the physical constraints the simulation should obey. The training and tuning process of a DL network greatly affects the quality of the model, but how this process differs when using physics-based loss functions or regularization terms is not commonly discussed. In this manuscript, several PBR methods are implemented to enforce stress equilibrium on a network predicting the stress fields of a high elastic contrast composite. Models with PBR enforced the equilibrium constraint more accurately than a model without PBR, and the stress equilibrium converged more quickly. More importantly, it was observed that independently fine-tuning each implementation resulted in more accurate models. More specifically, each loss formulation and dataset required different learning rates and loss weights for the best performance. This result has important implications on assessing the relative effectiveness of different DL models and highlights important considerations when making a comparison between DL methods.