Sophia: A second-order optimizer for language model training (arxiv.org)

Summary:

• Sophia is a new second-order clipped stochastic optimization algorithm that improves language model training efficiency.
• The optimizer achieves a 2x speed-up compared to Adam and adapts to the curvature in different components of the parameters.

Tags: