Benefits of stricter normalization for large-scale models (especially LLMs trained with RL)

Determine whether applying stricter normalization schemes—such as enforcing unit-norm constraints via hyperspherical (ℓ2) normalization—provides benefits for large-scale models, particularly large language models trained with reinforcement learning-based objectives, and establish in what contexts such normalization is advantageous relative to conventional normalization techniques.

Background

The paper introduces SimbaV2, a reinforcement learning architecture that stabilizes non-stationary optimization by constraining weight, feature, and gradient norms using hyperspherical normalization, along with distributional critics and reward scaling. SimbaV2 achieves state-of-the-art results across diverse continuous-control benchmarks and scales effectively with model size and compute.

In discussing future directions, the authors point to the growing interest in reinforcement learning for training LLMs. They explicitly note that it remains an open question whether the stricter normalization principles that benefit SimbaV2 in RL would also confer advantages for large models more broadly, including LLMs trained via RL.