Is the Frequency Principle always valid?

Abstract: We investigate the learning dynamics of shallow ReLU neural networks on the unit sphere (S^2\subset\mathbb{R}^3) in polar coordinates ((\tau,\phi)), considering both fixed and trainable neuron directions ({w_i}). For fixed weights, spherical harmonic expansions reveal an intrinsic low-frequency preference with coefficients decaying as (O(\ell^{5/2}/2^\ell)), typically leading to the Frequency Principle (FP) of lower-frequency-first learning. However, this principle can be violated under specific initial conditions or error distributions. With trainable weights, an additional rotation term in the harmonic evolution equations preserves exponential decay with decay order (O(\ell^{7/2}/2^\ell)) factor, also leading to the FP of lower-frequency-first learning. But like fixed weights case, the principle can be violated under specific initial conditions or error distributions. Our numerical results demonstrate that trainable directions increase learning complexity and can either maintain a low-frequency advantage or enable faster high-frequency emergence. This analysis suggests the FP should be viewed as a tendency rather than a rule on curved domains like (S^2), providing insights into how direction updates and harmonic expansions shape frequency-dependent learning.