Competing magnetic and spin vestigial orders from continuum field theory

Abstract: We study magnetic systems featuring competing antiferromagnetic and spin-nematic phases, described by dipolar primary and quadrupolar secondary order parameters. While the antiferromagnetic phase breaks time-reversal and spin-rotational symmetries, only spin-rotational symmetry is broken in the nematic phase, which can be understood as a spin-vestigial phase. We develop a classical continuum field theory with independent vector and tensor fields and use mean-field and renormalization group methods to analyze its phases and finite-temperature transitions. We determine the fixed-point structure using an $ε$ expansion about the upper critical dimension of six and track the resulting fixed points to lower dimensions using a perturbative fixed-dimension renormalization group approach. The multicritical fixed point governing the meeting of the antiferromagnetic, nematic, and paramagnetic phases occurs at imaginary coupling throughout the dimensions accessible to our analysis, indicating a first-order transition through the triple point. Away from the triple point, a direct paramagnetic-to-antiferromagnetic transition can be continuous when the tensor mass is sufficiently large, without an intervening vestigial phase. We argue that it is governed by the cubic universality class in $d=3$ and the Ising universality class in $d=2$ for three-component order parameters. The theory also supports a two-step transition with an intermediate spin-nematic phase. The paramagnetic-to-nematic transition is governed by the four-state Potts universality class in $d=2$ and is first order in $d=3$, while the nematic-to-antiferromagnetic transition is generically expected to be continuous and of Ising type in both dimensions. Our results provide a field-theoretical framework for understanding the finite-temperature transitions observed in the candidate Kitaev material Na$_2$Co$_2$TeO$_6$.