Quantum phase transition between spin liquid and spin nematics in spin-1 Kitaev honeycomb model

Abstract: Besides the exactly solvable spin-1/2 Kitaev model, higher spin-$S$ ones, not exactly solvable, are promising playgrounds for researches on the quantum spin liquid as well. One of the main interests in higher spin-S cases is the interplay between the Kitaev spin liquid (KSL) and spin nematics. We probe this interplay in a spin-1 model on the honeycomb lattice with competing bilinear-biquadratic and Kitaev interactions. Utilizing the 2D infinite projected entangled-pair state (iPEPS), we map out the phase diagram for the ferro-biquadratic interaction. In the phase diagram, we discover the direct KSL--spin-nematics transitions in the vicinity of pure Kitaev limits. It has been revealed that the ferro KSL exhibits robustness against perturbations from ferro-quadrupolar interactions. Also, the spin-nematic phase is extended to the parameter region near the antiferro-Kitaev limit.