Phase diagram of the Kitaev-Heisenberg-$Γ$ model: Classical and quantum magnetism, frustration, and subdominant interactions

Abstract: The Kitaev spin liquid provides a rare example of exactly solvable quantum spin liquid states. Intensive research over the past two decades has identified a variety of its candidate materials. In real materials, however, the Kitaev interaction is inevitably accompanied by additional magnetic interactions such as the Heisenberg and $Γ$ interactions. These interactions often induce magnetic ordering at low temperatures, making it essential to clarify their effects in the search for and design of Kitaev spin liquid candidate materials. In this study, we revisit the ground-state phase diagram of the Kitaev-Heisenberg-$Γ$ model from both classical and quantum perspectives, using state-of-the-art numerical techniques. In the classical case, we reveal a $zoo$ $of$ $noncollinear$ $orders$, where a variety of noncollinear multiple-$Q$ magnetic orders with and without incommensurate modulations emerge. In the quantum case, we unravel that quantum fluctuations suppress many of the competing orders found in the classical case, resulting in a reduced number of dominant incommensurate orders. We further identify $highly$ $frustrated$ regions, where spiral spin liquid states as well as new magnetically ordered states are potentially stabilized by other additional magnetic interactions. Our results provide a comprehensive perspective on the Kitaev-Heisenberg-$Γ$ model for both classical and quantum spins and offer a valuable guide not only for interpreting experimental results on candidate materials, but also for searching and designing new materials to realize the Kitaev spin liquid.