Phase diagram of the strongly correlated Kane-Mele-Hubbard model

Abstract: We explore the phase diagram of the strongly correlated Hubbard model with intrinsic spin orbit coupling on the honeycomb lattice. We obtain the low energy effective model describing the spin degree of freedom. We study the resulting model within the Schwinger boson and Schwinger fermion approaches. The Schwinger boson approach gives the boundary between the spin liquid phase and the magnetically ordered phases, Neel order and incommensurate Neel order. We find that increasing the strength of the spin orbit coupling, narrows the width of the spin liquid region. The Schwinger fermion approach sheds further light on the nature of the spin liquid phase. We obtain three different candidates for the spin liquid phase within the mean field approximation which are gapless spin liquid, topological Mott insulator, and the chiral spin liquid phases. We argue that the gauge fluctuations and the instanton effect may suppress the first two spin liquids, while the chiral spin liquid is stable against gauge fluctuations due to its nontrivial topology.