Role of the spin-orbit coupling in the Kugel-Khomskii model on the honeycomb lattice

Abstract: We study the effective spin-orbital model for honeycomb-layered transition metal compounds, applying the second-order perturbation theory to the three-orbital Hubbard model with the anisotropic hoppings. This model is reduced to the Kitaev model in the strong spin-orbit coupling limit. Combining the cluster mean-field approximations with the exact diagonalization, we treat the Kugel-Khomskii type superexchange interaction and spin-orbit coupling on an equal footing to discuss ground-state properties. We find that a zigzag ordered state is realized in the model within nearest-neighbor interactions. We clarify how the ordered state competes with the nonmagnetic state, which is adiabatically connected to the quantum spin liquid state realized in a strong spin-orbit coupling limit. Thermodynamic properties are also addressed. The present work should provide another route to account for the Kitaev-based magnetic properties in candidate materials.