Z_2-vortex lattice in the ground state of the triangular Kitaev-Heisenberg model

Abstract: The triangular-lattice Heisenberg antiferromagnet (HAF) is known to carry topological Z_2 vortex excitations which form a gas at finite temperatures. Here we show that the spin-orbit interaction, introduced via a Kitaev term in the exchange Hamiltonian, condenses these vortices into a triangular $Z_2$ vortex crystal at zero temperature. The cores of the Z_2 vortices show abrupt, soliton-like magnetization modulations and arise by a special intertwining of three honeycomb superstructures of ferromagnetic domains, one for each of the three sublattices of the 120-degree state of the pure HAF. This is a new example of a nucleation transition, analogous to the spontaneous formation of magnetic domains, Abrikosov vortices in type-II syperconductors, blue phases in cholesteric liquid crystals, and skyrmions in chiral helimagnets. As the mechanism relies on the interplay of geometric frustration and spin-orbital anisotropies, such vortex mesophases can materialize as a ground-state property in spin-orbit coupled correlated systems with nearly hexagonal topology, as in triangular or strongly frustrated honeycomb iridates.