Anisotropic magnetic interactions in a candidate Kitaev spin liquid close to a metal-insulator transition

Abstract: In the Kitaev honeycomb model, spins coupled by strongly-frustrated anisotropic interactions do not order at low temperature but instead form a quantum spin liquid with spin fractionalization into Majorana fermions and static fluxes. The realization of such a model in crystalline materials could lead to major breakthroughs in understanding entangled quantum states, however achieving this in practice is a very challenging task. The recently synthesized honeycomb material RuI$_3$ shows no long-range magnetic order down to the lowest probed temperatures and has been theoretically proposed as a quantum spin liquid candidate material on the verge of an insulator to metal transition. Here we report a comprehensive study of the magnetic anisotropy in un-twinned single crystals via torque magnetometry and detect clear signatures of strongly anisotropic and frustrated magnetic interactions. We attribute the development of sawtooth and six-fold torque signal to strongly anisotropic, bond-dependent magnetic interactions by comparing to theoretical calculations. As a function of magnetic field strength at low temperatures, torque shows an unusual non-parabolic dependence suggestive of a proximity to a field-induced transition. Thus, RuI$_3$, without signatures of long-range magnetic order, displays key hallmarks of an exciting new candidate for extended Kitaev magnetism with enhanced quantum fluctuations.