Absence of magnetic order and emergence of unconventional fluctuations in $J_{\rm eff} =1/2$ triangular lattice antiferromagnet YbBO$_3$

Abstract: We present the ground state properties of a new quantum antiferromagnet YbBO$3$ in which the isotropic Yb$^{3+}$ triangular layers are separated by a non-magnetic layer of partially occupied B and O(2) sites. The magnetization and heat capacity data establish a spin-orbit entangled effective spin $J{\rm eff} = 1/2$ state of Yb$^{3+}$ ions at low temperatures, interacting antiferromagnetically with an intra-layer coupling $J/k_{\rm B} \simeq 0.53$ K. The absence of oscillations and a $1/3$ tail in the zero-field muon asymmetries rule out the onset of magnetic long-range-order as well as spin-freezing down to 20~mK. An anomalous broad maximum in the temperature dependent heat capacity with a unusually reduced value and a broad anomaly in zero-field muon depolarization rate centered at $T^*\simeq 0.7 \frac{J}{k_{\rm B}}$ provide compelling evidence for a wide fluctuating regime ($0.182 \leq T/J \leq 1.63$) with slow relaxation. We infer that the fluctuating regime is a universal feature of a highly frustrated triangular lattice antiferromagnets while the absence of magnetic long-range-order is due to perfect two-dimensionality of the spin-lattice protected by non-magnetic site disorder.