Partial up-up-down order with the continuously distributed order parameter in the triangular antiferromagnet TmMgGaO4

Abstract: Frustrated quasidoublets without time-reversal symmetry can host highly unconventional magnetic structures with continuously distributed order parameters even in a single-phase crystal. Here, we report the comprehensive thermodynamic and neutron diffraction investigation on the single crystal of TmMgGaO$<em>4$, which entails non-Kramers Tm$<sup>{3+}$ ions arranged on a geometrically perfect triangular lattice. The crystal electric field (CEF) randomness caused by the site-mixing disorder of the nonmagnetic Mg$<sup>{2+}$ and Ga$<sup>{3+}$ ions, merges two lowest-lying CEF singlets of Tm$<sup>{3+}$ into a ground-state (GS) quasidoublet. Well below $T_c$ $\sim$ 0.7 K, a small fraction of the antiferromagnetically coupled Tm$<sup>{3+}$ Ising quasidoublets with small inner gaps condense into two-dimensional (2D) up-up-down magnetic structures with continuously distributed order parameters, and give rise to the \emph{columnar} magnetic neutron reflections below $\mu_0H_c$ $\sim$ 2.6 T, with highly anisotropic correlation lengths, $\xi</em>{ab}$ $\geq$ 250$a$ in the triangular plane and $\xi_c$ $&lt;$ $c$/12 between the planes. The remaining fraction of the Tm$<sup>{3+}$ ions remain nonmagnetic at 0 T and become uniformly polarized by the applied longitudinal field at low temperatures. We argue that the similar model can be generally applied to other compounds of non-Kramers rare-earth ions with correlated GS quasidoublets.