Competing states in the $S=1/2$ triangular-lattice $J_1$-$J_2$ Heisenberg model: a dynamical density-matrix renormalization group study

Abstract: Previous studies of the $S=1/2$ triangular-lattice $J_1$--$J_2$ Heisenberg antiferromagnet have inferred the existence of a non-magnetic ground-state phase for an intermediate range of $J_2$, but disagree concerning whether it is a gapped $\mathbb{Z}_2$ quantum spin liquid (QSL), a gapless (Dirac) QSL, or a weakly symmetry-broken phase. Using an improved dynamical density-matrix renormalization group method, we investigate the relevant intermediate $J_2$ regime for cylinders with circumferences from 6 to 9. Depending on the initial state and boundary conditions, we find two {\it distinct} variational states. The higher energy state is consistent with a Dirac QSL. In the lower-energy state, both the static and dynamical properties are qualitatively similar to the magnetically ordered state at $J_2=0$, suggestive of either a weakly magnetically ordered non-QSL or a gapped QSL proximate to a continuous transition to such an ordered state.