Time-reversal symmetry breaking fractional quantum spin Hall insulator in moiré MoTe2 (2501.02525v3)

Abstract: Twisted bilayer transition metal dichalcogenide semiconductors, which support flat Chern bands with enhanced interaction effects, realize a platform for fractional Chern insulators and fractional quantum spin Hall (FQSH) insulators. A recent experiment has reported the emergence of a FQSH insulator protected by spin-Sz conservation at a moir\'e lattice filling factor {\nu}=3 in 2.1-degree twisted bilayer MoTe2. Theoretical studies have proposed both time-reversal symmetric and asymmetric ground states as possible candidates for the observed FQSH insulator, but the nature of the state remains unexplored. Here we report the observation of spontaneous time-reversal symmetry breaking at generic fillings in 2.1-degree twisted bilayer MoTe2 from {\nu}<1 all the way to {\nu}>6 except at {\nu}=2, 4, and 6. Although zero Hall response is observed at {\nu}=3 for magnetic fields higher than 20 mT, a finite anomalous Hall response accompanied by a magnetic hysteresis is observed at lower magnetic fields, demonstrating spontaneous time-reversal symmetry breaking. Our work shows the tendency towards ferromagnetism by doping the first three pairs of conjugate Chern bands in the material; it also sheds light on the nature of the FQSH insulator at {\nu}=3.