Interaction-driven Roton Condensation in C = 2/3 Fractional Quantum Anomalous Hall State

Abstract: The interplay of topological order and charge order exhibits rich physics. Recent experiments that succesfully realized the frational quantum anomalous Hall (FQAH) effect in twisted MoTe$_2$ bilayers and rhombohedral multilayer graphene without external magnetic field further call for deeper understanding of the relation between topological order and charge order in quantum moir\'e materials. In the archetypal correlated flat-band model on checkerboard lattice, a FQAH smectic state with coexistent topological order and smectic charge order has been numerically discovered at filling $\nu$ = 2/3. In this work, we explore the global ground-state phase diagram of the model with competing interactions and find a C = 2/3 FQAH phase surrounded by four different charge density wave (CDW) phases. In particular, we identify a FQAH-CDW transition triggered by roton condensation, in that, the minimal roton gap continues to decrease at the same finite momentum, along with the diverging density flucuations at the transition point, after which the system enters into a CDW metal phase with the same ordered wavevector. Our discovery points out that the charge-neutral roton modes can play a significant role in a transition from FQAH topological order to CDW symmetry-breaking order, discussed in FQH literature while severely neglected in FQAH systems.