Fractional topological insulators at odd-integer filling: Phase diagram of two-valley quantum Hall model (2509.16335v1)

Abstract: The fractional quantum Hall effect has recently been shown to exist in heterostructures of van der Waals materials without an externally applied magnetic field, e.g. in twisted bilayers of MoTe$_2$. These fractional Chern insulators break time-reversal symmetry spontaneously through polarization of the electron spins in a quantum spin Hall insulator band structure with flat bands. This prompts the question, which states could be realized if the spins remain unpolarized or polarize partially. Specifically, the possibility of time-reversal symmetric topological order arises. Here, we study this problem for odd integer filling of the bands, specifically focusing on vanishing and half valley polarization. Short of reliable microscopic models for small twist angles around $2.1^\circ$, we study the idealized situation of two Landau levels with opposite chirality, the two-valley quantum Hall model. Using exact diagonalization, we identify different phases arising in this model by tuning the interaction. In the physically relevant regime, the system initially exhibits phase-separated or valley-polarized states, which eventually transition into paired states by reducing onsite Coulomb repulsion.