Band mixing and particle-hole asymmetry in moiré fractional Chern insulators

Abstract: We investigate the effect of remote band mixing on the stability of fractional Chern insulators in a family of models that approximate continuum descriptions of moiré materials. Our results suggest that the experimentally observed asymmetry between filling fractions $ν=1/3$ and $ν=2/3$ in twisted MoTe$_2$ originates from a competition between a fractional Chern insulator, an electron Wigner crystal, and a hole Wigner crystal. In the absence of band mixing, the leading instability at $ν= 1/3$ is the electron crystal, whereas at $ν= 2/3$ the main competing phase is the hole crystal. Remote band mixing substantially lowers the energy of the electron crystal but has only a weak effect on the hole crystal. Consequently, it destabilizes the fractional Chern insulator at $ν=1/3$ more strongly than at $ν=2/3$. This mechanism also provides an explanation for the emergence of re-entrant integer quantum anomalous Hall states in moiré MoTe$_2$ for fillings $ν>1/2$.

• The paper shows that remote band mixing preferentially stabilizes electron Wigner crystals, destabilizing ν=1/3 and reinforcing ν=2/3 FCIs.
• It employs a unified exact diagonalization across models (Landau, AC, adiabatic, continuum) to capture the influence of non-uniform quantum geometry and band dispersion.
• The study links competing charge-density-wave instabilities to experimental observations in tMoTe₂, providing a framework to tune FCI phase stability.

Band Mixing and Particle-Hole Asymmetry in Moiré Fractional Chern Insulators

Introduction

This work analyzes the influence of remote band mixing on the stability and particle-hole asymmetry of fractional Chern insulator (FCI) states in moiré systems, focusing on twisted MoTe$_2$ (tMoTe$_2$). The study targets a key experimental observation: FCIs in tMoTe$_2$ demonstrate enhanced robustness at filling fractions $\nu > 1/2$, particularly at $\nu = 2/3$, contradicting the behavior in Landau levels where the $\nu = 1/3$ state is typically the strongest. Using a unified exact diagonalization (ED) approach that systematically includes remote band effects, the authors propose that this asymmetry results from band mixing preferentially stabilizing electron Wigner crystals over hole Wigner crystals. This competition underlies both the strong asymmetry in FCI robustness and the emergence of re-entrant integer quantum anomalous Hall (RIQAH) phases near $\nu = 2/3$.

Theoretical Framework and Model Hierarchy

The authors construct a family of models interpolating between paradigmatic quantum Hall physics and realistic moiré bands:

• Landau Level Model: All bands are flat with ideal uniform quantum geometry, realized for $B_1=U_1=0$.
• Aharonov-Casher (AC) Model: $B_1\neq 0, U_1=0$, yielding a flat but non-ideal lowest band with a dispersive remote band.
• Adiabatic Model: $B_1, U_1 \neq 0$ introduces finite bandwidth and non-uniform quantum geometry in both the lowest and first remote bands.
• Full Continuum tMoTe$_2$0: A continuum model that accurately captures the moiré superlattice electronic structure and band topology.

Interactions are projected onto $_2$1 and $_2$2 bands, and the effect of the band-mixing parameter $_2$3 is systematically explored for each case.

Figure 1: Schematic illustration of the single-particle models, displaying the increasing complexity of quantum geometry and band dispersions.

Band Mixing Effects in Landau Levels

In the pure Landau level regime, incorporating Landau level (LL) mixing via two-band projection (as opposed to traditional one-band projection) breaks the particle-hole symmetry manifest at $_2$4. As $_2$5 increases, the many-body gap at $_2$6 monotonically decreases, whereas at $_2$7, it displays non-monotonic evolution due to a level crossing in the excitation spectrum. This behavior is traced to a shift in the wave vector of the magnetoroton minimum, marking a change in the competing charge-density-wave (CDW) instability. The crossover reflects the growing influence of electron Wigner crystal states as LL mixing increases.

Figure 2: Exact diagonalization spectra for Landau levels. The non-monotonic $_2$8 gap arises from a momentum sector crossing as $_2$9 increases.

Role of Competing Charge-Density-Wave States

The competing CDW states—electron and hole Wigner crystals—are characterized by distinct real-space periodicities and topological characteristics.

Figure 3: Real-space charge densities of electron and hole Wigner crystals at $_2$0 and $_2$1.

The key observation is that remote band (LL) mixing lowers the energy of electron Wigner crystals much more efficiently than for their hole counterparts. For $_2$2, the system is driven more rapidly towards an electron crystal instability with increasing $_2$3, directly suppressing the FCI gap. At $_2$4, however, a transition between dominant hole and electron Wigner crystal character occurs with increasing band mixing, reflected in a shift in the excitation spectrum and structure factor.

AC and Adiabatic Band Models: Quantum Geometry and Further Asymmetry

Extending to AC bands, the non-uniform Berry curvature introduces particle-hole asymmetry already at the one-band level, but remote band mixing further amplifies this effect. The structural feature persists: the $_2$5 gap monotonically decreases, while the $_2$6 gap is stabilized over a broad range of $_2$7 due to a shift in the competing CDW instability.

Figure 4: Many-body gaps for AC bands as function of $_2$8. Band mixing reverses the relative robustness of the $_2$9 and $\nu > 1/2$0 gaps.

Including a nonzero periodic scalar potential ($\nu > 1/2$1) in the adiabatic model simulates finite bandwidth and increased quantum geometry complexity. Still, the primary mechanism for particle-hole asymmetry and the stabilization of $\nu > 1/2$2 FCI persists. For all parameter regimes considered, the two-band projected gap at $\nu > 1/2$3 is larger than at $\nu > 1/2$4.

Figure 5: Gap evolution for adiabatic bands. At $\nu > 1/2$5, band mixing is essential for enhanced FCI stability.

Continuum tMoTe$\nu > 1/2$6 Models: Connection to Experiment

Applying the framework to continuum models fit to experimental tMoTe$\nu > 1/2$7 band structures corroborates the mechanisms previously identified. ED results consistently show that, when remote band effects are included, the FCI gap at $\nu > 1/2$8 exceeds that at $\nu > 1/2$9. The non-monotonic spectrum evolution and the CDW competition mechanism generalize to realistic quasi-atomic moiré models.

Figure 6: Many-body FCI gap evolution for two continuum tMoTe$\nu = 2/3$0 models, establishing robust $\nu = 2/3$1 FCI for realistic band-mixing parameters.

Implications and Outlook

This work identifies a key microscopic mechanism underlying particle-hole asymmetry and the relative stability of higher-filling FCIs in strongly interacting moiré systems: remote band mixing fundamentally stabilizes electron Wigner crystals relative to holes, tipping the competition in favor of $\nu = 2/3$2 FCIs for realistic parameters. The analysis delineates the roles of quantum geometry (Berry curvature, quantum metric) and direct energetic competition between CDW and FCI order. These findings provide a consistent framework for interpreting the observed phase diagrams of tMoTe$\nu = 2/3$3 and other moiré materials.

The implications are clear for both theory and experiment: accurate modeling of FCIs and related phenomena in narrow-band moiré systems must incorporate remote-band mixing for qualitative correctness. On the experimental side, tuning the dielectric environment, remote-band structure, and bandwidth offers a concrete method to manipulate FCI phase stability and explore the predicted transitions between FCI and competing CDW/RIQAH phases.

Conclusion

The systematic inclusion of remote band mixing within model Hamiltonians for moiré FCIs elucidates the dynamical origin of particle-hole asymmetry and FCI robustness observed in tMoTe$\nu = 2/3$4. The study reveals that band mixing preferentially stabilizes electron Wigner crystals, which, in turn, destabilizes $\nu = 2/3$5 FCIs and reinforces those at $\nu = 2/3$6. This effect is robust across model hierarchies and connects directly with current experimental findings, offering a framework for future exploration of correlated topological phases in moiré matter.