Quantum oscillation in narrow-gap topological insulators

Abstract: The canonical understanding of quantum oscillation in metals is challenged by the observation of de Haas-van Alphen effect in an insulator, SmB$_{6}$ [Tan \emph{et al}, Science {\bf349}, 287 (2015)]. Based on a two-band model with inverted band structure, we show that the periodically narrowing hybridization gap in magnetic fields can induce the oscillation of low-energy density of states in the bulk, which is observable provided that the hybridization gap is narrow and comparable to the Landau level spacing. Its temperature dependence strongly deviates from the Lifshitz-Kosevich theory. The nontrivial band topology manifests itself as a nonzero Berry phase in the oscillation pattern, which crosses over to a trivial Berry phase by increasing the temperature or the magnetic field. Further predictions to experiments are also proposed.