Unveiling Excitonic Insulator Signatures in Ta$_\mathrm{2}$NiSe$_\mathrm{5}$

Abstract: The high-temperature phase of Ta$\mathrm{2}$NiSe$\mathrm{5}$, a near-zero-gap semiconductor ($E_G$ = 0), is a promising candidate for an excitonic insulator. Given the dome-like evolution expected for an excitonic insulator around $E_G$, we investigated Ta$\mathrm{2}$NiSe$\mathrm{5}$, the more semi-metallic Ta$\mathrm{2}$(Ni,Co)Se$\mathrm{5}$, and semiconducting Ta$\mathrm{2}$NiS$\mathrm{5}$ using high-resolution single-crystal x-ray diffraction and near-edge x-ray absorption fine structure (NEXAFS). Our findings reveal a second-order structural phase transition from orthorhombic (space group: $Cmcm$) to monoclinic (space group: $C2/c$) in Ta$\mathrm{2}$NiSe$\mathrm{5}$ and Ta$\mathrm{2}$(Ni,Co)Se$\mathrm{5}$, but no transition in Ta$\mathrm{2}$NiS$\mathrm{5}$ down to 2 K. This transition breaks two mirror symmetries, enabling and enhancing the hybridization of Ta, Ni, and Se atoms, shortening bond lengths, and strengthening orbital interactions. NEXAFS data confirm stronger hybridization, significant changes in excitonic binding energies, and a key alteration in orbital character, suggesting an excitonic insulating state in Ta$\mathrm{2}$NiSe$\mathrm{5}$ and emphasizing the crucial electronic role of orbitals in the formation of the excitonic insulator state.