Destabilization of ferromagnetism by frustration and realization of a nonmagnetic Mott transition in the quarter-filled two-orbital Hubbard model

Abstract: The two-orbital Hubbard model on a square lattice at quarter filling (electron number per site $n=1$) is investigated by the variational Monte Carlo method. For the variational wave function, we include short-range doublon-holon binding factors. We find that the energy of this wave function is lower than that of the density-density Jastrow wave function partially including long-range correlations used in a previous study. We introduce frustration to the model by the next-nearest-neighbor hopping $t'$ in addition to the nearest-neighbor hopping $t$. For $t'=0$, a ferromagnetic state with staggered orbital order occurs by increasing the Coulomb interaction $U$ before the Mott transition takes place. By increasing $t'$, the region of this ferromagnetic phase shrinks, and the Mott transition without magnetic order occurs.