Magnetic properties and Mott transition in the Hubbard model on the anisotropic triangular lattice

Abstract: Magnetic phase diagram and Mott transition are studied in the Hubbard model on the anisotropic triangular lattice at zero temperature and half-filling by the variational cluster approximation, taking into account N\'eel, 120$<sup>\circ$ N\'eel, and collinear orderings. Paramagnetic insulator (spin liquid) is realized above the metallic phase around the isotropic point. In general, this spin liquid state, continuously connected with the metallic state, changes to a magnetic state as the on-site Coulomb repulsion $U$ increases, but it persists up to large $U$ limit in a small window between 120$<sup>\circ$ N\'eel and collinear phases. For very large $U$ another spin liquid state, separated from the metallic state by magnetic states, emerges around a narrow region where both N\'eel and 120$<sup>\circ$ N\'eel orderings are highly suppressed due to the frustration and anisotropy. Implications for the $\kappa$-(BEDT-TTF)$_2$Cu$_2$(CN)$_3$ are discussed. As for the Mott transition, the structure of the self-energy in the spectral representation is studied in detail. As $U$ increases around the Mott transition point, single dispersion evolves in the spectral weights of the self-energy which splits the non-interacting band into the upper and lower Hubbard bands.