Differences of the Selberg trace formula and Selberg type zeta functions for Hilbert modular surfaces

Abstract: We present the first example of the Selberg type zeta function for noncompact higher rank locally symmetric spaces. We study certain Selberg type zeta functions and Ruelle type zeta functions attached to the Hilbert modular group of a real quadratic field. We show that they have meromorphic extensions to the whole complex plane and satisfy functional equations. The method is based on considering the differences among several Selberg trace formulas with different weights for the Hilbert modular group. Besides as an application of the differences of the Selberg trace formula, we also obtain an asymptotic average of the class numbers of indefinite binary quadratic forms over the real quadratic integer ring.