Transfer operators and Hankel transforms between relative trace formulas, II: Rankin-Selberg theory

Abstract: The goal of this article and its precursor is to demonstrate, by example, the existence of "transfer operators" betweeen relative trace formulas, which generalize the scalar transfer factors of endoscopy. These transfer operators have all properties that one could expect from a trace formula comparison: matching, fundamental lemma for the Hecke algebra, transfer of (relative) characters. Most importantly, and quite surprisingly, they appear to be of abelian nature (at least, in the low-rank examples considered in this paper), even though they encompass functoriality relations of non-abelian harmonic analysis. Thus, they are amenable to application of the Poisson summation formula in order to perform the global comparison. Moreover, we show that these abelian transforms have some structure -- which presently escapes our understanding in its entirety -- as deformations of well-understood operators when the spaces under consideration are replaced by their "asymptotic cones". In this second paper we use Rankin-Selberg theory to prove the local transfer behind Rudnick's 1990 thesis (comparing the stable trace formula for $\operatorname{SL}_2$ with the Kuznetsov formula) and Venkatesh's 2002 thesis (providing a "beyond endoscopy" proof of functorial transfer from tori to $\operatorname{GL}_2$). As it turns out, the latter is not completely disjoint from endoscopic transfer -- in fact, our proof "factors" through endoscopic transfer. We also study the functional equation of the symmetric-square $L$-function for $\operatorname{GL}_2$, and show that it is governed by an explicit "Hankel operator" at the level of the Kuznetsov formula, which is also of abelian nature. A similar theory for the standard $L$-function was previously developed (in a different language) by Jacquet.