Orthogonality of invariant vectors

Abstract: Let $G$ be a finite group with given subgroups $H$ and $K$. Let $\pi$ be an irreducible complex representation of $G$ such that its space of $H$-invariant vectors as well as the space of $K$-invariant vectors are both one dimensional. Let $v_H$ (resp. $v_K$) denote an $H$-invariant (resp. $K$-invariant) vector of unit norm in the standard $G$-invariant inner product $\langle ~,~ \rangle_\pi$ on $\pi$. Our interest is in computing the square of the absolute value of $\langle v_H,v_K \rangle_\pi$. This is the correlation constant $c(\pi;H,K)$ defined by Gross. In this paper, we give a sufficient condition for $\langle v_H, v_K \rangle_\pi$ to be zero and a sufficient condition for it to be non-zero (i.e., $H$ and $K$ are correlated with respect to $\pi$), when $G={\rm GL}2(\mathbb F_q)$, where $\mathbb F_q$ is the finite field of $q=p^f$ elements of odd characteristic $p$, $H$ is its split torus and $K$ is a non-split torus. The key idea in our proof is to analyse the mod $p$ reduction of $\pi$. We give an explicit formula for $|\langle v_H,v_K \rangle\pi|^2$ modulo $p$. Finally, we study the behaviour of $\langle v_H,v_K \rangle_\pi$ under the Shintani base change and give a sufficient condition for $\langle v_H,v_K \rangle_\pi$ to vanish for an irreducible representation $\pi={\rm BC}(\tau)$ of ${\rm PGL}_2(\mathbb E)$, in terms of the epsilon factor of the base changing representation $\tau$ of ${\rm PGL}_2(\mathbb F)$, where $\mathbb E/\mathbb F$ is a finite extension of finite fields. This is reminiscent of the vanishing of $L(1/2, {\rm BC}(\tau))$, in the theory of automorphic forms, when the global root number of $\tau$ is $-1$.