On Sato--Tate distributions, extremal traces, and real multiplication in genus 2

Abstract: The vertical Sato--Tate conjectures gives expected trace distributions for for families of curves. We develop exact expression for the distribution associated to degree-$4$ representations of $\mathrm{USp}(4)$, $\mathrm{SU}(2)\times\mathrm{SU}(2)$ and $\mathrm{SU}(2)$ in the neighborhood of the extremities of the Weil bound. As a consequence we derive qualitative distinctions between the extremal traces arising from generic genus-$2$ curves and genus-$2$ curves with real or quaternionic multiplication. In particular we show, in a specific sense, to what extent curves with real multiplication dominate the contribution to extremal traces.