Meson spectroscopy in the $Sp(4)$ gauge theory with three antisymmetric fermions

Abstract: We report the results of an extensive numerical study of the $Sp(4)$ lattice gauge theory with three (Dirac) flavors of fermion in the two-index antisymmetric representation. In the presence of (degenerate) fermion masses, the theory has an enhanced global $SU(6)$ symmetry, broken explicitly and spontaneously to its $SO(6)$ subgroup. This symmetry breaking pattern makes the theory interesting for applications in the context of composite Higgs models, as well as for the implementation of top partial compositeness. It can also provide a dynamical realisation of the strongly interacting massive particle paradigm for the origin of dark matter. We adopt the standard plaquette gauge action with the Wilson-Dirac formulation for the fermions and apply the (rational) hybrid Monte Carlo algorithm in our ensemble generation process. We monitor the autocorrelation and topology of the ensembles. We explore the bare parameter space, and identify the weak and strong coupling regimes separated by a line of first-order bulk phase transitions. We measure two-point correlation functions between meson operators that transform as non-trivial representations of $SO(6)$, and extract the ground-state masses and the decay constants, in all accessible spin and parity channels. In addition, we measure the mass of the first excited state for the vector meson by solving a generalised eigenvalue problem. Spectral quantities show a mass dependence that is compatible with the expectation that, at long distances, the theory undergoes confinement, accompanied by the spontaneous breaking of the approximate global symmetries acting on the matter fields. Finally, we discuss the continuum and massless extrapolations, after setting the physical scale using the gradient flow method, and compare the results to those of existing studies in the quenched approximation, as well as to the literature on closely related theories.