New Signatures of Flavor Violating Higgs Couplings

Abstract: We explore several novel LHC signatures arising from quark or lepton flavor violating couplings in the Higgs sector, and we constrain such couplings using LHC data. Since the largest signals are possible in channels involving top quarks or tau leptons, we consider in particular the following flavor violating processes: (1) $pp \to thh$ (top plus di-Higgs final state) arising from a dimension six coupling of up-type quarks to three insertions of the Higgs field. We develop a search strategy for this final state and demonstrate that detection is possible at the high luminosity LHC if flavor violating top--up--Higgs couplings are not too far below the current limit. (2) $pp \to tH^0$, where $H^0$ is the heavy neutral CP-even Higgs boson in a two Higgs doublet model (2HDM). We consider the decay channels $H^0 \to tu, WW, ZZ, hh$ and use existing LHC data to constrain the first three of them. For the fourth, we adapt our search for the $thh$ final state, and we demonstrate that in large regions of the parameter space, it is superior to other searches, including searches for flavor violating top quark decays ($t \to hq$). (3) $H^0 \to \tau\mu$, again in the context of a 2HDM. This channel is particularly well motivated by the recent CMS excess in $h \to \tau\mu$, and we use the data from this search to constrain the properties of $H^0$.