Scalar coupling limits and diphoton Higgs decay from LHC in an $U(1)'$ model with scalar dark matter

Abstract: In the context of an nonuniversal $U(1)'$ extension of the standard model free from anomalies, we introduce a complex scalar singlet candidate to be dark matter. In addition, an extra scalar doublet and a heavy scalar singlet are required to provide masses to all fermions and to break spontaneously the symmetries. From unitarity and stability of the Higgs potential, we find the full set of bounds and order relations for the scalar coupling constants. Using recent data from the CERN-LHC collider, we study the signal strenght of the diphoton Higgs decay $R_{\gamma \gamma}$, which imposes very stringent bounds to the scalar couplings and other scalar parameters. We obtain constraints in different scenarios of the space of parameters, where decays into dark matter may or may not contribute according to the mass of the scalar dark matter candidate. By assuming that the lightest scalar boson of the model corresponds to the observed Higgs boson, we evaluate deviations from the SM of the trilineal Higgs self-coupling. The conditions from unitarity, stability and Higgs diphoton decay data allow trilineal deviations in the range $0 \leq \delta g \lesssim -72\%$.