Reduction of the effective population size in a branching particle system in the moderate mutation-selection regime

Abstract: We consider a system of particles performing a one-dimensional dyadic branching Brownian motion with positive drift $\beta\in(0,1)$, branching rate 1/2, killed at $L(\beta)>0$, and reflected at 0. The killing boundary $L(\beta)$ is chosen so that the total population size is approximately constant, proportional to $N\in\mathbb{N}$. This branching system is interpreted as a population accumulating deleterious mutations. We prove that, when the typical width of the cloud of particles is of order $c\log(N)$, $c\in(0,1)$, the demographic fluctuations of the system converge to a Feller diffusion on the time scale $N^{1-c}$. In addition, we show that the limiting genealogy of the system comprises only binary mergers and that these mergers are concentrated in the vicinity of the reflective boundary. This model is a version of the branching Brownian motion with absorption studied by Berestycki, Berestycki and Schweinsberg to describe the effect of natural selection on the genealogy of a population accumulating beneficial mutations. In the latter case, the genealogical structure of the system is described by a Bolthausen-Sznitman coalescent on a logarithmic time scale. In this work, we show that, when the population size in the fittest class is mesoscopic, namely of order $N^{1-c}$, the genealogy of the system is given by a Kingman coalescent on a polynomial time scale.