Semilinear elliptic equations with the pseudo-relativistic operator on a bounded domain

Abstract: We study the Dirichlet problem for the semilinear equations involving the pseudo-relativistic operator on a bounded domain, (\sqrt{-\Delta + m^2} - m)u =|u|^{p-1}u \quad \textrm{in}~\Omega, with the Dirichlet boundary condition $u=0$ on $\partial \Omega$. Here, $p \in (1,\infty)$ and the operator $(\sqrt{-\Delta + m^2} - m)$ is defined in terms of spectral decomposition. In this paper, we investigate existence and nonexistence of a nontrivial solution, depending on the choice of $p$, $m$ and $\Omega$. Precisely, we show that $(i)$ if $p$ is not $H<sup>1$ subcritical ($p \geq \frac{n+2}{n-2}$) and $\Omega$ is star-shaped, the equation has no nontrivial solution for all $m &gt; 0$; $(ii)$ if $p$ is not $H<sup>{1/2}$ supercritical ($1 &lt;p \leq \frac{n+1}{n-1}$), then there exists a least energy solution for all $$m\&gt;0$$ and any bounded domain $\Omega$; $(iii)$ finally, in the intermediate range ($\frac{n+1}{n-1}&lt;p&lt;\frac{n+2}{n-2}$), the problem has a nontrivial solution, provided that $m$ is sufficiently large and the problem -\Delta u = |u|^{p-1}u \quad \textrm{in}~\Omega, \qquad u =0\quad \textrm{on}~\partial \Omega admits a non-degenerate nontrivial solution, for example, when $\Omega$ is a ball or an annulus.