Multiplicity and concentration of solutions for fractional Schrödinger systems via penalization method

Abstract: The aim of this paper is to investigate the existence, multiplicity and concentration of positive solutions for the following nonlocal system of fractional Schr\"odinger equations \begin{equation*} \left{ \begin{array}{ll} \varepsilon^{2s} (-\Delta)^{s}u+V(x)u=Q_{u}(u, v) &\mbox{ in } \mathbb{R}^{N}, \varepsilon^{2s} (-\Delta)^{s}v+W(x)v=Q_{v}(u, v) &\mbox{ in } \mathbb{R}^{N}, u, v>0 &\mbox{ in } \mathbb{R}^{N}, \end{array} \right. \end{equation*} where $\varepsilon>0$ is a parameter, $s\in (0, 1)$, $N>2s$, $(-\Delta)^{s}$ is the fractional Laplacian, $V:\mathbb{R}^{N}\rightarrow \mathbb{R}$ and $W:\mathbb{R}^{N}\rightarrow \mathbb{R}$ are positive continuous potentials, $Q$ is a homogeneous $C^{2}$-function with subcritical growth. In order to relate the number of solutions with the topology of the set where the potentials $V$ and $W$ attain their minimum values, we apply penalization techniques, Nehari manifold arguments and Ljusternik-Schnirelmann theory.