On a Rayleigh-Faber-Krahn inequality for the regional fractional Laplacian

Abstract: We study a Rayleigh-Faber-Krahn inequality for regional fractional Laplacian operators. In particular, we show that there exists a compactly supported nonnegative Sobolev function $u_0$ that attains the infimum (which will be a positive real number) of the set [ \left{ \int\int_{{u > 0}\times{u>0}} \frac{|u(x) - u(y)|^2}{|x - y|^{n + 2 \sigma}}d x d y : u \in \mathring H^\sigma(\mathbb{R}^n), \int_{\mathbb{R}^n} u^2 = 1, |{u > 0 }| \leq 1\right}. ] Unlike the corresponding problem for the usual fractional Laplacian, where the domain of the integration is $\mathbb{R}^n \times \mathbb{R}^n$, symmetrization techniques may not apply here. Our approach is instead based on the direct method and new a priori diameter estimates. We also present several remaining open questions concerning the regularity and shape of the minimizers, and the form of the Euler-Lagrange equations.