On the Success Probability of Three Detectors for the Box-Constrained Integer Linear Model

Abstract: This paper is concerned with detecting an integer parameter vector inside a box from a linear model that is corrupted with a noise vector following the Gaussian distribution. One of the commonly used detectors is the maximum likelihood detector, which is obtained by solving a box-constrained integer least squares problem, that is NP-hard. Two other popular detectors are the box-constrained rounding and Babai detectors due to their high efficiency of implementation. In this paper, we first present formulas for the success probabilities (the probabilities of correct detection) of these three detectors for two different situations: the integer parameter vector is deterministic and is uniformly distributed over the constraint box. Then, we give two simple examples to respectively show that the success probability of the box-constrained rounding detector can be larger than that of the box-constrained Babai detector and the latter can be larger than the success probability of the maximum likelihood detector when the parameter vector is deterministic, and prove that the success probability of the box-constrained rounding detector is always not larger than that of the box-constrained Babai detector when the parameter vector is uniformly distributed over the constraint box. Some relations between the results for the box constrained and ordinary cases are presented, and two bounds on the success probability of the maximum likelihood detector, which can easily be computed, are developed. Finally, simulation results are provided to illustrate our main theoretical findings.