Visibility in Polygonal Environments with Holes: Finding Best Spots for Hiding and Surveillance

Abstract: Visibility plays an important role for decision making in cluttered, uncertain environments. This paper considers the problem of identifying optimal hiding spots for an agent against line-of-sight detection by an adversary whose location is unknown. We consider environments modeled as polygons with holes. We develop a set of mathematical tools for reasoning about visibility as a function of position and rely on non-smooth analysis to formally characterize the regularity properties of various visibility-based metrics. These metrics are non-smooth and non-convex, so off-the-shelf optimization algorithms can only guarantee convergence to Clarke critical points. To address this, the proposed Normalized Descent algorithm leverages the structure of non-smooth points in visibility problems and introduces randomness to escape saddle points. Our technical analysis allows for the non-monotonic decrease in the visibility metric and strengthens the algorithm guarantees, ensuring convergence to local minima with high probability. Simulations on two hide-and-seek scenarios showcase the effectiveness of the proposed approach.