Generalized Hankel/Toeplitz matrix for array signal processing

Abstract: In this paper, we introduce generalized Hankel/Toeplitz matrices (GHM/GTM) and the associated generalized Vandermonde decomposition for nonuniform array signal processing and multi-dimensional super-resolution. The proposed framework was discovered from the study of resolution limit theory and extends the classical Hankel/Toeplitz structure by allowing substantially more flexible sampling geometries while preserving the underlying low-rank Vandermonde factorization. Through devising an optimal algorithm based on this GHM framework, we derive the state-of-the-art upper bound estimate for the computational resolution limit (CRL) of source-number detection in general $d$-dimensional super-resolution problems. For segmented sampling sets, whose geometry is closely related to sparse and distributed arrays, we establish deterministic lower bounds for the minimum singular values of the associated generalized Vandermonde matrices and derive corresponding stability and number-detection guarantees for multi-clump source configurations. To address the computational bottleneck of conventional multi-level Hankel constructions in high dimensions, we further introduce randomized GHM constructions whose matrix dimensions scale with the effective degrees of freedom rather than with the full tensor-product grid, together with deterministic recovery guarantees conditional on the realized Vandermonde factors. We also extend the framework to source localization by developing GHM-based MUSIC algorithms for nonuniform measurements, with stability characterized through the conditioning of the generalized Vandermonde factors. Numerical experiments on synthetic data demonstrate that the proposed GHM-based methods achieve competitive resolution and recovery accuracy while substantially reducing matrix size and computational cost, especially in high-dimensional settings.