Characterizing the ε1–ε2 trade-off in FTG numerical stability and independence checks

Characterize and optimize the trade-off between the tolerance ε1 used to validate the numerical inverse of the Gram matrix via SVD (by checking that G·G^{-1} is within ε1 of the identity) and the threshold ε2 used to treat the inner product as zero in the linear independence predicate of Fourier Tree Growing (FTG), including its impact on conditioning of the Gram matrix, feasibility of generating linearly independent compositions, computational complexity, and overall performance.

Background

To mitigate numerical issues from ill-conditioned Gram matrices in FTG, the authors use SVD to approximate the inverse and accept it only if the product with the Gram matrix is close to identity within a tolerance ε1. Separately, they treat small inner products as zero using ε2 to enforce linear independence in generated function compositions. They note that increasing ε2 makes independence harder to achieve but yields better-conditioned Gram matrices.

This creates a practical trade-off between ε1 and ε2 that affects numerical stability, conditioning, and the algorithm’s ability to progress. The authors explicitly defer a detailed investigation of this trade-off.