Mathematical characterization of dominance between constructive interference and ReLU-based filtering

Establish a complete mathematical characterization of when constructive interference versus ReLU-based interference filtering dominates in the representations learned by tied-weight autoencoders with ReLU decoders trained on correlated bag-of-words data, specifically in settings where both mechanisms coexist within the same model.

Background

The paper argues that in realistic, correlated data settings, non-linear autoencoders can exploit constructive interference—arranging features according to co-activation patterns—rather than solely relying on ReLU-based interference filtering. This leads to efficient reconstructions and accounts for observed semantic clusters and cyclical structures in language-model-like settings.

While empirical evidence demonstrates that constructive interference and ReLU filtering can both appear, and even coexist for the same feature, the authors note they have not provided a complete mathematical theory delineating the conditions under which each mechanism is favored or dominant. Developing such a characterization would clarify the regimes and constraints (e.g., covariance structure, bottleneck size, weight decay) that select one mechanism over the other.