Develop an empirical procedure to measure insensitivity to synonyms in image datasets

Develop an empirical methodology for benchmark image datasets to measure how neural networks’ internal representations and outputs change under synonym exchanges analogous to those defined in the Sparse Random Hierarchy Model, where each informative s-patch of low-level features produced by a given high-level latent variable is replaced by one of its m−1 synonyms while preserving positions. Construct practical tools that enable controlled synonym substitutions at multiple scales and quantify sensitivity by comparing activations or outputs between original and synonym-exchanged inputs.

Background

A central prediction of the paper is that high performance, insensitivity to diffeomorphisms, and insensitivity to synonym exchanges are learned concomitantly. The authors operationalize sensitivity to synonyms in their generative framework, but they note that measuring this insensitivity on standard image benchmarks requires the ability to modify image composition with controlled synonym-like transformations at different scales.

They suggest that diffusion-based generative models may enable such interventions, but an established empirical procedure to perform and quantify synonym exchanges in real image data is currently lacking. Creating such a procedure would allow direct testing of the paper’s core prediction on widely used datasets.