Test the disconnected-manifold hypothesis for ImageNet counterfactual edits

Investigate whether the lower rate of minimal, successfully rendered counterfactual flips on ImageNet is caused by the task spanning multiple visual modes whose manifolds need not be connected, thereby forcing representation edits to recompose entire scenes or jump between modes.

Background

The paper reports that, when using a representation autoencoder on ImageNet, substantially fewer latent flips survive rendering than on CelebA or Square, and that many surviving flips are not minimal because the decoded edits recompose the scene rather than making a localized semantic change.

The authors attribute this behavior to differences in the data geometry: CelebA and Square are described as essentially unimodal, whereas the two-class ImageNet task spans multiple visual modes whose manifolds may not be connected. Under this explanation, a straight-line edit in representation space can leave the relevant image manifold and require the decoder to jump between modes. The paper explicitly states that this explanation has not been empirically tested.

References

We attribute this to the data: CelebA and Square are essentially unimodal, so a straight edit in the representation stays on one connected manifold of faces or squares; a two-class ImageNet task spans many visual modes whose manifolds need not be connected, and a flip may require the decoder to jump between them. We have not tested this hypothesis.

— Towards Fast and Disentangled Counterfactuals for Visual Foundation Models  (2610.00895 - Bender et al., 1 Oct 2026) in Section 6, Limitations, subsection “Minimal flips are harder on ImageNet”