Do latent variables learned by deep models represent real-world sources or artifacts?

Determine whether hidden latent variables learned by deep machine learning models contain information about the real-world source generating the data or whether the internal-state representations are artifacts of the training and model architecture.

Background

The review highlights opacity challenges specific to ML models in scientific contexts, notably whether learned latent representations correspond to genuine features of the physical systems under study or are merely artifacts of model training.

Resolving this uncertainty is critical for trusting ML-derived scientific insights; if latent variables do reflect real sources, they can serve as interpretable, physically meaningful descriptors. If they are artifacts, reliance on them may mislead scientific inference.

References

It is often unclear whether the hidden latent variables within the model (see Sec.~\ref{chapter:algorithms}) contain information about the source of the data or if the representations of the internal states are merely artifacts .

— Interpretable Machine Learning in Physics: A Review  (2503.23616 - Wetzel et al., 30 Mar 2025) in Section 3.2, The opacity/black-box problem

The observed coordinates are best read as geometry of source proposals shaped by changed files and lineages; the experiment did not test whether they correspond to distinct mechanisms or runtime behavior.

— Loreley: Repository-Scale Program Evolution with Quality-Diversity Search  (2608.19703 - Chen, 20 Aug 2026) in Section 5.2, “Archive and descriptor behavior”