Papers
Topics
Authors
Recent
Detailed Answer
Quick Answer
Concise responses based on abstracts only
Detailed Answer
Well-researched responses based on abstracts and relevant paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses
Gemini 2.5 Flash
Gemini 2.5 Flash 34 tok/s
Gemini 2.5 Pro 49 tok/s Pro
GPT-5 Medium 27 tok/s Pro
GPT-5 High 30 tok/s Pro
GPT-4o 80 tok/s Pro
Kimi K2 198 tok/s Pro
GPT OSS 120B 461 tok/s Pro
Claude Sonnet 4 38 tok/s Pro
2000 character limit reached

Autoencoding Under Normalization Constraints (2105.05735v3)

Published 12 May 2021 in cs.LG

Abstract: Likelihood is a standard estimate for outlier detection. The specific role of the normalization constraint is to ensure that the out-of-distribution (OOD) regime has a small likelihood when samples are learned using maximum likelihood. Because autoencoders do not possess such a process of normalization, they often fail to recognize outliers even when they are obviously OOD. We propose the Normalized Autoencoder (NAE), a normalized probabilistic model constructed from an autoencoder. The probability density of NAE is defined using the reconstruction error of an autoencoder, which is differently defined in the conventional energy-based model. In our model, normalization is enforced by suppressing the reconstruction of negative samples, significantly improving the outlier detection performance. Our experimental results confirm the efficacy of NAE, both in detecting outliers and in generating in-distribution samples.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Ganomaly: Semi-supervised anomaly detection via adversarial training. In Asian conference on computer vision, pp.  622–637. Springer, 2018.
  2. What regularized auto-encoders learn from the data-generating distribution. The Journal of Machine Learning Research, 15(1):3563–3593, 2014.
  3. Improving unsupervised defect segmentation by applying structural similarity to autoencoders. arXiv preprint arXiv:1807.02011, 2018.
  4. Bishop, C. M. Novelty detection and neural network validation. IEE Proceedings-Vision, Image and Signal processing, 141(4):217–222, 1994.
  5. Auto-association by multilayer perceptrons and singular value decomposition. Biological cybernetics, 59(4-5):291–294, 1988.
  6. Flows for simultaneous manifold learning and density estimation. arXiv preprint arXiv:2003.13913, 2020.
  7. Hyperspherical variational auto-encoders. In 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pp.  856–865. Association For Uncertainty in Artificial Intelligence (AUAI), 2018.
  8. Implicit generation and modeling with energy based models. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alche-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32, pp.  3608–3618. Curran Associates, Inc., 2019.
  9. From variational to deterministic autoencoders. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=S1g7tpEYDS.
  10. Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection. In IEEE International Conference on Computer Vision (ICCV), 2019.
  11. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01367, 2018.
  12. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=Hkxzx0NtDB.
  13. Representations of knowledge in complex systems. Journal of the Royal Statistical Society: Series B (Methodological), 56(4):549–581, 1994.
  14. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp.  297–304, 2010.
  15. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=HyxCxhRcY7.
  16. Gans trained by a two time-scale update rule converge to a local nash equilibrium. arXiv preprint arXiv:1706.08500, 2017.
  17. Hinton, G. E. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771–1800, 2002.
  18. A novelty detection approach to classification. In Proceedings of the International Joint Conference on Artificial Intelligence, volume 1, pp.  518–523, 1995.
  19. Measuring compositional generalization: A comprehensive method on realistic data. arXiv preprint arXiv:1912.09713, 2019.
  20. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  21. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pp. 10215–10224, 2018.
  22. Auto-encoding variational bayes. International Conference on Learning Representations (ICLR), 2014.
  23. Perfect density models cannot guarantee anomaly detection. arXiv preprint arXiv:2012.03808, 2020.
  24. Anomaly detection for skin disease images using variational autoencoder. arXiv preprint arXiv:1807.01349, 2018.
  25. Lyudchik, O. Outlier detection using autoencoders. Technical report, 2016.
  26. Autoregressive score matching. arXiv preprint arXiv:2010.12810, 2020.
  27. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111–3119, 2013.
  28. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.
  29. Do deep generative models know what they don’t know? In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=H1xwNhCcYm.
  30. Autoregressive energy machines. In International Conference on Machine Learning, pp. 1735–1744. PMLR, 2019.
  31. Ng, A. et al. Sparse autoencoder. CS294A Lecture notes, 2011.
  32. Learning non-convergent non-persistent short-run mcmc toward energy-based model. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32, pp.  5232–5242. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/2bc8ae25856bc2a6a1333d1331a3b7a6-Paper.pdf.
  33. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759, 2016.
  34. Neural discrete representation learning. arXiv preprint arXiv:1711.00937, 2017.
  35. Parisi, G. Correlation functions and computer simulations. Nuclear Physics B, 180(3):378–384, 1981.
  36. Generative probabilistic novelty detection with adversarial autoencoders. In Advances in neural information processing systems, pp. 6822–6833, 2018.
  37. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems, pp. 14680–14691, 2019.
  38. Contractive auto-encoders: explicit invariance during feature extraction. In Proceedings of the 28th International Conference on International Conference on Machine Learning, pp.  833–840, 2011.
  39. Exponential convergence of langevin distributions and their discrete approximations. Bernoulli, 2(4):341–363, 1996.
  40. Learning Internal Representations by Error Propagation, pp. 318–362. MIT Press, Cambridge, MA, USA, 1986. ISBN 026268053X.
  41. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517, 2017.
  42. Input complexity and out-of-distribution detection with likelihood-based generative models. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SyxIWpVYvr.
  43. Improved techniques for training score-based generative models. arXiv preprint arXiv:2006.09011, 2020.
  44. How to train your energy-based models. arXiv preprint arXiv:2101.03288, 2021.
  45. A classification framework for anomaly detection. Journal of Machine Learning Research, 6(Feb):211–232, 2005.
  46. Tieleman, T. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference on Machine learning, pp.  1064–1071, 2008.
  47. Fixing bias in reconstruction-based anomaly detection with lipschitz discriminators. arXiv preprint arXiv:1905.10710, 2019.
  48. Vincent, P. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661–1674, 2011.
  49. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp.  1096–1103, 2008.
  50. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600–612, 2004.
  51. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-11), pp.  681–688, 2011.
  52. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp.  3–19, 2018.
  53. Likelihood regret: An out-of-distribution detection score for variational auto-encoder. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  20685–20696. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/eddea82ad2755b24c4e168c5fc2ebd40-Paper.pdf.
  54. {VAEBM}: A symbiosis between variational autoencoders and energy-based models. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=5m3SEczOV8L.
  55. A theory of generative convnet. In International Conference on Machine Learning, pp. 2635–2644. PMLR, 2016.
  56. Spherical latent spaces for stable variational autoencoders. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.  4503–4513, 2018.
  57. Younes, L. On the convergence of markovian stochastic algorithms with rapidly decreasing ergodicity rates. Stochastics: An International Journal of Probability and Stochastic Processes, 65(3-4):177–228, 1999.
  58. Latent variables on spheres for autoencoders in high dimensions. arXiv, pp.  arXiv–1912, 2019.
  59. Energy-based generative adversarial network. arXiv preprint arXiv:1609.03126, 2016.
  60. Spatio-temporal autoencoder for video anomaly detection. In Proceedings of the 25th ACM international conference on Multimedia, pp.  1933–1941, 2017.
  61. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=BJJLHbb0-.
Citations (36)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
Dice Question Streamline Icon: https://streamlinehq.com

Follow-Up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube