2000 character limit reached
SPARLING: Learning Latent Representations with Extremely Sparse Activations
Published 3 Feb 2023 in cs.LG | (2302.01976v2)
Abstract: Real-world processes often contain intermediate state that can be modeled as an extremely sparse tensor. We introduce Sparling, a technique that allows you to learn models with intermediate layers that match this state from only end-to-end labeled examples (i.e., no supervision on the intermediate state). Sparling uses a new kind of informational bottleneck that enforces levels of activation sparsity unachievable using other techniques. We find that extreme sparsity is necessary to achieve good intermediate state modeling. On our synthetic DigitCircle domain as well as the LaTeX-OCR and Audio-MNIST-Sequence domains, we are able to precisely localize the intermediate states up to feature permutation with > 90% accuracy, even though we only train end-to-end.
- Cross-modal scene networks. IEEE transactions on pattern analysis and machine intelligence, 40(10):2303–2314, 2017.
- Interpreting and explaining deep neural networks for classification of audio signals. CoRR, abs/1807.03418, 2018.
- Sparsely activated networks. IEEE Transactions on Neural Networks and Learning Systems, 32(3):1304–1313, 2020.
- Auto-association by multilayer perceptrons and singular value decomposition. Biological cybernetics, 59(4):291–294, 1988.
- What you get is what you see: A visual markup decompiler. arXiv preprint arXiv:1609.04938, 10:32–37, 2016.
- Disentangling factors of variation via generative entangling. arXiv preprint arXiv:1210.5474, 2012.
- Identity mappings in deep residual networks. In European conference on computer vision, pp. 630–645. Springer, 2016.
- beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations, 2016.
- An empirical analysis of different sparse penalties for autoencoder in unsupervised feature learning. In 2015 international joint conference on neural networks (IJCNN), pp. 1–8. IEEE, 2015.
- Stochastic gradient vb and the variational auto-encoder. In Second International Conference on Learning Representations, ICLR, volume 19, pp. 121, 2014.
- Concept bottleneck models. In International Conference on Machine Learning, pp. 5338–5348. PMLR, 2020.
- Lassonet: A neural network with feature sparsity. The Journal of Machine Learning Research, 22(1):5633–5661, 2021.
- Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 4114–4124. PMLR, 2019.
- Transformed l1 regularization for learning sparse deep neural networks. Neural Networks, 119:286–298, 2019. ISSN 0893-6080. doi: https://doi.org/10.1016/j.neunet.2019.08.015. URL https://www.sciencedirect.com/science/article/pii/S0893608019302321.
- Recurrent models of visual attention. Advances in neural information processing systems, 27, 2014.
- Andrew Ng. Cs294a lecture notes: Sparse autoencoder, Winter 2011. URL https://web.stanford.edu/class/cs294a/sparseAutoencoder.pdf.
- Group sparse regularization for deep neural networks. Neurocomputing, 241:81–89, 2017. ISSN 0925-2312. doi: https://doi.org/10.1016/j.neucom.2017.02.029. URL https://www.sciencedirect.com/science/article/pii/S0925231217302990.
- Grad-cam: Why did you say that? arXiv preprint arXiv:1611.07450, 2016.
- Is attention interpretable? arXiv preprint arXiv:1906.03731, 2019.
- Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
- Ladder variational autoencoders. Advances in neural information processing systems, 29, 2016.
- Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267–288, 1996.
- Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
- Byung-Jun Yoon. Hidden markov models and their applications in biological sequence analysis. Current genomics, 10(6):402–415, 2009.
- Concept embedding models. arXiv preprint arXiv:2209.09056, 2022.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
