2000 character limit reached
WavPool: A New Block for Deep Neural Networks (2306.08734v1)
Published 14 Jun 2023 in cs.LG and stat.ML
Abstract: Modern deep neural networks comprise many operational layers, such as dense or convolutional layers, which are often collected into blocks. In this work, we introduce a new, wavelet-transform-based network architecture that we call the multi-resolution perceptron: by adding a pooling layer, we create a new network block, the WavPool. The first step of the multi-resolution perceptron is transforming the data into its multi-resolution decomposition form by convolving the input data with filters of fixed coefficients but increasing size. Following image processing techniques, we are able to make scale and spatial information simultaneously accessible to the network without increasing the size of the data vector. WavPool outperforms a similar multilayer perceptron while using fewer parameters, and outperforms a comparable convolutional neural network by ~ 10% on relative accuracy on CIFAR-10.
- Primeval Adiabatic Perturbation in an Expanding Universe. The Astrophysical Journal, 162:815, December 1970. doi: 10.1086/150713.
- J. R. Bond and G. Efstathiou. Cosmic background radiation anisotropies in universes dominated by nonbaryonic dark matter. The Astrophysical Journal Letters, 285:L45–L48, October 1984. doi: 10.1086/184362.
- G. Turin. An introduction to matched filters. IRE Transactions on Information Theory, 6(3):311–329, 1960. doi: 10.1109/TIT.1960.1057571.
- F. Rosenblatt. The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6):386–408, 1958. ISSN 0033-295X. doi: 10.1037/h0042519. URL http://dx.doi.org/10.1037/h0042519.
- George V. Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2:303–314, 1989.
- Multilayer feedforward networks are universal approximators. Neural Networks, 2:359–366, 1989.
- Gradient-based learning applied to document recognition. Proceedings of the Institute of Radio Engineers, 86(11):2278–2323, 1998. ISSN 0018-9219. doi: 10.1109/5.726791.
- Attention Is All You Need. arXiv e-prints, art. arXiv:1706.03762, June 2017. doi: 10.48550/arXiv.1706.03762.
- Ingrid Daubechies. Ten Lectures on Wavelets. Society for Industrial and Applied Mathematics, 1992. doi: 10.1137/1.9781611970104. URL https://epubs.siam.org/doi/abs/10.1137/1.9781611970104.
- S.G. Mallat. A theory for multiresolution signal decomposition: the wavelet representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(7):674–693, 1989. doi: 10.1109/34.192463.
- Yves Meyer. Bull. Amer. Math. Soc., 28:350–360, 1993. doi: 10.1090/S0273-0979-1993-00363-X.
- I. Daubechies. Where do wavelets come from? a personal point of view. Proceedings of the IEEE, 84(4):510–513, 1996. doi: 10.1109/5.488696.
- The JPEG 2000 still image compression standard. IEEE Signal Processing Magazine, 18(5):36–58, September 2001. doi: 10.1109/79.952804.
- Learning a Wavelet-like Auto-Encoder to Accelerate Deep Neural Networks. arXiv e-prints, art. arXiv:1712.07493, December 2017. doi: 10.48550/arXiv.1712.07493.
- Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution. arXiv e-prints, art. arXiv:1704.03915, April 2017. doi: 10.48550/arXiv.1704.03915.
- Multi-level wavelet-cnn for image restoration. CoRR, abs/1805.07071, 2018. URL http://arxiv.org/abs/1805.07071.
- Wavelet convolutional neural networks. CoRR, abs/1805.08620, 2018. URL http://arxiv.org/abs/1805.08620.
- Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learning. arXiv e-prints, art. arXiv:2207.04978, July 2022. doi: 10.48550/arXiv.2207.04978.
- Fully learnable deep wavelet transform for unsupervised monitoring of high-frequency time series. Proceedings of the National Academy of Science, 119(8):e2106598119, February 2022. doi: 10.1073/pnas.2106598119.
- P. Burt and E. Adelson. The laplacian pyramid as a compact image code. IEEE Transactions on Communications, 31(4):532–540, 1983. doi: 10.1109/TCOM.1983.1095851.
- Convolutional neural networks at constrained time cost. CoRR, abs/1412.1710, 2014. URL http://arxiv.org/abs/1412.1710.
- Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010.
- Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms, 2017.
- Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
- Pytorch: An imperative style, high-performance deep learning library. CoRR, abs/1912.01703, 2019. URL http://arxiv.org/abs/1912.01703.
- Pywavelets: A python package for wavelet analysis. Journal of Open Source Software, 4(36):1237, 2019. doi: 10.21105/joss.01237. URL https://doi.org/10.21105/joss.01237.
- Fernando Nogueira. Bayesian Optimization: Open source constrained global optimization tool for Python, 2014–. URL https://github.com/fmfn/BayesianOptimization.
- Stphane Mallat. A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way. Academic Press, Inc., USA, 3rd edition, 2008. ISBN 0123743702.
- Quantum neural networks: Concepts, applications, and challenges, 2021.
- Stéphane Mallat. Group Invariant Scattering. arXiv e-prints, art. arXiv:1101.2286, January 2011. doi: 10.48550/arXiv.1101.2286.
- Wavelets and their use. Phys. Usp., 44:447, 2001. doi: 10.1070/PU2001v044n05ABEH000918.
Sponsor
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.