Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

As large as it gets: Learning infinitely large Filters via Neural Implicit Functions in the Fourier Domain (2307.10001v2)

Published 19 Jul 2023 in cs.CV

Abstract: Recent work in neural networks for image classification has seen a strong tendency towards increasing the spatial context. Whether achieved through large convolution kernels or self-attention, models scale poorly with the increased spatial context, such that the improved model accuracy often comes at significant costs. In this paper, we propose a module for studying the effective filter size of convolutional neural networks. To facilitate such a study, several challenges need to be addressed: 1) we need an effective means to train models with large filters (potentially as large as the input data) without increasing the number of learnable parameters 2) the employed convolution operation should be a plug-and-play module that can replace conventional convolutions in a CNN and allow for an efficient implementation in current frameworks 3) the study of filter sizes has to be decoupled from other aspects such as the network width or the number of learnable parameters 4) the cost of the convolution operation itself has to remain manageable i.e. we cannot naively increase the size of the convolution kernel. To address these challenges, we propose to learn the frequency representations of filter weights as neural implicit functions, such that the better scalability of the convolution in the frequency domain can be leveraged. Additionally, due to the implementation of the proposed neural implicit function, even large and expressive spatial filters can be parameterized by only a few learnable weights. Our analysis shows that, although the proposed networks could learn very large convolution kernels, the learned filters are well localized and relatively small in practice when transformed from the frequency to the spatial domain. We anticipate that our analysis of individually optimized filter sizes will allow for more efficient, yet effective, models in the future. https://github.com/GeJulia/NIFF.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (54)
  1. Improving stability during upsampling–on the importance of spatial context. arXiv preprint arXiv:2311.17524, 2023.
  2. Spectral-based convolutional neural network without multiple spatial-frequency domain switchings. Neurocomputing, 364:152–167, 2019.
  3. The fourier transform and its applications. American Journal of Physics, 34(8):712–712, 1966.
  4. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  8628–8638, 2021.
  5. Fast fourier convolution. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  4479–4488. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/2fd5d41ec6cfab47e32164d5624269b1-Paper.pdf.
  6. Neural unsigned distance fields for implicit function learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  21638–21652. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/f69e505b08403ad2298b9f262659929a-Paper.pdf.
  7. Steerable cnns. arXiv preprint arXiv:1612.08498, 2016.
  8. An algorithm for the machine calculation of complex fourier series. Mathematics of computation, 19(90):297–301, 1965.
  9. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp.  764–773, 2017.
  10. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.  248–255. Ieee, 2009.
  11. Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  11963–11975, June 2022.
  12. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  12124–12134, 2022.
  13. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  14. George H Dunteman. Principal components analysis, volume 69. Sage, 1989.
  15. Watch your up-convolution: Cnn based generative deep neural networks are failing to reproduce spectral distributions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  16. D. Forsyth and J. Ponce. Computer Vision: A Modern Approach. An Alan R. Apt book. Prentice Hall, 2003. ISBN 9780130851987. URL https://books.google.de/books?id=VAd5QgAACAAJ.
  17. Frequencylowcut pooling – plug & play against catastrophic overfitting. In European Conference on Computer Vision, 2022. URL https://arxiv.org/abs/2204.00491.
  18. Fix your downsampling asap! be natively more robust via aliasing and spectral artifact free pooling. arXiv preprint arXiv:2307.09804, 2023.
  19. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=uYLFoz1vlAC.
  20. Specnet: spectral domain convolutional neural network. arXiv preprint arXiv:1905.10915, 2019.
  21. Segnext: Rethinking convolutional attention design for semantic segmentation. arXiv preprint arXiv:2209.08575, 2022.
  22. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016.
  23. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  770–778, 2016.
  24. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  1314–1324, 2019.
  25. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  4700–4708, 2017.
  26. Spectral distribution aware image generation. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp.  1734–1742, 2021.
  27. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):1–41, 2022.
  28. More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. arXiv preprint arXiv:2207.03620, 2022a.
  29. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp.  10012–10022, 2021.
  30. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
  31. Improving native cnn robustness with filter frequency regularization. Transactions on Machine Learning Research, 2023.
  32. Hyper-convolution networks for biomedical image segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.  1933–1942, 2022.
  33. Hyper-convolutions via implicit kernels for medical image analysis. Medical Image Analysis, 86:102796, 2023. ISSN 1361-8415. doi: https://doi.org/10.1016/j.media.2023.102796. URL https://www.sciencedirect.com/science/article/pii/S1361841523000579.
  34. Fast training of convolutional networks through ffts. arXiv preprint arXiv:1312.5851, 2013.
  35. Learning convolutional neural networks in the frequency domain, 2022. URL https://arxiv.org/abs/2204.06718.
  36. Large kernel matters – improve semantic segmentation by global convolutional network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  37. Resolution learning in deep convolutional networks using scale-space theory. IEEE Transactions on Image Processing, 30:8342–8353, 2021.
  38. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pp.  28043–28078. PMLR, 2023.
  39. Fcnn: Fourier convolutional neural networks. In Michelangelo Ceci, Jaakko Hollmén, Ljupčo Todorovski, Celine Vens, and Sašo Džeroski (eds.), Machine Learning and Knowledge Discovery in Databases, pp.  786–798, Cham, 2017. Springer International Publishing. ISBN 978-3-319-71249-9.
  40. Global filter networks for image classification. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=K_Mnsw5VoOW.
  41. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. In International Conference on Learning Representations, 2022a. URL https://openreview.net/forum?id=3jooF27-0Wy.
  42. CKConv: Continuous kernel convolution for sequential data. In International Conference on Learning Representations, 2022b. URL https://openreview.net/forum?id=8FhxBtXSl0.
  43. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  4510–4520, 2018.
  44. Implicit neural representations with periodic activation functions. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp.  7462–7473. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/53c04118df112c13a8c34b38343b9c10-Paper.pdf.
  45. Scale-equivariant steerable networks. arXiv preprint arXiv:1910.11093, 2019.
  46. Nergis Tomen and Jan C van Gemert. Spectral leakage and rethinking the kernel size in cnns. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  5138–5147, 2021.
  47. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp.  10347–10357. PMLR, 2021.
  48. Deep complex networks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=H1T2hmZAb.
  49. Fast convolutional nets with fbfft: A gpu performance evaluation. arXiv preprint arXiv:1412.7580, 2014.
  50. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  51. Combining fft and spectral-pooling for efficient convolution neural network model. In 2016 2nd International Conference on Artificial Intelligence and Industrial Engineering (AIIE 2016), pp.  203–206. Atlantis Press, 2016.
  52. Image classification in frequency domain with 2srelu: a second harmonics superposition activation function. Applied Soft Computing, 112:107851, 2021.
  53. Shmuel Winograd. On computing the discrete fourier transform. Mathematics of computation, 32(141):175–199, 1978.
  54. Deep scale-spaces: Equivariance over scale. Advances in Neural Information Processing Systems, 32, 2019.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Julia Grabinski (8 papers)
  2. Janis Keuper (66 papers)
  3. Margret Keuper (77 papers)
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets