Dynamic Universal Approximation Theory: Foundations for Parallelism in Neural Networks
Abstract: Neural networks are increasingly evolving towards training large models with big data, a method that has demonstrated superior performance across many tasks. However, this approach introduces an urgent problem: current deep learning models are predominantly serial, meaning that as the number of network layers increases, so do the training and inference times. This is unacceptable if deep learning is to continue advancing. Therefore, this paper proposes a deep learning parallelization strategy based on the Universal Approximation Theorem (UAT). From this foundation, we designed a parallel network called Para-Former to test our theory. Unlike traditional serial models, the inference time of Para-Former does not increase with the number of layers, significantly accelerating the inference speed of multi-layer networks. Experimental results validate the effectiveness of this network.
- Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis. arXiv:1802.09941.
- An Analysis of Single-Layer Networks in Unsupervised Feature Learning. In International Conference on Artificial Intelligence and Statistics.
- Cybenko, G. 2007. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems, 303–314.
- Sparse GPU Kernels for Deep Learning. SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, 1–14.
- Generative adversarial nets. Advances in neural information processing systems, 27.
- Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778.
- Multilayer feedforward networks are universal approximators. Neural Networks, 2: 359–366.
- Krizhevsky, A. 2009. Learning Multiple Layers of Features from Tiny Images.
- Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177.
- LLM-Pruner: On the Structural Pruning of Large Language Models. ArXiv, abs/2305.11627.
- Automated Flower Classification over a Large Number of Classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, 722–729.
- Cats and dogs. 2012 IEEE Conference on Computer Vision and Pattern Recognition, 3498–3505.
- U-Net: Convolutional Networks for Biomedical Image Segmentation. ArXiv, abs/1505.04597.
- A Simple and Effective Pruning Approach for Large Language Models. ArXiv, abs/2306.11695.
- Attention is All you Need. In Neural Information Processing Systems.
- Universal Approximation Theory: The basic theory for deep learning-based computer vision models. arXiv:2407.17480.
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning. ArXiv, abs/2310.06694.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
