2000 character limit reached
Increasing biases can be more efficient than increasing weights
Published 3 Jan 2023 in cs.NE and cs.LG | (2301.00924v3)
Abstract: We introduce a novel computational unit for neural networks that features multiple biases, challenging the traditional perceptron structure. This unit emphasizes the importance of preserving uncorrupted information as it is passed from one unit to the next, applying activation functions later in the process with specialized biases for each unit. Through both empirical and theoretical analyses, we show that by focusing on increasing biases rather than weights, there is potential for significant enhancement in a neural network model's performance. This approach offers an alternative perspective on optimizing information flow within neural networks. See source code at https://github.com/CuriosAI/dac-dev.
- Learning Activation Functions to Improve Deep Neural Networks, 2015. arXiv:1412.6830 [cs, stat].
- Sobol tensor trains for global sensitivity analysis. Reliability Engineering & System Safety, 183:311–322, 2019.
- François Chollet. Xception: Deep Learning with Depthwise Separable Convolutions. pages 1800–1807. IEEE Computer Society, 2017.
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs), 2016. arXiv:1511.07289 [cs.LG].
- CurioSAI. Increasing biases can be more efficient than increasing weights, 2023. https://github.com/CuriosAI/dac-dev.
- RepVGG: Making VGG-Style ConvNets Great Again. pages 13733–13742, 2021.
- An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
- Hyperspectral Image Classification With Squeeze Multibias Network. IEEE Transactions on Geoscience and Remote Sensing, 57(3):1291–1301, 2019.
- Generative adversarial networks. Commun. ACM, 63(11):139–144, 2020.
- Maxout networks. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, volume 28 of JMLR Workshop and Conference Proceedings, pages 1319–1327, 2013.
- Deep residual learning for image recognition. In IEEE CVPR, pages 770–778, 2016.
- Identity mappings in deep residual networks. In Computer Vision - ECCV 2016 - 14th European Conference Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pages 630–645. Springer, 2016.
- ISIC. Isic 2019: Skin lesion analysis towards melanoma detection. https://challenge.isic-archive.com, 2019. Accessed: 2023-05-03.
- Jeremy Howard. Imagenette and Imagewoof datasets, 2019. https://github.com/fastai/imagenette.
- Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, 2014.
- Activation Ensembles for Deep Neural Networks. In 2019 IEEE International Conference on Big Data (Big Data), pages 206–214, 2019.
- CIFAR-10 and CIFAR-100 datasets, 2009. https://www.cs.toronto.edu/~kriz/cifar.html.
- Matthew E. Larkum. Are Dendrites Conceptually Useful? Neuroscience, 489:4–14, 2022.
- Multi-bias non-linear activation in deep neural networks. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 221–229, 2016.
- A ConvNet for the 2020s. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11966–11976, 2022.
- Rectifier Nonlinearities Improve Neural Network Acoustic Models. In Proceedings of the 30th International Conference on Machine Learning, ICML 2013, volume 30, page 3, 2013.
- Jeffrey C Magee. Dendritic integration of excitatory synaptic input. Nature Reviews Neuroscience, 1(3):181–190, 2000.
- SAI: A sensible artificial intelligence that plays with handicap and targets high scores in 9×\times×9 go. In ECAI 2020, volume 325, pages 403–410, 2020.
- SAI: a Sensible Artificial Intelligence that plays with handicap and targets high scores in 9x9 Go (extended version). AAAI21-RLG workshop, 2021. arXiv:1905.10863 [math.CS].
- SAI a Sensible Artificial Intelligence that plays Go. In IJCNN, pages 1–8, 2019.
- Score vs. winrate in score-based games: which reward for reinforcement learning? In 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022, pages 573–578. IEEE, 2022.
- Illuminating dendritic function with computational models. Nature Reviews Neuroscience, 21(6):303–321, 2020.
- Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, pages 4510–4520, 2018.
- Green AI. Commun. ACM, 63(12):54–63, 2020.
- SGEMM. SGEMM GPU Kernel Performance. https://archive.ics.uci.edu/ml/datasets/SGEMM+GPU+kernel+performance, 2018. Accessed: 2023-05-03.
- Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2217–2225. PMLR, 2016.
- Mastering the Game of Go with Deep Neural Networks and Tree Search. Nature, 529(7587):484–489, 2016.
- Active dendrites and local field potentials: Biophysical mechanisms and computational explorations. Neuroscience, 489:111–142, 2022. Dendritic contributions to biological and artificial computations.
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97 of Proceedings of Machine Learning Research, pages 6105–6114. PMLR, 2019.
- Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 6000–6010, 2017.
- David J Wu. Accelerating Self-Play Learning in Go. AAAI20-RLG workshop, 2020. https://arxiv.org/abs/1902.10565.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.