2000 character limit reached
Sequence Length Independent Norm-Based Generalization Bounds for Transformers
Published 19 Oct 2023 in stat.ML and cs.LG | (2310.13088v1)
Abstract: This paper provides norm-based generalization bounds for the Transformer architecture that do not depend on the input sequence length. We employ a covering number based approach to prove our bounds. We use three novel covering number bounds for the function class of bounded linear transformations to upper bound the Rademacher complexity of the Transformer. Furthermore, we show this generalization bound applies to the common Transformer training technique of masking and then predicting the masked word. We also run a simulated study on a sparse majority data set that empirically validates our theoretical findings.
- Layer normalization. arXiv preprint arXiv:1607.06450.
- Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems, 30.
- Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
- An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
- Dudley, R. M. (1967). The sizes of compact subsets of hilbert space and continuity of gaussian processes. Journal of Functional Analysis, 1(3):290–330.
- Inductive biases and variable creation in self-attention mechanisms. In International Conference on Machine Learning, pages 5793–5831. PMLR.
- l∞\infty∞ vector contraction for rademacher complexity. arXiv preprint arXiv:1911.06468, 6.
- What can a single attention layer learn? a study through the random features lens. arXiv preprint arXiv:2307.11353.
- Generalization and representational limits of graph neural networks. In International Conference on Machine Learning, pages 3419–3430. PMLR.
- Size-independent sample complexity of neural networks. In Conference On Learning Theory, pages 297–299. PMLR.
- On the complexity of linear prediction: Risk bounds, margin bounds, and regularization. Advances in neural information processing systems, 21.
- Gpt-4 passes the bar exam. Available at SSRN 4389233.
- Fat-shattering dimension of k𝑘kitalic_k-fold maxima. arXiv preprint arXiv:2110.04763.
- Probability in Banach Spaces: isoperimetry and processes, volume 23. Springer Science & Business Media.
- Generalization bounds for convolutional neural networks. arXiv preprint arXiv:1910.01487.
- Comparison of lstm, transformers, and mlp-mixer neural networks for gaze based human intention prediction. Frontiers in Neurorobotics, 17:1157957.
- Pisier, G. (1981). Remarques sur un résultat non publié de b. maurey. Séminaire d’Analyse fonctionnelle (dit” Maurey-Schwartz”), pages 1–12.
- Understanding Machine Learning: From Theory to Algorithms. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press.
- Truong, L. V. (2022). On rademacher complexity-based generalization bounds for deep learning. arXiv preprint arXiv:2208.04284.
- Attention is all you need. Advances in neural information processing systems, 30.
- Learning deep transformer models for machine translation. arXiv preprint arXiv:1906.01787.
- Statistically meaningful approximation: a case study on approximating turing machines with transformers. Advances in Neural Information Processing Systems, 35:12071–12083.
- Multistep short-term wind speed forecasting using transformer. Energy, 261:125231.
- Understanding and improving layer normalization. Advances in Neural Information Processing Systems, 32.
- Zhang, T. (2002). Covering number bounds of certain regularized linear function classes. Journal of Machine Learning Research, 2(Mar):527–550.
- An analysis of attention via the lens of exchangeability and latent variable models. arXiv preprint arXiv:2212.14852.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.