Papers
Topics
Authors
Recent
Detailed Answer
Quick Answer
Concise responses based on abstracts only
Detailed Answer
Well-researched responses based on abstracts and relevant paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses
Gemini 2.5 Flash
Gemini 2.5 Flash 30 tok/s
Gemini 2.5 Pro 46 tok/s Pro
GPT-5 Medium 18 tok/s Pro
GPT-5 High 12 tok/s Pro
GPT-4o 91 tok/s Pro
Kimi K2 184 tok/s Pro
GPT OSS 120B 462 tok/s Pro
Claude Sonnet 4 36 tok/s Pro
2000 character limit reached

Bidirectional Attention as a Mixture of Continuous Word Experts (2307.04057v2)

Published 8 Jul 2023 in cs.CL, cs.LG, and stat.ML

Abstract: Bidirectional attention $\unicode{x2013}$ composed of self-attention with positional encodings and the masked LLM (MLM) objective $\unicode{x2013}$ has emerged as a key component of modern LLMs. Despite its empirical success, few studies have examined its statistical underpinnings: What statistical model is bidirectional attention implicitly fitting? What sets it apart from its non-attention predecessors? We explore these questions in this paper. The key observation is that fitting a single-layer single-head bidirectional attention, upon reparameterization, is equivalent to fitting a continuous bag of words (CBOW) model with mixture-of-experts (MoE) weights. Further, bidirectional attention with multiple heads and multiple layers is equivalent to stacked MoEs and a mixture of MoEs, respectively. This statistical viewpoint reveals the distinct use of MoE in bidirectional attention, which aligns with its practical effectiveness in handling heterogeneous data. It also suggests an immediate extension to categorical tabular data, if we view each word location in a sentence as a tabular feature. Across empirical studies, we find that this extension outperforms existing tabular extensions of transformers in out-of-distribution (OOD) generalization. Finally, this statistical perspective of bidirectional attention enables us to theoretically characterize when linear word analogies are present in its word embeddings. These analyses show that bidirectional attention can require much stronger assumptions to exhibit linear word analogies than its non-attention predecessors.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (35)
  1. Transformers learn to implement preconditioned gradient descent for in-context learning. arXiv preprint arXiv:2306.00297, 2023.
  2. In-context learning through the Bayesian prism. arXiv preprint arXiv:2306.04891, 2023.
  3. What learning algorithm is in-context learning? Investigations with linear models. In International Conference on Learning Representations, 2022.
  4. C. Allen and T. Hospedales. Analogies explained: Towards understanding word embeddings. In International Conference on Machine Learning, 2019.
  5. S. Ö. Arik and T. Pfister. TabNet: Attentive interpretable tabular learning. In AAAI Conference on Artificial Intelligence, 2021.
  6. A latent variable model approach to PMI-based word embeddings. In Association for Computational Linguistics, 2016.
  7. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. arXiv preprint arXiv:2306.04637, 2023.
  8. K. Church and P. Hanks. Word association norms, mutual information, and lexicography. Computational Linguistics, 1990.
  9. Why can GPT learn in-context? Language models secretly perform gradient descent as meta optimizers. In Association for Computational Linguistics, 2023.
  10. BERT: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics - Human Language Technologies, 2019.
  11. Inductive biases and variable creation in self-attention mechanisms. In International Conference on Machine Learning, 2022.
  12. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021.
  13. Towards understanding linear word analogies. In Association for Computational Linguistics, 2019.
  14. Skip-gram - Zipf + uniform = vector additivity. In Association for Computational Linguistics, 2017.
  15. Revisiting deep learning models for tabular data. In Neural Information Processing Systems, 2021.
  16. TabTransformer: Tabular data modeling using contextual embeddings. arXiv preprint arXiv:2012.06678, 2020.
  17. Adaptive mixtures of local experts. Neural Computation, 1991.
  18. M. Joseph. PyTorch Tabular: A framework for deep learning with tabular data. arXiv preprint arXiv:2104.13638, 2021.
  19. O. Levy and Y. Goldberg. Neural word embedding as implicit matrix factorization. In Neural Information Processing Systems, 2014.
  20. How do transformers learn topic structure: Towards a mechanistic understanding. In International Conference on Machine Learning, 2023.
  21. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  22. Distributed representations of words and phrases and their compositionality. In Neural Information Processing Systems, 2013.
  23. A mixture of h−1ℎ1h-1italic_h - 1 heads is better than hℎhitalic_h heads. In Association for Computational Linguistics, 2020.
  24. GloVe: Global vectors for word representation. In Empirical Methods in Natural Language Processing, 2014.
  25. Language models are unsupervised multitask learners. OpenAI Blog, 2019.
  26. AutoInt: Automatic feature interaction learning via self-attentive neural networks. In ACM International Conference on Information and Knowledge Management, 2019.
  27. Attention word embedding. arXiv preprint arXiv:2006.00988, 2020.
  28. M. Sugiyama and A. J. Storkey. Mixture regression for covariate shift. In Neural Information Processing Systems, 2006.
  29. Transformer dissection: An unified understanding for transformer’s attention via the lens of kernel. In Empirical Methods in Natural Language Processing and International Joint Conference on Natural Language Processing, 2019.
  30. BERT is to NLP what AlexNet is to CV: Can pre-trained language models identify analogies? In Association for Computational Linguistics and International Joint Conference on Natural Language Processing, 2021.
  31. Attention is all you need. In Neural Information Processing Systems, 2017.
  32. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, 2023.
  33. Large language models are latent variable models: Explaining and finding good demonstrations for in-context learning. In Neural Information Processing Systems, 2023.
  34. An explanation of in-context learning as implicit Bayesian inference. In International Conference on Learning Representations, 2021.
  35. Trained transformers learn linear models in-context. In Workshop on Robustness of Zero/Few-Shot Learning in Foundation Models at NeurIPS, 2023.
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-Up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now