Revisiting Word Embeddings in the LLM Era (2402.11094v3)

Abstract: LLMs have recently shown remarkable advancement in various NLP tasks. As such, a popular trend has emerged lately where NLP researchers extract word/sentence/document embeddings from these large decoder-only models and use them for various inference tasks with promising results. However, it is still unclear whether the performance improvement of LLM-induced embeddings is merely because of scale or whether underlying embeddings they produce significantly differ from classical encoding models like Word2Vec, GloVe, Sentence-BERT (SBERT) or Universal Sentence Encoder (USE). This is the central question we investigate in the paper by systematically comparing classical decontextualized and contextualized word embeddings with the same for LLM-induced embeddings. Our results show that LLMs cluster semantically related words more tightly and perform better on analogy tasks in decontextualized settings. However, in contextualized settings, classical models like SimCSE often outperform LLMs in sentence-level similarity assessment tasks, highlighting their continued relevance for fine-grained semantics.

Analyzing the Latent Vector Semantics of LLM-Generated Word Embeddings

Introduction to Embedding Models and Semantic Analysis

The evolution of word embedding techniques has been a focal point in NLP research since the advent of models like Word2Vec and GLoVe. The introduction of transformer-based architectures and, subsequently, LLMs, has significantly expanded the scope of embedding models, facilitating the creation of embeddings not only for words but also for longer text sequences. Despite the advancements, the fundamental issue of generating meaningful word embeddings for effective context understanding and robust LLMing remains essential.

Experimentation with Modern Embedding Models

The paper analyzed a spectrum of embedding models, categorizing them into "LLM-based" models with over 1 billion parameters and "Classical" models with fewer than 1 billion parameters. Notable among the tested models were the LLaMA2-7B, OpenAI's ADA-002, and Google's PaLM2 from the LLM category, and LASER, Universal Sentence Encoder (USE), and SentenceBERT (SBERT) from the classical category. These models were evaluated using a comprehensive comparison framework examining the latent vector semantics they produce.

Comparative Analysis of Embedding Models

Word-Pair Similarity Evaluation

The paper analyzed cosine similarity distributions between pairs of words categorized as semantically related, morphologically related, and unrelated. It found that LLMs, particularly ADA and PaLM, demonstrated a higher expected cosine similarity for random pairs of words compared to classical models. However, SBERT showed a remarkable capability to distinguish semantically related pairs almost as effectively as the heavier LLMs, despite being a lighter model.

Word Analogy Task Performance

The paper further scrutinized the models' performance on word analogy tasks using the Bigger Analogy Test Set (BATS). LLMs like ADA and PaLM emerged as superior in performing these tasks, further demonstrating their advanced semantic understanding. Interestingly, SBERT was frequently ranked third, indicating that it could serve as an efficient alternative in scenarios where using large models like PaLM and ADA might not be feasible due to resource constraints.

Findings and Implications

• LLM-based embeddings tend to generate semantically richer representations, offering higher accuracy in word analogy tasks compared to classical models.
• SBERT, despite its relative simplicity, can closely compete with more sophisticated LLMs in distinguishing semantically related word pairs, making it a practical choice for resource-constrained environments.
• While LLMs show promise in improving the semantic understanding encapsulated in word embeddings, their significant resource requirements pose a challenge for widespread adoption.
• The presence of meaningful agreement between the embeddings generated by SBERT and those generated by ADA-002 implies possible convergences in the semantic spaces captured by vastly different models.

Future Directions in AI and LLMing

The continuous refinement of word and sentence embedding models, especially with the incorporation of LLMs, suggests an optimistic future for NLP applications. However, the findings advocate for a balanced approach to leveraging these models, considering both the qualitative improvements they offer and the practical constraints of deploying large-scale models. Future research could explore optimizing the performance of lighter models like SBERT for broader applicability or developing methods to reduce the computational expenses of LLMs without compromising their semantic understanding capabilities.

Conclusion

This paper presents a thorough investigation into the latent semantic differences and similarities between classical and LLM-based word embeddings. By systematically analyzing and comparing these embeddings through word-pair similarity distributions and word analogy tasks, it contributes significantly to our understanding of the evolving landscape of LLMs. The nuanced insights it offers into the performance trade-offs and potential applications of different embedding models serve as a valuable guide for future research in the field of NLP.