Contextual Word Embeddings Overview

Updated 10 November 2025

Contextual word embeddings are high-dimensional, context-dependent vectors generated by neural language models, effectively capturing polysemy and syntactic nuances.
Models like ELMo, BERT, and RoBERTa use architectures such as BiLSTMs and transformers with subword tokenization to integrate global context and token-specific details.
These embeddings drive applications in semantic shift detection, word sense disambiguation, and term extraction, while posing challenges like sensitivity to minor orthographic changes.

Contextual word embeddings constitute a paradigm-shifting methodology in natural language processing, replacing static, type-level encodings with context-sensitive, token-level vector representations. In modern architectures such as ELMo, BERT, RoBERTa, and their derivatives, each token in a sequence is mapped to a high-dimensional vector whose value is a learned function of the entire surrounding context. This context dependence enables the modeling of polysemy, fine-grained semantic nuance, and syntactic ambiguity, resulting in measurable performance improvements across linguistic tasks ranging from syntactic parsing to coreference and semantic shift detection. At the same time, probing and analytic studies have revealed that these representations are not pure instantiations of abstract meaning; their semantic reliability is mediated by factors including subword tokenization, surface-form preservation, contextual variance, and the architecture of the underlying neural model.

1. Formal Properties and Theoretical Foundations

Contextual word embeddings (CWEs) are context-dependent vector representations generated by neural LLMs for individual word tokens. Given a sentence context $C$ containing word $w$ , a CWE is expressed as $e(w, C) \in \mathbb{R}^d$ , with $d$ typically in the hundreds or thousands. In contrast to static embeddings such as word2vec or GloVe, which assign a fixed vector $v_w$ to each word type $w$ irrespective of context, CWEs encode token-specific lexical and contextual information by integrating sentence-level features via the deep neural network's architecture and parameters (Liu et al., 2020).

Architectures for generating CWEs include:

BiLSTM-based models (e.g., ELMo): Each layer processes input token sequences bidirectionally, producing context-dependent hidden states $h_{k,i}$ for each layer $i$ and position $k$ , combined into a single embedding via a learned scalar mix (Peters et al., 2018).
Transformer-based models (e.g., BERT, RoBERTa): Self-attention layers integrate information from all tokens at each position. BERT uses a masked LLM (MLM) and next-sentence prediction pretraining, while RoBERTa removes NSP and adopts dynamic masking for further improvements (Liu et al., 2020).
Subword tokenization: Input tokens are split via Byte Pair Encoding (BPE) or WordPiece into varying numbers of subwords, which are then individually embedded and pooled (often via mean-pooling) to yield the final token vector.

Formally, if a word $w$ is split into $n$ subword pieces with final hidden states $\{h_1, \ldots, h_n\}$ , then $e(w, C) = \frac{1}{n} \sum_{k=1}^n h_k$ (Matthews et al., 8 Aug 2024). Cosine similarity between such embeddings is widely interpreted as a proxy for semantic similarity: $\cos(e(w_1, C_1), e(w_2, C_2))$ .

2. Semantic Interpretation and Model Layer Analysis

The underlying assumption in the use of CWEs is that the model's hidden states primarily encode semantic information, flexibly adapting to local context to disambiguate senses and encode meaningful relatedness. Empirical evidence, however, points to a layered organization of linguistic abstraction across neural model depth (Peters et al., 2018):

Input and lower layers predominantly encode surface form and morphological features. For example, character-level CNNs or initial layers concentrate on spelling, affixes, and simple parallels to static embedding analogies.
Middle layers exhibit a transition to local syntax, enabling linear classifiers on these representations to achieve near-maximal part-of-speech tagging and unsupervised span classification accuracy.
Upper and final layers are specialized for long-range syntax and deep semantics, including coreference, semantic relatedness, and discourse structure.

Task-specific performance gains accrue from integrating multiple layers, typically via learned mixes (e.g., ELMo's scalar mix), rather than relying solely on the final layer.

Furthermore, continuous-space representations from pre-trained models such as BERT quantitatively correlate with human-graded distinctions among word senses: homonymous senses (unrelated meanings, e.g., “bat” as animal vs. tool) occupy more distant regions of embedding space than polysemous senses (closely related, e.g., “chicken” as animal vs. meat). Spearman's rank correlation between model-derived and human-relatedness matrices yields $\rho=0.565$ across a collection of WordNet-anchored word senses (Nair et al., 2020).

3. Probing, Robustness, and Orthographic Sensitivity

Recent analytic work challenges the notion that CWEs exclusively capture semantics. In particular, Matthews et al. (Matthews et al., 8 Aug 2024) demonstrate that:

CWEs are highly sensitive to minor orthographic perturbations, such as a single character swap in the input word, which can induce a 40–60 percentage point drop in cosine similarity between embeddings, even when the word is embedded in a 100-word context.
This sensitivity is tightly coupled to subword tokenization: words mapped to a small number of subword tokens ( $n_\text{subwords} = 1$ or $2$) display pronounced vulnerability, with Spearman $\rho \approx 0.2–0.5$ between normal and noised variants, as opposed to $\rho \rightarrow 1.0$ for words decomposed into many subwords.
Context alone cannot reliably “repair” the embedding space after such a perturbation. PLMs such as RoBERTa and BLOOM exhibit almost no contextual repair, while BERT and XLNet recover only partially.

The mechanism for this phenomenon is rooted in tokenizer-induced segmentation differences and frequency bias. A minor change in spelling can shift an in-vocabulary word (single token, frequent, well-trained) into an out-of-vocabulary multi-subword composition, mapping to rare subword vectors that reside elsewhere in the embedding space. Semantic cues from context are inadequate to override the dominant signal from initial token-level discrepancies, which propagate throughout the network layers.

Implications: CWEs encode significant surface-form information. In practice, up to half of a word's “meaning signal” in the embedding can be erased by a trivial typo. Consequently, representational similarity is not a reliable semantic proxy in noisy, low-resource, or user-generated text (Matthews et al., 8 Aug 2024).

4. Variance, Contextual Consistency, and Sense Representation

The distributional geometry of CWEs reflects both desired and undesired variance. Models such as BERT and ELMo generate embeddings that cluster tokens sharing both word form and sense label, with mean cosine similarity (Sim_ss) between same-sense tokens ranging from 0.45 (ELMo) to 0.99 (GPT-2), exceeding random baselines in all cases (Wang et al., 2022).

Additional findings:

Part-of-speech and polysemy: Nouns yield the highest sense-consistency; polysemous words with many senses show greater cross-context variance and less reliable clustering.
Sentence length and position: Embeddings for the same sense in short sentences are more consistent; first-word tokens manifest position bias, with spuriously high similarity across contexts, especially in models with absolute positional embeddings.
Contextual masking: Masking the surface form during embedding extraction reduces sense-consistency only marginally, confirming the importance of surrounding context in signal recovery.

The position bias can be mitigated via prompting—introducing neutral tokens or phrases—thus reducing systematic variance not attributable to semantic content.

5. Applications, Pitfalls, and Extensions

Applications

Semantic shift detection: Statistical protocols combining sample-based permutation tests and false-discovery-rate (FDR) control can robustly identify true lexical semantic shifts in diachronic and domain adaptation scenarios (Liu et al., 2021). Averaged CWEs serve as inputs to compute shift scores, with permutation-based p-values and FDR controlling the expected false discovery proportion across the vocabulary.
Word Sense Disambiguation: Sense-level embeddings for all WordNet synsets can be constructed by propagating contextual embeddings over the lexical graph, then applying k-NN or cosine similarity rules to new tokens. This approach achieves state-of-the-art F1 across several standard all-words WSD benchmarks (Loureiro et al., 2019).
Term extraction: Augmenting classical term extraction pipelines with contextual features (e.g., domain-specific ELMo vectors, cosine similarity to general-language prototypes) yields measurable gains in F1 for low-frequency and domain-specific term identification (Repar et al., 24 Feb 2025).
Cross-lingual mapping: When using contextual mapping for bilingual lexicon induction, naïvely averaging token embeddings over multi-sense words can introduce alignment noise. Empirical remedies include exclusion (noise removal) or cluster-level anchor replacement, both of which improve unsupervised alignment precision by up to 12 percentage points (Zhang et al., 2019).
Hybridization: Concatenating contextual embeddings with knowledge graph embeddings leverages their complementary strengths—capturing either distributional or relational patterns—and yields consistent improvement in semantic-typing tasks (micro/macro F1 gains up to 5 points over single-source baselines) (Dieudonat et al., 2020).

Pitfalls and Mitigations

Surface-form leakage: Orthographic perturbations can severely degrade embedding similarity, especially when subword units change. Using character-level or tokenizer-free models (e.g., CANINE), or regularization strategies to enforce tokenizer robustness, are proposed mitigations (Matthews et al., 8 Aug 2024).
Variance and position effects: Distance-based clustering tasks (e.g., WiC-style WSD) are susceptible to position and context-length biases. Prompting or pooling over multiple context positions can alleviate these confounds (Wang et al., 2022).
Model selection: Transformer and CNN architectures offer superior inference speed over LSTMs, with only marginal drops in accuracy, but all architectures exhibit comparable depth-wise specialization and benefit from scalar-mix approaches (Peters et al., 2018).

6. Extensions and Alternative Frameworks

Research continues to extend and reframe the conceptual foundation of contextual embeddings:

Dynamic Contextualized Word Embeddings: By parameterizing type-level embeddings as a function of extralinguistic metadata—time and social graph—dynamic models interpolate between strictly context-derived token representations and temporally/societally aware type vectors. Quantitatively, incorporating temporal and social offsets modestly reduces perplexity on masked language modeling tasks and improves sentiment F1 on dynamic datasets (Hofmann et al., 2020).
Quantum-contextual approaches: An alternative static vector framework situates each word as a unit vector in a Hilbert space, with contexts represented as orthonormal bases (maximal observables). Contextual meaning is recovered by projecting the word vector onto basis vectors of the chosen context, leveraging quantum notions of complementarity and intertwining. This approach provides a geometric analog to polysemy, with fixed word vectors acquiring different semantic interpretations depending on the basis (Svozil, 18 Apr 2025). The mathematical structure supports interpretable context differentiation and potentially more efficient inference, though practical training and task alignment remain open questions.

Compression and Deployment: Given the computational intensity of full contextual models, methods such as X2Static distill context-derived knowledge from pre-trained transformers into static embeddings with CBOW-style negative sampling, achieving state-of-the-art results on word similarity and classification benchmarks at orders-of-magnitude reduced inference cost (Gupta et al., 2021).

7. Practical Considerations and Recommendations

Empirical studies indicate a three-way trade-off among embedding complexity, annotation cost, and downstream accuracy (Arora et al., 2020). Contextual embeddings yield the largest gains in low-resource regimes, for linguistically complex tasks, high ambiguity, and high OOV rates. In industrial or resource-constrained scenarios with abundant data and simpler language, even random or static embeddings may approach the performance of CWEs, while incurring much lower computational and storage overheads.

Key recommendations:

Audit subword tokenization: Track the number of subword pieces for target words; words with one or two subwords are especially vulnerable to surface-level noise.
Validate semantic similarity measures: In any application interpreting cosine similarity of CWEs as meaningful, perform controlled perturbation (e.g., orthographic swaps) to assess semantic robustness.
For low-resource or noisy data, prefer character-based or neural tokenizer-free models; regularize subword tokenizers or use position-insensitive embeddings.
When deploying highly compressed or on-device systems, consider distilled static embeddings derived from the contextual teacher, balancing between efficiency and task performance.
In cross-lingual and WSD contexts, do not conflate token-level embeddings for multi-sense words; apply clustering or supervised sense detection when possible.

In conclusion, contextual word embeddings have established themselves as the dominant paradigm for lexical representation in NLP, driven by their ability to encode context-dependent meaning and linguistic abstraction. However, their semantic reliability is contingent upon complex interactions among architecture, tokenization, and surface form. Ongoing research seeks to mitigate vulnerabilities, extend the expressive domain (e.g., through dynamic and quantum-contextual frameworks), and optimize deployment for real-world tasks (Matthews et al., 8 Aug 2024, Liu et al., 2020, Peters et al., 2018, Hofmann et al., 2020, Svozil, 18 Apr 2025).