Performance-Aware Contextual Embeddings

• Performance-aware contextual embeddings are techniques that design word representations by integrating context types such as window, dependency, and substitute to enhance downstream task performance.
• Optimizing embedding dimensionality and using concatenation of complementary embeddings have proven effective in improving results across intrinsic and extrinsic NLP tasks.
• Incorporating weighted substitute-based modeling refines context sensitivity, thereby enhancing performance in tasks like word sense disambiguation and lexical substitution.

Performance-aware contextual embeddings refer to the design, selection, and optimization of word or token representations whose construction explicitly considers the requirements—and empirical successes—of downstream tasks, rather than relying solely on intrinsic linguistic properties or generic evaluation metrics. These embeddings are contextual in the sense that their representations for a word or token are tuned to its surrounding context and are performance-aware in how choices of context type, dimensionality, combination method, and training algorithm are selected or adapted to maximize task-specific utility.

1. Context Types in Embedding Learning

Performance-aware contextual embeddings are shaped fundamentally by the type of context used during training. Three primary context definitions are widely examined:

• Window-based contexts (Wn): Classic skip-gram models define the context of a target word as the set of words within a symmetric window of size $n$ (e.g., $n=1,5,10$). Empirically, large windows (e.g., W10) capture topical or associative similarities (words in related topics), while smaller windows (W1) produce embeddings more attuned to functional or syntactic similarity.
• Dependency-based contexts (DEP): Contexts are defined by syntactic relations from dependency parses, linking words through grammatical functions (subject, object, modifier, etc.) rather than proximity. These embeddings excel at encoding functional similarity (words with similar roles).
• Substitute-based contexts (SUB): Contexts are derived from weighted sets of plausible substitute words, assigned probabilities via LLMs. Substitute contexts enable the fine-grained learning of functionally equivalent words, offering a probabilistic and more nuanced reflection of context than pure surface co-occurrence.

The choice of context type has direct consequences for performance across diverse tasks. For instance, embeddings trained using large windows are optimal for topical similarity benchmarks (WordSim-353-R), whereas dependency-based and substitute-based embeddings systematically outperform others on functional similarity tasks (SimLex-999, WordSim-353-S).

2. Dimensionality and Its Task-Dependent Effects

The dimension of word embeddings is a key axis of optimization. Results indicate sharply different behaviors depending on task type:

• Intrinsic tasks (e.g., word similarity and analogy): Performance increases monotonically with greater dimensions, typically saturating around 300.
• Extrinsic tasks (e.g., parsing, named entity recognition, sentiment classification): Optimal dimensions are typically lower; values as modest as 50 can suffice, while higher dimensionality may even harm performance due to overfitting or feature sparsity.
• The relationship between task type, context, and optimal dimensionality is nontrivial and mandates downstream validation. Superior intrinsic performance does not guarantee extrinsic gains.

3. Combining Contextual Embeddings: Beyond Single-Space Representations

When single embeddings no longer yield performance improvements with increasing dimensionality, combining complementary embeddings has been shown to yield additional gains:

• Concatenation: Simply concatenating vectors trained with different context types generally produces the highest benefit for challenging extrinsic tasks, often outperforming more complex combination or reduction techniques.
• SVD and CCA: Contrary to expectations from prior literature, combining embeddings using singular value decomposition or canonical correlation analysis does not lead to further improvement and can even degrade extrinsic task performance—likely due to dilution of contextually specialized features.

Guideline: Concatenate after exhausting performance gains from boosting dimension within a single context type, especially for tasks where combined topical and functional similarity are beneficial.

4. Weighted Context Modeling and Probabilistic Training

The weighted substitute-based skip-gram model introduces a key advancement for capturing fine-grained, probabilistic context cues:

$$L = \sum_{(t, c) \in \text{PAIRS}} \alpha_{t,c} \cdot L_{t,c}$$

where $\alpha_{t,c}$ is the conditional probability $p(c|C)$ obtained from a LLM and $L_{t,c}$ is the standard negative sampling loss.

This framework enables embeddings to be tuned by the likelihood of substitute contexts rather than mere occurrence, producing embeddings that excel in cases where context involves graded or ambiguous possibilities (e.g., lexical substitution or word sense disambiguation). These probabilistic approaches more closely align with cognitive theories of meaning and more effectively resolve polysemous or multi-function terms.

5. Performance-Driven Embedding Selection and Application

The following recommendations for performance-aware contextual embedding design directly reflect empirical findings:

1. Intrinsic tasks are insufficient: Do not optimize solely for word similarity or analogy scores, as these do not consistently predict downstream, task-centric performance.
2. Align context type with task requirements: For semantics-heavy tasks, use large-window embeddings; for syntactic or functional tasks, prefer dependency- or substitute-based embeddings.
3. Tune dimensionality to the task: Avoid defaulting to "bigger is better." Supervised tasks often reach optimal performance at lower dimensions.
4. Combine embeddings after saturating single-type gains: Use concatenation to merge complementary embeddings (e.g., topical + functional) when performance plateaus with dimension increases.
5. Leverage weighted context modeling for complex context: The substitute-based, weighted skip-gram model is particularly beneficial for tasks that require acute sensitivity to context or meaning ambiguity.

6. Empirical Illustration: Task–Context Matrix

Task Type	Best Context Type	Notes
Topical Similarity	Large Window (W10)	Intrinsic, WordSim-353-R
Functional Similarity	Dependency (DEP), Substitute (SUB)	Intrinsic, SimLex-999, WordSim-353-S
Dependency Parsing	Dependency (DEP), W1, SUB	Syntactic
NER, Coreference, Sentiment	Varies; often combinations best	Gains from tuning and concatenation

This mapping stresses the necessity of careful context and architecture selection—optimal choices are task-specific and sometimes counterintuitive.

7. Implications and Best Practices for System Builders

Performance-aware contextual embeddings are not a fixed set of vectors but a process—a workflow involving thoughtful tuning and evaluation. System designers are strongly advised to:

• Validate embeddings on the genuine downstream tasks of interest, not just on generic benchmarks.
• Use context types matched to the dominant linguistic phenomena of the task.
• Refrain from indiscriminately increasing embedding dimensionality; instead, diagnose performance plateaus.
• Consider embedding concatenation to synthesize information sources, especially for complex downstream classifiers able to select relevant subspaces.
• Choose weighted or substitute-based context modeling where the downstream phenomena (e.g., sense disambiguation, context-sensitive prediction) warrant it.

In summary, performance-aware contextual embeddings require architectural and training choices that are carefully tailored to the performance needs of each application, moving beyond the simplistic deployment of off-the-shelf embeddings or one-size-fits-all design. This precision fosters superior outcomes across the spectrum of modern NLP tasks.