Word Difference Representations

• Word Difference Representations (WDR) are vector-based techniques that capture and quantify semantic differences between words using explicit difference vectors and finite-difference algebra.
• They are applied to discriminative attribute identification, sociolinguistic divergence analysis, and the improvement of language modeling through context-sensitive targets.
• WDR methods combine transparent, interpretable logic with deep neural approaches, enhancing both explanation and predictive accuracy across various NLP tasks.

Word Difference Representations (WDR) are a class of vector-based methods for capturing, quantifying, and exploiting semantic differences between words, attributes, or contexts. These representations are employed across several distinct NLP tasks, from identifying discriminative lexical attributes between concepts to measuring divergence in word interpretations across social groups, and from reparameterizing targets in language modeling objectives to enhancing contextualized predictions. WDRs formalize comparison in vector space via explicit, interpretable difference vectors, centroids, or finite-difference algebra over embedding spaces, and have been instantiated in both transparent symbolic models and neural network architectures (Stepanjans et al., 2019, Hu et al., 2017, Heo et al., 2024).

1. Mathematical Formalizations of Word Difference Representations

WDR construction varies with task and underlying representation but consistently encodes the differential semantic properties between word pairs or sets via vector arithmetic.

Discriminative Attribute Identification (Stepanjans et al., 2019)

For explicit word vector spaces defined over attributes from distinct knowledge sources $s \in \{\text{DBM}, \text{VFM}, \text{CKG}\}$, each term $t$ is associated with a sparse vector $v^s(t)\in\mathbb{R}^{d^s}$, normalized as

$\hat v^s(t) = \frac{v^s(t)}{\|v^s(t)\|_1}$

A WDR for a pivot $t_p$ vs. a contrast $t_c$ in source $s$ is then: $\delta^s(t_p, t_c) = \hat v^s(t_p) - \hat v^s(t_c)$ These source-specific WDRs are linearly (or Boolean) combined, yielding a score for attribute $f$: $\text{score}(f) = \delta(t_p, t_c)[f]$ with discriminativeness predicted if $t$0.

Divergent Word Interpretations (Hu et al., 2017)

Given corpora $t$1, $t$2 for two social groups and their respective embedding spaces $t$3, $t$4, for word $t$5:

• Extract top-$t$6 similar words (interpreting sets $t$7, $t$8) with similarity weights ($t$9, $v^s(t)\in\mathbb{R}^{d^s}$0).
• Project all $v^s(t)\in\mathbb{R}^{d^s}$1 into a shared global embedding $v^s(t)\in\mathbb{R}^{d^s}$2. Compute weighted centroids:

$v^s(t)\in\mathbb{R}^{d^s}$3

• The WDR is the cosine distance between centroids:

$v^s(t)\in\mathbb{R}^{d^s}$4

Contextualized Targets in Language Modeling (Heo et al., 2024)

For a sequence $v^s(t)\in\mathbb{R}^{d^s}$5 and contextually parameterized encoder embeddings $v^s(t)\in\mathbb{R}^{d^s}$6:

• The $v^s(t)\in\mathbb{R}^{d^s}$7-th level finite difference WDR is recursively:

$v^s(t)\in\mathbb{R}^{d^s}$8

• The conjugate term for reconstructing $v^s(t)\in\mathbb{R}^{d^s}$9:

$\hat v^s(t) = \frac{v^s(t)}{\|v^s(t)\|_1}$0

• At training, the model predicts $\hat v^s(t) = \frac{v^s(t)}{\|v^s(t)\|_1}$1, and the actual target embedding is recomposed as:

$\hat v^s(t) = \frac{v^s(t)}{\|v^s(t)\|_1}$2

2. Data Sources and Scope of Representation

The informativeness and interpretability of WDRs critically depend on the sources used to define base vectors and features:

• Definition-Based Model (DBM): Lexicographic glosses parsed into semantic roles from WordNet or Wiktionary; supports extraction of essential and absolute logical attributes.
• Visual Feature Model (VFM): VisualGenome scene graphs encoding object-attribute and object-relation triples; captures incidental, sensory, and spatial features.
• Commonsense Knowledge Graph (CKG): ConceptNet triple relations, encompassing lexico-semantic and affordance information absent from text or images (Stepanjans et al., 2019).
• Custom Embedding Spaces: For social interpretation divergence, group-specific corpora yield group-specific word2vec/skipgram spaces, which are projected into a global coordinate system that supports cross-group semantic comparison (Hu et al., 2017).
• Causal LLM Embeddings: For language modeling, embedding matrices from transformer/CLM models serve as the base for finite-difference WDRs. These are parameterization-invariant under the CLM shared-embedding regime (Heo et al., 2024).

Combinations of these sources enable explicit coverage of attribute types and semantic differentiae, supporting both deterministic logic and data-driven contextual variation.

3. Procedures for Extraction, Scoring, and Classification

Discriminative Attribute Identification (Stepanjans et al., 2019)

1. Vector Construction: For each term $\hat v^s(t) = \frac{v^s(t)}{\|v^s(t)\|_1}$3, attributes are extracted, lemmatized, assigned semantic roles, and $\hat v^s(t) = \frac{v^s(t)}{\|v^s(t)\|_1}$4-weighted.
2. Normalization: L1 normalization of each source-specific vector.
3. Difference Calculation: Compute $\hat v^s(t) = \frac{v^s(t)}{\|v^s(t)\|_1}$5.
4. Scoring Rule: Attribute $\hat v^s(t) = \frac{v^s(t)}{\|v^s(t)\|_1}$6 is deemed discriminative iff $\hat v^s(t) = \frac{v^s(t)}{\|v^s(t)\|_1}$7.
5. Decision Transparency: No statistical learning; decision is a deterministic lookup.

Social Divergence via WDR (Hu et al., 2017)

1. Corpus Partition: Stratify corpora by demographic.
2. Embedding Training: Learn separate spaces $\hat v^s(t) = \frac{v^s(t)}{\|v^s(t)\|_1}$8; train global $\hat v^s(t) = \frac{v^s(t)}{\|v^s(t)\|_1}$9.
3. Construct Interpreting Sets: Extract top-$t_p$0 neighbors per word per group.
4. Projection and Centroiding: Map neighbor vectors into $t_p$1 and compute centroids.
5. Semantic Distance: Compute cosine distance as divergence/WDR.
6. Manual Validation: Inspect high-WDR words for sociolinguistic interpretability.

Contextual CLM with WDR Targets (Heo et al., 2024)

1. N-gram Extension: For $t_p$2, predict not word embeddings, but finite-difference representations via per-step MLP heads.
2. Target Recomposition: Add the detached conjugate term to yield a target embedding for softmax.
3. Loss Aggregation: Combine losses over all predicted future steps, emphasizing the standard next-word objective.
4. Ensembling at Inference: Fuse next-word and future-word predictions to enhance contextual awareness.

4. Evaluation and Empirical Results

Attribute Discrimination (Stepanjans et al., 2019)

• Benchmark: SemEval 2018 Task 10 (2,340 attribute triples).
• Metric: Micro-averaged F1; model achieves 0.69 (top explainable system among 18).
• Component-wise Analysis: DBM excels on Sensory/Incidental, CKG on Logical/Essential, VFM fills unique visual and spatial gaps.
• Complementarity: Combined model yields 17–48% recall gain versus best single source.

Socio-semantic Divergence (Hu et al., 2017)

• Analysis Protocol: Rank WDR for every word, manually inspect top divergences.
• Findings: Top divergences correlate with known gender/region semantics: e.g., “bitter” (taste vs. emotion), “windows” (OS vs. architecture).
• Validation: Primarily qualitative; no large-scale sweep of precision/recall.

Language Modeling (Heo et al., 2024)

• Perplexity Reduction: WDR 4-gram models achieve lower next-word PPL compared to both baselines and naïve N-gram extension (e.g., 44.4 vs. 55.0 on PTB-Transformer).
• Generalization: WDR increases gradient diversity during training, promoting improved convergence and robustness.
• Translation Quality: NMT BLEU is improved by 0.5–1.0 points when using WDR ensemble targets compared to classical methods.

5. Interpretability, Transparency, and Explanation

WDR frameworks provide enhanced interpretability through explicit, human-readable vector spaces and deterministic, traceable inference steps:

• Attribute Explanation: For discriminative attributes, the triggering source and logical path (e.g., WordNet gloss role, VFM image set, ConceptNet relation) is explicitly returned alongside the classification result (Stepanjans et al., 2019).
• Interpretation Divergence: The shift in centroids and the constituent neighbor words give concrete sociolinguistic bases for observed divergence (Hu et al., 2017).
• Language Modeling Contextualization: WDR targets enforce contextual parameterization of the prediction head, as the finite difference is context-sensitive, and the algebraic recomposition guarantees transparency in target construction (Heo et al., 2024).

A table summarizing transparency and explainability properties across applications:

Model	Transparency Mechanism	Decision Auditability
Discriminative	Explicit attribute vectors, path	Full: role/feature trace
Socio-semantic	Interpreting neighbors, centroids	High: list of neighbors
CLM/NMT N-gram	Algebraic WDR, MLP heads	Moderate: finite-diff meta

6. Limitations and Extensions

Key constraints, as documented in the literature:

• Representation Dependence: Effectiveness is bounded by the coverage and granularity of base sources or corpora. Lexicographic and knowledge-graph spaces are incomplete for noisy, open-domain attributes (Stepanjans et al., 2019). Social-divergence measures depend on robust neighborhood structure, sensitive to data scale and embedding hyperparameters (Hu et al., 2017).
• Evaluation Limitations: Attribute discrimination offers quantitative ground-truth comparison, but social WDR applications are validated mainly via manual inspection and sociolinguistic plausibility, with no statistical significance or error analysis available.
• Contextual Variation: In language modeling, the geometric invertibility of WDRs ensures algebraic equivalence but may amplify embedding noise at high $t_p$3 or in highly non-linear contextual settings (Heo et al., 2024).
• Potential Extensions: Applying WDR mechanisms to richer contextualized embeddings (BERT-family), broader demographic strata or temporally dynamic corpora, and leveraging automatable ground-truths in corpus-based WDR validation are suggested as future research avenues (Hu et al., 2017).

7. Connections and Comparative Utility

WDR methods bridge explicit, interpretable feature engineering and deep contextualization, synthesizing strengths of both paradigms:

• In discriminative attribute modeling, WDR confers explainability and explicit logical structure lacking in neural black-box approaches, enabling auditability and modularity (Stepanjans et al., 2019).
• In socio-semantic analysis, WDR offers a data-driven metric for group linguistic divergence, complementing census- or survey-based techniques with scalable, empirical quantification (Hu et al., 2017).
• As a surrogate for fixed embedding targets in LLMs and NMT, WDR regularizes prediction heads towards learning dynamic, local differences, empirically shown to improve performance while increasing training gradient diversity and convergence speed (Heo et al., 2024).

Through these instantiations, WDR frameworks have established themselves as flexible, interpretable, and empirically effective mechanisms for operationalizing and measuring semantic difference across a wide array of computational linguistic contexts.