---
title: Skip-gram with Negative Sampling
url: https://www.emergentmind.com/topics/skip-gram-with-negative-sampling-sgns
type: topic
---

# Skip-gram with Negative Sampling

Skip-gram with Negative Sampling (SGNS) is a neural probabilistic model for learning distributed representations of words and nodes, and a foundational technique for unsupervised representation learning across natural language processing and network science. SGNS forms the core of word2vec and its derivatives, as well as graph embedding algorithms such as node2vec, LINE, and DeepWalk. The defining principle of SGNS is to maximize, for each observed (target, context) pair, the probability of co-occurrence under a shallow neural model, while repelling representations of randomly sampled “negative” context pairs to avoid trivial solutions. Empirically, SGNS embeddings preserve semantic regularities, encode higher-order co-occurrence structure, and serve as a universal substrate for downstream similarity and classification tasks.

## 1. Mathematical Formulation and Objective

Let $V$ be a vocabulary of $n$ items (e.g., words), and $D$ a collection of observed $(w, c)$ pairs from a training corpus. SGNS learns two sets of $d$-dimensional embeddings, $u_w \in \mathbb{R}^d$ (“input”/“target”) and $v_c \in \mathbb{R}^d$ (“context”/“output”), via the following stochastic binary classification objective:

\[
\ell_{\mathrm{SGNS}}(w, c) = \log \sigma(u_w^\top v_c) + \sum_{i=1}^k \mathbb{E}_{c_i \sim p_n}\left[ \log \sigma(-u_w^\top v_{c_i}) \right]
\]
where $\sigma(x) = 1 / (1 + e^{-x})$ is the sigmoid nonlinearity, $p_n$ is a noise distribution (typically a smoothed or empirical unigram), and $k$ is the number of negative samples per positive pair [1411.5595][1705.09755]. The global objective, summed over all positive pairs, is maximized via stochastic gradient ascent.

Summing over all word–context pairs and writing the empirical distribution as $p(w, c)$, the corpus-level loss is:
\[
L_{\mathrm{SGNS}} = -\sum_{w, c} \left[ p(w, c) \log \sigma(u_w^\top v_c) + k p(w) p_n(c) \log \sigma(-u_w^\top v_c) \right]
\]
This loss decomposes into an attraction term for observed co-occurrences and a repulsion term for negative (random) samples [2002.12005][1411.5595].

## 2. Theoretical Properties and PMI Factorization

A key theoretical insight is that SGNS implicitly factorizes a shifted Pointwise Mutual Information (PMI) matrix. Under ideal conditions ($d \to n$), the optimal inner product satisfies:
\[
u_x^\top v_y = \mathrm{PMI}(x, y) - \log k
\]
where
\[
\mathrm{PMI}(x, y) = \log \frac{p(x, y)}{p(x) p(y)}
\]
This relation establishes SGNS as a stochastic approximation to low-rank matrix factorization of the shifted PMI, unifying it with explicit count-model approaches such as GloVe and SPPMI-SVD [1411.5595][1705.09755][2002.12005].

From the weighted logistic PCA perspective, SGNS can be viewed as fitting a low-rank log-odds model to a matrix of binomial proportions constructed from observed and negatively sampled pairs—each $(w, c)$ cell accumulates $N_{w,c} = n_{w,c} + k n_w p_n(c)$ trials, with “successes” $x_{w,c} = n_{w,c} / N_{w,c}$, yielding [1705.09755]:
\[
\ell = \sum_{w, c} N_{w,c}\left[ x_{w,c} (u_w^\top v_c) - \log(1 + e^{u_w^\top v_c}) \right]
\]

## 3. Optimization, Incrementality, and Regularization

SGNS is universally trained using stochastic gradient descent (SGD) due to the potentially vast number of unique $(w, c)$ pairs and negatives [1411.5595][1704.03956][1906.03586].

**Incremental SGNS:** Recent works have developed “single-pass” SGNS variants for streaming or dynamic data, where the noise distribution $p_n$ is adapted incrementally based on observed frequencies [1704.03956][1906.03586]. Theoretical bounds confirm that, under mild assumptions, incremental SGNS achieves almost the same objective value and embedding quality as conventional multi-pass algorithms, with substantial efficiency gains.

**Regularization:** The SGNS objective is invariant under invertible linear transformations of the embeddings, introducing ambiguity in geometry absent further constraints. Quadratic (Frobenius-norm) regularization removes all but orthogonal ambiguities, yielding uniqueness up to rotation and improving analogy accuracy at higher embedding dimensions [1804.00306].

**Riemannian Optimization:** Advanced algorithms such as RO-SGNS optimize directly over the Riemannian manifold of fixed-rank matrices, providing more stable solutions with higher SGNS objective values and improved word similarity results, especially for large embedding dimensions [1704.08059].

## 4. Noise Distributions and Negative Sampling

The behavior of SGNS is fundamentally governed by the negative sampling distribution $p_n$ [2009.04413][1804.04212][1710.09805]. The most common choice is an “$\alpha$-smoothed” unigram with exponent $\alpha$ in $[0, 1]$ (with $\alpha=3/4$ standard for words, but task-specific in recommendation and graph embedding):

\[
p_n(c) = \frac{f(c)^\alpha}{\sum_{c'} f(c')^\alpha}
\]

Key findings include:
- Uniform $p_n$ leads to slower convergence and poor embeddings.
- Empirical unigram $p_n$ quickly overfits frequent words.
- $3/4$-power smoothing offers a balance between informativeness and variance.
- Adaptive and context-conditional noise distributions (caSGN) further improve performance by selectively targeting “hard” negatives [2009.04413][1710.09805].

In the batch setting, negative sampling efficiently approximates the expensive full softmax normalization [1411.5595].

## 5. Generalizations and Interpretations

**Matrix and Tensor Factorization:** SGNS extends naturally to higher-order data: for time-evolving graphs or multi-relational data, SGNS is generalized to multidimensional tensor factorization, either as explicit Canonical-Polyadic (CP) decompositions or as implicit factorization via higher-order skip-gram objectives [2006.14330].

**Information-Theoretic View:** At optimality, squared Euclidean distance in SGNS embeddings is a monotonic function of $-\mathrm{csPMI}$ (co-occurrence shifted PMI), directly connecting geometric distance to likelihood-based similarity [1810.04882].

**Graph Embedding:** In networks, SGNS serves as a flexible primitive, fitting node embeddings using co-occurrence statistics from random walks or edge lists, with the SGNS negative term approximating a dimension-level re-centering operator [2405.00172]. Dimension regularization can, in specific regimes, replace explicit negative sampling, yielding substantial computational savings while preserving downstream accuracy.

## 6. Empirical Properties and Applications

Empirically, SGNS-derived embeddings perform robustly across disparate domains:

- **Word Embeddings:** High-quality results on analogy, similarity, and analogy transfer tasks. Captures both first- and second-order similarity structure, similar to low-rank SVD but more scalable [1906.02479][1810.04882].
- **Recommendation and Item Embedding:** Sensitivity to hyperparameters such as window size, negative exponent, and subsampling threshold is acute; tuned parameters can yield up to 700% performance improvements over NLP defaults [1804.04212].
- **Graph Embedding:** Supports large-scale, dynamic, and attributed graphs, matching or exceeding static retraining in both link prediction and clustering, with up to 22× speedups for local updates [1906.03586][2405.00172].
- **Generalized Similarity Learning:** SGNS-style cross-entropy objectives provide better preservation of high-similarity pairs compared to standard $\ell_2$ losses in matrix factorization frameworks, especially for node classification and link prediction [2105.09182].

## 7. Extensions, Limitations, and Broader Connections

SGNS forms the foundation for advanced representation learning models:
- **Bayesian Neural Embeddings:** Variational-Bayes SGNS yields uncertainty-aware embeddings, slightly outperforming point-estimate SGNS on standard benchmarks [1603.06571].
- **Word-Context Classification Framework:** SGNS is a special case of a broader “Word–Context Classification” model family, parameterized by the choice of noise distribution, factorization structure, and objective [2009.04413]. Adaptive negative sampling via GAN-style objectives further enhances robustness.
- **Hyperbolic Embedding:** The removal of the sigmoid in SGNS results in direct low-rank factorizations of squashed shifted PMI matrices, connecting learned probabilities to hyperbolic geometry and complex network structure [2002.12005].

Limitations of SGNS include sensitivity to negative sampling strategies, ambiguity in embedding geometry absent regularization, and the assumption of independence between word and context distributions. Quadratic regularization, incremental training, and dimension-level constraints have each been proposed to address these issues. SGNS’s pervasive influence on language, network, and item representation learning, combined with ongoing theoretical developments, ensure its continued relevance as both a methodological benchmark and a substrate for future model development.

Source: https://www.emergentmind.com/topics/skip-gram-with-negative-sampling-sgns