---
title: 'AdaGram: Adaptive Skip-gram Model'
url: https://www.emergentmind.com/topics/adagram
type: topic
---

# AdaGram: Adaptive Skip-gram Model

Searching arXiv for the specified AdaGram-related papers to ground the article.
{"query":"id:1502.07257 OR id:1707.04860 OR id:2508.21106","max_results":10}
{"query":"1502.07257 Adaptive Skip-gram AdaGram","max_results":5}
{"query":"1707.04860 Automated Detection of Non-Relevant Posts on the Russian Imageboard 2ch AdaGram","max_results":5}
AdaGram most commonly denotes **Adaptive Skip-gram**, a nonparametric Bayesian extension of Skip-gram that learns **multiple sense-specific embeddings per word** and **automatically** infers how many representations are needed for each lexical item at a chosen semantic resolution [1502.07257]. The model was introduced to address the fact that standard Skip-gram and related single-embedding architectures compress polysemous words into a single vector, thereby mixing distinct meanings. In subsequent work, AdaGram has also appeared as a practical multi-sense representation in downstream NLP pipelines, including Russian forum-thread relatedness and non-relevant post detection on the imageboard “2ch” [1707.04860]. A later, unrelated paper reused the name **AdaGram** for a full-matrix adaptive optimizer for generalized linear models, creating a naming ambiguity that is external to the original lexical-semantic model [2508.21106].

## 1. Origin, motivation, and scope

AdaGram was introduced in "Breaking Sticks and Ambiguities with Adaptive Skip-gram" as a response to a central limitation of Skip-gram: a single vector per word conflates heterogeneous usages, so that either the most frequent sense dominates or several senses become blended in a way that degrades interpretability and downstream utility [1502.07257]. The core design objective is therefore **adaptive multi-sense embedding learning**: for each token occurrence, the model infers a latent sense conditioned on context, and for each vocabulary item it learns as many sense-specific target embeddings as the data support.

The distinguishing feature of AdaGram is that the number of senses is **not fixed a priori**. Instead, it is governed by a **Dirichlet Process stick-breaking prior**, with a single global concentration parameter $\alpha$ controlling semantic granularity [1502.07257]. Larger $\alpha$ yields finer partitions of meaning, while smaller $\alpha$ collapses toward coarser inventories. The paper describes this as a **semantic resolution dial**, and notes that the expected number of senses for a word scales roughly as $\alpha \log n_w$, where $n_w$ is the number of occurrences of word $w$.

This makes AdaGram conceptually distinct from earlier multi-prototype approaches such as MSSG, NP-MSSG, and fixed-cardinality multi-prototype Skip-gram variants. Those methods either require a known number of senses or rely on greedy heuristics, whereas AdaGram uses a **principled DP stick-breaking prior** together with variational inference and online learning [1502.07257].

## 2. Probabilistic structure and sense-specific prediction

In AdaGram, each vocabulary item $w$ is associated with a potentially unbounded sequence of sense-specific input embeddings $u_{w,k} \in \mathbb{R}^D$, while context prediction parameters are shared through a hierarchical softmax tree [1502.07257]. Only the **target embeddings** depend on sense; the context-side representation does not.

The per-word sense distribution is defined by the stick-breaking construction
$$
\beta_{w,k} \sim \mathrm{Beta}(1,\alpha), \quad
\pi_{w,k} = \beta_{w,k}\,\prod_{i=1}^{k-1} (1-\beta_{w,i}), \quad k=1,2,\dots
$$
and each token occurrence $i$ of target word $x_i=w$ draws a latent sense
$$
z_i \sim \mathrm{Categorical}\!\left(\pi^{(w)}\right).
$$
Conditioned on sense assignment, local context words are generated with hierarchical softmax. For a context word $v$:
$$
p(v \mid w,s\!=\!k,\theta)
= \prod_{n \in \mathrm{path}(v)} \sigma\big(\mathrm{ch}(n)\, u_{w,k}^{\top} v_n\big).
$$
The full joint model over contexts $Y$, senses $Z$, and stick variables $\beta$ is given as
$$
p(Y,Z,\beta \mid X,\alpha,\theta)
\;=\;
\prod_{w=1}^{V} \prod_{k=1}^{\infty} p(\beta_{w,k}\mid\alpha)\;
\prod_{i=1}^{N} \Big[
p(z_i \mid x_i,\beta)\;
\prod_{j=1}^{C} p(y_{ij} \mid x_i,z_i,\theta)
\Big].
$$

This formulation preserves the predictive structure of Skip-gram while replacing the single target vector with a latent mixture over sense-specific target vectors [1502.07257]. A plausible implication is that AdaGram is best understood not as an entirely new embedding family, but as a **sense-adaptive generalization** of Skip-gram in which ambiguity is modeled explicitly rather than absorbed into a single distributional prototype.

## 3. Variational inference, online updates, and semantic resolution

Exact posterior inference over latent senses and stick-breaking variables is intractable, so AdaGram uses a **truncated mean-field variational approximation**:
$$
q(Z,\beta)
= \Big(\prod_{i=1}^{N} q(z_i)\Big)\;
  \Big(\prod_{w=1}^{V} \prod_{k=1}^{T} q(\beta_{w,k})\Big),
$$
where $T$ is a truncation level used in the variational approximation [1502.07257]. The variational objective is the ELBO
$$
\mathcal{L}
= \mathbb{E}_q\big[\log p(Y,Z,\beta \mid X,\alpha,\theta)\big]
  - \mathbb{E}_q\big[\log q(Z,\beta)\big].
$$

The local posterior over senses for token $i$ takes the form
$$
\log q(z_i=k)
= \mathbb{E}_{q(\beta)}\!\Big[\log \beta_{x_i,k} + \sum_{r=1}^{k-1} \log(1-\beta_{x_i,r})\Big]
  + \sum_{j=1}^{C} \log p(y_{ij}\mid x_i, z_i=k, \theta)
  + \mathrm{const},
$$
and the variational posterior over stick variables is
$$
q(\beta_{w,k}) = \mathrm{Beta}(a_{w,k}, b_{w,k}),
\quad
a_{w,k} = 1 + n_{w,k},\quad
b_{w,k} = \alpha + \sum_{r=k+1}^{T} n_{w,r},
$$
with expected counts $n_{w,k} = \sum_{i: x_i=w} q(z_i=k)$ [1502.07257].

The embedding parameters are updated by maximizing the sense-weighted Skip-gram objective
$$
\sum_{i=1}^{N}\sum_{j=1}^{C}\sum_{k=1}^{T} q(z_i=k)\, \log p(y_{ij}\mid x_i,z_i=k,\theta),
$$
and the paper emphasizes a **streaming stochastic variational inference** procedure. Learning rates $\rho_t$ and $\lambda_t$ decay linearly from $0.025$ to $0$ over one pass; embeddings are initialized uniformly in $[-0.5/D, 0.5/D]$; and counts are initialized conservatively with one sense per word, $n_{w,1}=n_w$ and $n_{w,k>1}=0$ [1502.07257].

For deployment after training, posterior sense selection for a word $x$ in context $\mathbf{y}$ is computed from
$$
p(z=k \mid x, \mathbf{y}, \theta, \alpha)
\propto
p(\mathbf{y} \mid x, z=k, \theta)\;\times\;
\mathbb{E}_{q(\beta)}[\pi_{x,k}],
$$
which combines a prior term from the learned stick-breaking distribution with a context-likelihood term [1502.07257].

The computational trade-off is explicit: AdaGram remains linear in corpus size and practical with online updates and hierarchical softmax, but training is reported as about $T$ times slower than Skip-gram at the same embedding dimensionality because each update scores context words under all truncated senses [1502.07257].

## 4. Original empirical results and qualitative behavior

The original AdaGram study trained on **Wikipedia April 2010**, approximately **2M articles** and approximately **990M tokens**, discarding words with fewer than **20 occurrences** [1502.07257]. The reported hyperparameters for the reference configuration were embedding dimension $D=300$, context window size $C=10$, truncation level $T=30$, and hierarchical softmax with a Huffman tree.

On **predictive likelihood**, AdaGram improves over Skip-gram. The paper reports average test log-likelihoods on a large Wikipedia slice as follows: Skip-gram 300D at **−7.403**, Skip-gram 600D at **−7.387**, and AdaGram 300D ranging from **−7.399** at $\alpha=0.05$ to **−7.375** at $\alpha=0.25$, with **$\alpha=0.15$ yielding −7.382** [1502.07257]. The authors note that AdaGram 300D at $\alpha=0.15$ outperforms Skip-gram 600D on this measure.

On **word-sense induction**, evaluated by **Adjusted Rand Index (ARI)**, AdaGram consistently outperforms the baselines reported in the paper. The reported scores are:

- **SemEval-2007**: AdaGram **0.069**, MSSG **0.048**, NP-MSSG up to **0.033**, MPSG **0.044**
- **SemEval-2010**: AdaGram **0.097**, MSSG **0.085**, NP-MSSG up to **0.058**, MPSG **0.077**
- **SemEval-2013**: AdaGram **0.061**, MSSG **0.033**, NP-MSSG up to **0.033**, MPSG **0.014**
- **WWSI (test)**: AdaGram **0.286**, MSSG **0.194**, NP-MSSG up to **0.163**, MPSG **0.160** [1502.07257]

The qualitative examples in the paper are central to its interpretability claims. At $\alpha=0.1$, AdaGram separates “python” into programming, snake, and Monty Python senses; “apple” into fruit and company senses; and “rock” into music, geology, and place-name usages [1502.07257]. The paper also shows how increasing $\alpha$ produces progressively finer-grained partitions for frequent words such as “light” and “core.”

These findings support the original claim that a **nonparametric multi-sense extension of Skip-gram** can improve both context prediction and unsupervised sense induction. At the same time, the paper notes limitations: for large $\alpha$, very frequent words may over-split into near-duplicate senses; rare words may remain effectively single-sense due to data sparsity; and the training cost scales with the sense truncation level [1502.07257].

## 5. AdaGram in Russian forum-thread relatedness and non-relevant post detection

A later study, "Automated Detection of Non-Relevant Posts on the Russian Imageboard '2ch': Importance of the Choice of Word Representations," used AdaGram as one of seven compared embedding models for approximating **non-relevant post detection** by **semantic relatedness** between a post and the opening post of its thread [1707.04860]. In that study, AdaGram is treated as an extension of Word2Vec that learns multiple embeddings per word to capture different meanings.

The embeddings were trained on **1,906,120 posts** from the Russian imageboard “2ch,” comprising **614,707 unique words**. Preprocessing removed HTML tags, hyperlinks, and non-alphabetic symbols, and applied lemmatization with **pymorphy2** [1707.04860]. The downstream semantic-relatedness dataset, **2SR**, was constructed from **45 discussion threads** and contains **2,663 pairs** of short Russian texts, with posts of up to **216 characters**. Each instance is a triple $(\text{post}, \text{op\_post}, \text{is\_related})$, and the label distribution is roughly balanced at **48% vs. 52%** [1707.04860].

The operational use of AdaGram in this paper is narrower than in the original model description. The authors report that AdaGram can predict multiple meanings for a single word, but in their setup they **limited the model to 2 prototypes per word** and, at inference time, **used the most probable predicted meaning** [1707.04860]. The paper does **not** present the AdaGram objective, priors, or probability model; it references the original work for algorithmic details.

For short-text relatedness, each post was represented by the **arithmetic mean** of its word embeddings, ignoring OOV tokens, and pairs of post vectors were composed via three strategies: **SUM**, **CON**, and **CON+PCA**. Classification used **K-Nearest Neighbors (KNN)** with **cosine distance**, evaluated by **10-fold cross-validation** on 2SR, with **F1-score** as the reported measure [1707.04860].

The reported AdaGram results were:

| Setting | AdaGram | Best reported result |
|---|---:|---:|
| HJ (Spearman’s $p$) | 0.11 | 0.52 |
| RT (AP) | 0.57 | 0.76 |
| AE (AP) | 0.66 | 0.79 |
| SUM (F1) | 0.800 | 0.839 |
| CON (F1) | 0.819 | 0.854 |
| CON+PCA (F1) | 0.790 | 0.842 |

On the **RuSSE** similarity benchmarks, AdaGram underperformed the other compared models on **HJ**, **RT**, and **AE**. On relatedness classification, its best result was **0.819 F1** with concatenation, whereas the best overall result in the study was **0.854 F1**, achieved by **FastText with concatenation** [1707.04860].

The paper does not present explicit AdaGram-specific error cases, but it states or implies several conditions that likely constrained performance: the 2ch domain contains **web slang, misspellings, typos, obscene vocabulary, and “abnormal” grammar**; contexts are short and noisy; the model was constrained to **two prototypes per word**; inference used **hard selection** of the most probable sense; and post vectors were built by **mean aggregation**, which may dilute sense distinctions [1707.04860]. This suggests that AdaGram’s theoretical advantage on polysemy did not transfer cleanly to a noisy short-text Russian forum setting under the particular preprocessing and composition choices used in that experiment.

## 6. Limitations, practical interpretation, and naming ambiguity

Across the two NLP papers, AdaGram occupies a specific methodological niche. It is motivated by polysemy, provides a principled Bayesian mechanism for discovering per-word sense inventories, and shows clear gains on word-sense induction and context prediction in the original study [1502.07257]. However, its downstream value depends heavily on whether the rest of the pipeline preserves and exploits those sense distinctions. The 2ch study is a concrete counterexample: AdaGram was not the top-performing representation for either intrinsic similarity or semantic-relatedness-based non-relevant post detection, and robust alternatives such as FastText, Swivel, Wang2Vec, and Word2Vec performed better in that domain [1707.04860].

The practical trade-off is therefore not simply **single-sense versus multi-sense**. AdaGram adds complexity through latent-sense inference and sense selection, and the benefit can diminish when contexts are short, noisy, or heavily normalized, or when downstream composition collapses the representation back to a simple average [1707.04860]. Conversely, the original paper indicates that cleaner corpora, richer contexts, and tasks centered on sense induction or diversification are settings in which the model’s design aligns more directly with the objective [1502.07257].

A separate source of confusion is terminological rather than conceptual. In 2025, the name **AdaGram** was reused for an optimizer that performs **dynamic low-rank approximation of a full-matrix preconditioner** for generalized linear models [2508.21106]. That work concerns full-matrix adaptive optimization, inverse Cholesky recursion, projector-splitting, and low-rank preconditioning, not lexical ambiguity or multi-sense embeddings. The later optimizer paper explicitly defines AdaGram through update rules such as
$$
\theta \leftarrow \theta - \eta\,G_t^{-\nicefrac{1}{2}}g_t,
$$
with
$$
G_t = \epsilon I + \sum_{\tau=1}^t g_\tau g_\tau^\top,
$$
and is unrelated to the Adaptive Skip-gram model except for the shared name [2508.21106].

In contemporary scholarly usage, therefore, **AdaGram** most often refers to the **Adaptive Skip-gram** model unless the surrounding context is clearly optimization-focused. When precision matters, citing **Adaptive Skip-gram** and the original paper [1502.07257] avoids ambiguity.

Source: https://www.emergentmind.com/topics/adagram