AdaGram: Adaptive Skip-gram Model

• AdaGram is a nonparametric Bayesian extension of Skip-gram that dynamically infers multiple sense-specific word embeddings.
• It employs a Dirichlet Process stick-breaking prior to adapt semantic resolution and effectively capture word polysemy.
• Empirical results show AdaGram outperforms single-embedding models in predictive likelihood and word-sense induction, though its benefits depend on context quality.

Searching arXiv for the specified AdaGram-related papers to ground the article. {"query":"id:(Bartunov et al., 2015) OR id:(Bakarov et al., 2017) OR id:(Matveeva et al., 28 Aug 2025)","max_results":10} {"query":"(Bartunov et al., 2015) Adaptive Skip-gram AdaGram","max_results":5} {"query":"(Bakarov et al., 2017) Automated Detection of Non-Relevant Posts on the Russian Imageboard 2ch AdaGram","max_results":5} AdaGram most commonly denotes Adaptive Skip-gram, a nonparametric Bayesian extension of Skip-gram that learns multiple sense-specific embeddings per word and automatically infers how many representations are needed for each lexical item at a chosen semantic resolution (Bartunov et al., 2015). The model was introduced to address the fact that standard Skip-gram and related single-embedding architectures compress polysemous words into a single vector, thereby mixing distinct meanings. In subsequent work, AdaGram has also appeared as a practical multi-sense representation in downstream NLP pipelines, including Russian forum-thread relatedness and non-relevant post detection on the imageboard “2ch” (Bakarov et al., 2017). A later, unrelated paper reused the name AdaGram for a full-matrix adaptive optimizer for generalized linear models, creating a naming ambiguity that is external to the original lexical-semantic model (Matveeva et al., 28 Aug 2025).

1. Origin, motivation, and scope

AdaGram was introduced in "Breaking Sticks and Ambiguities with Adaptive Skip-gram" as a response to a central limitation of Skip-gram: a single vector per word conflates heterogeneous usages, so that either the most frequent sense dominates or several senses become blended in a way that degrades interpretability and downstream utility (Bartunov et al., 2015). The core design objective is therefore adaptive multi-sense embedding learning: for each token occurrence, the model infers a latent sense conditioned on context, and for each vocabulary item it learns as many sense-specific target embeddings as the data support.

The distinguishing feature of AdaGram is that the number of senses is not fixed a priori. Instead, it is governed by a Dirichlet Process stick-breaking prior, with a single global concentration parameter $\alpha$ controlling semantic granularity (Bartunov et al., 2015). Larger $\alpha$ yields finer partitions of meaning, while smaller $\alpha$ collapses toward coarser inventories. The paper describes this as a semantic resolution dial, and notes that the expected number of senses for a word scales roughly as $\alpha \log n_w$, where $n_w$ is the number of occurrences of word $w$.

This makes AdaGram conceptually distinct from earlier multi-prototype approaches such as MSSG, NP-MSSG, and fixed-cardinality multi-prototype Skip-gram variants. Those methods either require a known number of senses or rely on greedy heuristics, whereas AdaGram uses a principled DP stick-breaking prior together with variational inference and online learning (Bartunov et al., 2015).

2. Probabilistic structure and sense-specific prediction

In AdaGram, each vocabulary item $w$ is associated with a potentially unbounded sequence of sense-specific input embeddings $u_{w,k} \in \mathbb{R}^D$, while context prediction parameters are shared through a hierarchical softmax tree (Bartunov et al., 2015). Only the target embeddings depend on sense; the context-side representation does not.

The per-word sense distribution is defined by the stick-breaking construction

$$\beta_{w,k} \sim \mathrm{Beta}(1,\alpha), \quad \pi_{w,k} = \beta_{w,k}\,\prod_{i=1}^{k-1} (1-\beta_{w,i}), \quad k=1,2,\dots$$

and each token occurrence $i$ of target word $\alpha$0 draws a latent sense

$\alpha$1

Conditioned on sense assignment, local context words are generated with hierarchical softmax. For a context word $\alpha$2:

$\alpha$3

The full joint model over contexts $\alpha$4, senses $\alpha$5, and stick variables $\alpha$6 is given as

$\alpha$7

This formulation preserves the predictive structure of Skip-gram while replacing the single target vector with a latent mixture over sense-specific target vectors (Bartunov et al., 2015). A plausible implication is that AdaGram is best understood not as an entirely new embedding family, but as a sense-adaptive generalization of Skip-gram in which ambiguity is modeled explicitly rather than absorbed into a single distributional prototype.

3. Variational inference, online updates, and semantic resolution

Exact posterior inference over latent senses and stick-breaking variables is intractable, so AdaGram uses a truncated mean-field variational approximation:

$\alpha$8

where $\alpha$9 is a truncation level used in the variational approximation (Bartunov et al., 2015). The variational objective is the ELBO

$\alpha$0

The local posterior over senses for token $\alpha$1 takes the form

$\alpha$2

and the variational posterior over stick variables is

$\alpha$3

with expected counts $\alpha$4 (Bartunov et al., 2015).

The embedding parameters are updated by maximizing the sense-weighted Skip-gram objective

$\alpha$5

and the paper emphasizes a streaming stochastic variational inference procedure. Learning rates $\alpha$6 and $\alpha$7 decay linearly from $\alpha$8 to $\alpha$9 over one pass; embeddings are initialized uniformly in $\alpha \log n_w$0; and counts are initialized conservatively with one sense per word, $\alpha \log n_w$1 and $\alpha \log n_w$2 (Bartunov et al., 2015).

For deployment after training, posterior sense selection for a word $\alpha \log n_w$3 in context $\alpha \log n_w$4 is computed from

$\alpha \log n_w$5

which combines a prior term from the learned stick-breaking distribution with a context-likelihood term (Bartunov et al., 2015).

The computational trade-off is explicit: AdaGram remains linear in corpus size and practical with online updates and hierarchical softmax, but training is reported as about $\alpha \log n_w$6 times slower than Skip-gram at the same embedding dimensionality because each update scores context words under all truncated senses (Bartunov et al., 2015).

4. Original empirical results and qualitative behavior

The original AdaGram study trained on Wikipedia April 2010, approximately 2M articles and approximately 990M tokens, discarding words with fewer than 20 occurrences (Bartunov et al., 2015). The reported hyperparameters for the reference configuration were embedding dimension $\alpha \log n_w$7, context window size $\alpha \log n_w$8, truncation level $\alpha \log n_w$9, and hierarchical softmax with a Huffman tree.

On predictive likelihood, AdaGram improves over Skip-gram. The paper reports average test log-likelihoods on a large Wikipedia slice as follows: Skip-gram 300D at −7.403, Skip-gram 600D at −7.387, and AdaGram 300D ranging from −7.399 at $n_w$0 to −7.375 at $n_w$1, with $n_w$2 yielding −7.382 (Bartunov et al., 2015). The authors note that AdaGram 300D at $n_w$3 outperforms Skip-gram 600D on this measure.

On word-sense induction, evaluated by Adjusted Rand Index (ARI), AdaGram consistently outperforms the baselines reported in the paper. The reported scores are:

• SemEval-2007: AdaGram 0.069, MSSG 0.048, NP-MSSG up to 0.033, MPSG 0.044
• SemEval-2010: AdaGram 0.097, MSSG 0.085, NP-MSSG up to 0.058, MPSG 0.077
• SemEval-2013: AdaGram 0.061, MSSG 0.033, NP-MSSG up to 0.033, MPSG 0.014
• WWSI (test): AdaGram 0.286, MSSG 0.194, NP-MSSG up to 0.163, MPSG 0.160 (Bartunov et al., 2015)

The qualitative examples in the paper are central to its interpretability claims. At $n_w$4, AdaGram separates “python” into programming, snake, and Monty Python senses; “apple” into fruit and company senses; and “rock” into music, geology, and place-name usages (Bartunov et al., 2015). The paper also shows how increasing $n_w$5 produces progressively finer-grained partitions for frequent words such as “light” and “core.”

These findings support the original claim that a nonparametric multi-sense extension of Skip-gram can improve both context prediction and unsupervised sense induction. At the same time, the paper notes limitations: for large $n_w$6, very frequent words may over-split into near-duplicate senses; rare words may remain effectively single-sense due to data sparsity; and the training cost scales with the sense truncation level (Bartunov et al., 2015).

5. AdaGram in Russian forum-thread relatedness and non-relevant post detection

A later study, "Automated Detection of Non-Relevant Posts on the Russian Imageboard '2ch': Importance of the Choice of Word Representations," used AdaGram as one of seven compared embedding models for approximating non-relevant post detection by semantic relatedness between a post and the opening post of its thread (Bakarov et al., 2017). In that study, AdaGram is treated as an extension of Word2Vec that learns multiple embeddings per word to capture different meanings.

The embeddings were trained on 1,906,120 posts from the Russian imageboard “2ch,” comprising 614,707 unique words. Preprocessing removed HTML tags, hyperlinks, and non-alphabetic symbols, and applied lemmatization with pymorphy2 (Bakarov et al., 2017). The downstream semantic-relatedness dataset, 2SR, was constructed from 45 discussion threads and contains 2,663 pairs of short Russian texts, with posts of up to 216 characters. Each instance is a triple $n_w$7, and the label distribution is roughly balanced at 48% vs. 52% (Bakarov et al., 2017).

The operational use of AdaGram in this paper is narrower than in the original model description. The authors report that AdaGram can predict multiple meanings for a single word, but in their setup they limited the model to 2 prototypes per word and, at inference time, used the most probable predicted meaning (Bakarov et al., 2017). The paper does not present the AdaGram objective, priors, or probability model; it references the original work for algorithmic details.

For short-text relatedness, each post was represented by the arithmetic mean of its word embeddings, ignoring OOV tokens, and pairs of post vectors were composed via three strategies: SUM, CON, and CON+PCA. Classification used K-Nearest Neighbors (KNN) with cosine distance, evaluated by 10-fold cross-validation on 2SR, with F1-score as the reported measure (Bakarov et al., 2017).

The reported AdaGram results were:

Setting	AdaGram	Best reported result
HJ (Spearman’s $n_w$8)	0.11	0.52
RT (AP)	0.57	0.76
AE (AP)	0.66	0.79
SUM (F1)	0.800	0.839
CON (F1)	0.819	0.854
CON+PCA (F1)	0.790	0.842

On the RuSSE similarity benchmarks, AdaGram underperformed the other compared models on HJ, RT, and AE. On relatedness classification, its best result was 0.819 F1 with concatenation, whereas the best overall result in the study was 0.854 F1, achieved by FastText with concatenation (Bakarov et al., 2017).

The paper does not present explicit AdaGram-specific error cases, but it states or implies several conditions that likely constrained performance: the 2ch domain contains web slang, misspellings, typos, obscene vocabulary, and “abnormal” grammar; contexts are short and noisy; the model was constrained to two prototypes per word; inference used hard selection of the most probable sense; and post vectors were built by mean aggregation, which may dilute sense distinctions (Bakarov et al., 2017). This suggests that AdaGram’s theoretical advantage on polysemy did not transfer cleanly to a noisy short-text Russian forum setting under the particular preprocessing and composition choices used in that experiment.

6. Limitations, practical interpretation, and naming ambiguity

Across the two NLP papers, AdaGram occupies a specific methodological niche. It is motivated by polysemy, provides a principled Bayesian mechanism for discovering per-word sense inventories, and shows clear gains on word-sense induction and context prediction in the original study (Bartunov et al., 2015). However, its downstream value depends heavily on whether the rest of the pipeline preserves and exploits those sense distinctions. The 2ch study is a concrete counterexample: AdaGram was not the top-performing representation for either intrinsic similarity or semantic-relatedness-based non-relevant post detection, and robust alternatives such as FastText, Swivel, Wang2Vec, and Word2Vec performed better in that domain (Bakarov et al., 2017).

The practical trade-off is therefore not simply single-sense versus multi-sense. AdaGram adds complexity through latent-sense inference and sense selection, and the benefit can diminish when contexts are short, noisy, or heavily normalized, or when downstream composition collapses the representation back to a simple average (Bakarov et al., 2017). Conversely, the original paper indicates that cleaner corpora, richer contexts, and tasks centered on sense induction or diversification are settings in which the model’s design aligns more directly with the objective (Bartunov et al., 2015).

A separate source of confusion is terminological rather than conceptual. In 2025, the name AdaGram was reused for an optimizer that performs dynamic low-rank approximation of a full-matrix preconditioner for generalized linear models (Matveeva et al., 28 Aug 2025). That work concerns full-matrix adaptive optimization, inverse Cholesky recursion, projector-splitting, and low-rank preconditioning, not lexical ambiguity or multi-sense embeddings. The later optimizer paper explicitly defines AdaGram through update rules such as

$n_w$9

with

$w$0

and is unrelated to the Adaptive Skip-gram model except for the shared name (Matveeva et al., 28 Aug 2025).

In contemporary scholarly usage, therefore, AdaGram most often refers to the Adaptive Skip-gram model unless the surrounding context is clearly optimization-focused. When precision matters, citing Adaptive Skip-gram and the original paper (Bartunov et al., 2015) avoids ambiguity.