---
title: Count-Based Embeddings
url: https://www.emergentmind.com/topics/count-based-embeddings
type: topic
---

# Count-Based Embeddings

Searching arXiv for the cited papers and related work on count-based embeddings.
Count-based embeddings are representations induced directly from observed counts or co-occurrence statistics rather than solely from a local predictive objective. In the classical distributional setting, they operationalize the hypothesis that words occurring in similar contexts tend to have similar meanings by aggregating word–context statistics and transforming them into dense vectors via PMI/PPMI, SVD, Hellinger geometry, or weighted regression; later work extends the same count-centric logic to conditional Poisson models for sparse transactional data, to higher-order tensors, to event logs, and to structural graph features. A distinct recent usage applies the phrase to latent numerical information carried by token or visual embeddings during counting in large language and vision-language models [1901.09069] [1412.4930] [1608.00778] [1704.02686] [2509.09440] [2308.15283] [2511.17699].

## 1. Distributional foundations and classical matrix constructions

The classical theory of count-based embeddings is grounded in the distributional hypothesis of Harris and Firth: entities occurring in similar contexts are similar. In this formulation, one builds a global word–context matrix and derives embeddings from its statistics rather than by directly training a context-prediction model. The survey literature explicitly distinguishes this family from prediction-based models such as CBOW and Skip-gram, while also noting that both families can become closely related after suitable transformations of co-occurrence counts [1901.09069].

The canonical object is a sparse matrix $X$ with entries
$$
X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.
$$
From $X$, one defines empirical probabilities
$$
P(w) = \frac{\sum_j X_{w j}}{\sum_{i,j} X_{i j}}, \qquad
P(c) = \frac{\sum_i X_{i c}}{\sum_{i,j} X_{i j}}, \qquad
P(w,c) = \frac{X_{w c}}{\sum_{i,j} X_{i j}}.
$$
The standard association score is pointwise mutual information,
$$
\mathrm{PMI}(w,c) = \log \frac{P(w,c)}{P(w)\,P(c)},
$$
with the clipped variant
$$
\mathrm{PPMI}(w,c) = \max(\mathrm{PMI}(w,c), 0),
$$
and the shifted version
$$
\mathrm{SPPMI}(w,c) = \max\big(\mathrm{PMI}(w,c) - \log k,\, 0\big).
$$
These transformations suppress the dominance of ubiquitous contexts and expose statistically informative associations.

Several influential constructions are matrix-based. Latent Semantic Analysis applies SVD to a term–document matrix. HAL scans a corpus and collects word–word co-occurrence counts within a fixed window, often distance-weighted by $1/d$, and reported an optimal window of $8$. COALS normalizes co-occurrence to emphasize conditional co-occurrence that is “more likely than random.” LR-MVL uses CCA between left and right contexts, Hellinger PCA applies spectral reduction to square-rooted conditional distributions, and GloVe fits word and context vectors to the log of global co-occurrence counts [1901.09069].

For low-rank compression, one typically factorizes a weighted matrix $M$, often a PPMI or SPPMI matrix, via truncated SVD,
$$
M \approx U\,\Sigma\,V^{\top},
$$
and defines embeddings by reweighting the singular values,
$$
W = U\,\Sigma^{\alpha}, \qquad C = V\,\Sigma^{\alpha},
$$
with $\alpha \in [0,1]$. The survey literature notes common choices such as $\alpha = 0.5$ or $\alpha = 1$, and emphasizes that the resulting vectors are tied to global corpus statistics rather than to a purely local predictive loss [1901.09069].

## 2. Geometric formulations and weighted factorization objectives

A major rehabilitation of count-based models replaces raw count geometry with conditional distributions and the Hellinger distance. Let $n(c,w)$ denote the number of occurrences of context word $c$ in a window around target word $w$. The conditional co-occurrence distribution is
$$
p(c \mid w) = \frac{n(c,w)}{\sum_{c' \in \mathcal{D}} n(c',w)},
$$
and the word–context probability matrix is
$$
C_{w,c} = p(c \mid w).
$$
The square-root map
$$
x_w(c) = \sqrt{p(c \mid w)}
$$
places each word on the unit sphere because $\sum_c p(c \mid w)=1$, and Euclidean distances between these vectors correspond directly to Hellinger distances between the underlying distributions [1412.4930].

This geometry yields a count-based analogue of PCA. Writing $X_{w,c}=\sqrt{p(c\mid w)}$, one seeks a truncated factorization
$$
X \approx U_k \Sigma_k V_k^\top,
$$
or, in the stochastic low-rank approximation used by Lebret and Collobert,
$$
\min_{U,V} \left\| V U^\top x_w - x_w \right\|_2^2,
\qquad U,V \in \mathbb{R}^{|\mathcal{D}| \times k}.
$$
The same framework provides an explicit encoding function for unseen words or phrases: compute counts for the new item, normalize to $p(c\mid w^\ast)$, apply the square-root map, and project with the learned linear operator. That encoding function is presented as a clear advantage over predictive models, which must train new words [1412.4930].

GloVe represents a different count-based route. It learns word vectors $w_i$ and context vectors $\tilde w_j$ with biases $b_i,\tilde b_j$ by minimizing
$$
J = \sum_{i,j} f(X_{ij})\Big( w_i^{\top} \tilde{w}_j + b_i + \tilde{b}_j - \log X_{ij} \Big)^2,
$$
where the weighting function downweights extremely rare and extremely frequent co-occurrences. In the survey formulation, a common choice is
$$
f(x) =
\begin{cases}
(x/x_{\max})^{\alpha} & \text{if } x < x_{\max} \\
1 & \text{otherwise.}
\end{cases}
$$
Typical settings reported are $\alpha \approx 0.75$ and $x_{\max}$ in the range $50$–$100$ [1901.09069].

Empirically, the Hellinger-based study reports that symmetric windows consistently outperform asymmetric ones; window size $1$ favors syntactic similarity, whereas window size $10$ is preferred for semantic analogies. On English Wikipedia from January 2014, with about $1.6$B tokens and a target vocabulary of $191{,}268$ words, the scenario using mid-rare contexts with frequency between $10^{-5}$ and $10^{-6}$ and $k=300$ with symmetric window $10$ yields semantic analogy accuracy $62.5$, while stochastic low-rank approximation reports reconstruction error $440.3$ versus $532.2$ for PCA in a reduction from $10{,}000$ to $100$ dimensions [1412.4930].

## 3. Conditional count-based embeddings and exponential-family modeling

Count-based embeddings need not be derived from a single global matrix. Exponential Family Embeddings generalize the idea by modeling each observation conditionally on a context in an appropriate exponential family. For observations $x=(x_1,\ldots,x_p)$ and contexts $C(i)\subset \{1,\ldots,p\}$, the conditional distribution is
$$
p(x_i \mid x_{C(i)}; \eta_i) = h(x_i)\exp\!\big(\eta_i^\top T(x_i)-A(\eta_i)\big).
$$
The model uses two sets of vectors, item embeddings $\rho_i$ and context embeddings $\alpha_j$, defines the context summary
$$
s_i = \sum_{j \in C(i)} \alpha_j x_j,
$$
and sets the natural parameter to
$$
\eta_i = \rho_i^\top s_i.
$$
Here $\rho_i$ parameterizes how the target item responds to context, while $\alpha_j$ parameterizes how item $j$ contributes to the context signal; the magnitude of $x_j$ weights that contribution, so multi-counts $x_j>0$ amplify the influence of $j$ proportionally [1608.00778].

For count data, the central instance is the Poisson embedding. The conditional model is
$$
x_i \mid x_{C(i)} \sim \mathrm{Poisson}(\lambda_i), \qquad
\log \lambda_i = \rho_i^\top \Big(\sum_{j \in C(i)} \alpha_j x_j\Big).
$$
The conditional log-likelihood for observation $i$ is
$$
\ell_i(\rho,\alpha) = x_i \log \lambda_i - \lambda_i - \log(x_i!),
$$
and with $\ell_2$ regularization the objective becomes
$$
L(\rho,\alpha) =
\sum_i \big[x_i \log \lambda_i - \lambda_i - \log(x_i!)\big]
-\frac{\lambda_\rho}{2}\sum_i \|\rho_i\|_2^2
-\frac{\lambda_\alpha}{2}\sum_j \|\alpha_j\|_2^2.
$$
The gradients have the residual form
$$
\frac{\partial \ell_i}{\partial \rho_i}
=
(x_i-\lambda_i)\Big(\sum_{j \in C(i)} \alpha_j x_j\Big),
\qquad
\frac{\partial \ell_i}{\partial \alpha_j}
=
(x_i-\lambda_i)\rho_i x_j,
\quad j\in C(i),
$$
which matches the generalized linear model score: residual times the context design vector for $\rho_i$, and residual times item embedding times the context count for $\alpha_j$ [1608.00778].

In market basket data, the context for item $n$ in trip $t$ is all other items in the same basket,
$$
C(i)=\{(m,t)\mid m\neq n\}.
$$
Larger counts increase the weight of $\alpha_j$ in $s_i$, so frequent or multi-unit purchases exert stronger contextual influence. Exponential Family Embeddings are trained with stochastic gradient descent and Adagrad step-size adaptation; to handle sparsity, gradients are split into contributions from non-zero entries and zeros, the non-zero part is computed exactly, and zeros are subsampled. For shopping data, artificially downweighting the zeros by a factor such as $0.1$ can improve predictive performance, and mini-batching with negative-sampling-style subsampling such as $10$ negative samples per positive is used to scale to large item vocabularies [1608.00778].

The empirical profile is explicitly count-oriented. On market basket data with $137{,}632$ trips and $7{,}903$ items after filtering rare items, Poisson EFE outperforms hierarchical Poisson factorization and Poisson PCA on held-out data; for $K=100$, $p$-emb achieves the best log-likelihood, and downweighting zeros further improves performance. The learned geometry is asymmetric: nearest neighbors in embedding space are semantically coherent, while the inner products $\rho_i^\top \alpha_j$ expose complementarity and substitution. Large positive values indicate complements, such as chips with beer or pizza, whereas large negative values indicate substitutes or items rarely co-purchased, such as competing pasta sauces or soups. The same Poisson framework is also applied to movie ratings treated as nonnegative counts [1608.00778].

## 4. Higher-order, sequential, and structural generalizations

A first generalization of matrix count models is tensor factorization. Instead of pairwise co-occurrence, the tensor approach constructs a $3$-way PPMI tensor $M_3 \in \mathbb{R}^{|V|\times|V|\times|V|}$ with entries
$$
m_{3,ijk} = \mathrm{PPMI}(w_i,w_j,w_k),
$$
where $p(x_1,\ldots,x_N)$ is the probability that all $N$ words occur within a fixed-length window irrespective of order. The symmetric CP model solves
$$
W := \arg\min_U \| M_3 - \llbracket U,U,U\rrbracket \|_F,
$$
and in minibatch form minimizes
$$
\mathcal{L}(M_3^t,U) =
\sum_{m_{3,ijk}\in M_3^t}
\Big(m_{3,ijk}-\sum_{r=1}^R U_{ir}U_{jr}U_{kr}\Big)^2.
$$
The joint model JCP-S couples second- and third-order structure with a shared factor $U$:
$$
W := \arg\min_U \big[\mathcal{L}(M_2,U)+\mathcal{L}(M_3,U)\big].
$$
Because
$$
m_{3,ijk} \approx \sum_r U_{ir}U_{jr}U_{kr}
= \langle v_i * v_j, v_k\rangle,
$$
multiplicative composition $v_i * v_j$ acts as a context-sensitive meaning vector and is used to analyze polysemy. On the outlier-detection tasks reported in the paper, CP-S obtains OD2 OPP/accuracy $0.7078/0.4370$ and OD3 OPP/accuracy $0.6741/0.3597$, exceeding the corresponding CBOW values $0.6542/0.3731$ and $0.6162/0.3034$ [1704.02686].

A second generalization replaces text corpora with event logs. In this setting, a trace is a finite sequence $t=\langle a_1,\ldots,a_n\rangle$, contexts are extracted with a sliding window of length $n$, and a padding symbol PAD is added at trace boundaries. Two matrix families are defined: an activity–activity matrix $AA$ counting how often activities share contexts, and an activity–context matrix $AC$ with entries $AC(a,c)=\#(a,c)$. From any nonnegative count matrix $C$ one defines
$$
Z = \sum_{a,b} c_{ab}, \qquad
p(i) = \frac{\sum_j c_{ij}}{Z}, \qquad
p(j) = \frac{\sum_i c_{ij}}{Z}, \qquad
p(i,j) = \frac{c_{ij}}{Z},
$$
followed by
$$
\mathrm{PMI}(i,j)=\log\frac{p(i,j)}{p(i)p(j)}, \qquad
\mathrm{PPMI}(i,j)=\max(\mathrm{PMI}(i,j),0).
$$
The embeddings are simply the rows of the resulting matrix, and similarity is computed with cosine distance. No dimensionality reduction is used in the reported experiments. The best intrinsic results are obtained by $AA$/sequence $+$ PMI $+$ $n=3$, with $I_{\mathrm{comp}}=0.83$, $I_{\mathrm{nn}}=0.75$, $I_{\mathrm{prec}}=0.72$, and $I_{\mathrm{tri}}=0.83$, whereas next-activity prediction is strongest for $AC$/frequency with PMI or PPMI and $n=9$, yielding accuracy $0.64$ overall and $0.38$ on complex logs [2509.09440].

A third generalization defines count-based embeddings on graphs by counting small structural patterns. For a rooted pattern $(H,r)$ and node $v\in V(G)$, the node-anchored homomorphism count is
$$
\mathrm{hom}_r(H,G;v)
=
\big|\{\phi\in\mathrm{Hom}(H,G)\mid \phi(r)=v\}\big|.
$$
Given a family $K$ of rooted patterns, the embedding of $v$ is
$$
\mathrm{emb}_K^G(v) =
\big(\mathrm{hom}_r(H,G;v)\big)_{(H,r)\in K}.
$$
The framework supports weighted and labeled variants, uses paths, cycles, trees, and full binary trees as templates, and exploits the fact that counting homomorphisms from bounded-treewidth templates is polynomial-time. Paths connect the method to walk counts, trees characterize $1$-WL color refinement equivalence, and cycles capture spectral information. In the reported experiments, Tensor Cycles reaches $0.859\pm0.018$ on Cora, Paths+Features reaches $0.788\pm0.061$ on Citeseer, and Tensor Binary Trees reaches $0.551\pm0.001$ on OGBN-Arxiv; the paper emphasizes that each feature has a direct combinatorial meaning and supports end-to-end explainability with classical models such as Random Forests or SVMs [2308.15283].

## 5. Interpretability, trade-offs, and relation to predictive models

A recurring distinction inside the literature is between embeddings that reconstruct global statistics and embeddings that optimize conditional prediction. Classic count-based NLP methods such as PPMI$+$SVD and GloVe are built by constructing and factorizing global co-occurrence matrices or their transformations. Exponential Family Embeddings, by contrast, optimize a conditional likelihood at the level of individual observations: each target count is predicted from its current context by a GLM-like intensity, and the inner product $\rho_i^\top s_i$ is not a symmetric similarity score but a learned intensity function [1901.09069] [1608.00778].

The boundary between count-based and predictive models is therefore porous rather than absolute. A key link reported in the survey literature is that Skip-gram with Negative Sampling implicitly factorizes a shifted PMI matrix, so that the learned dot products approximate $\mathrm{PMI}(w,c)-\log k$. Exponential Family Embeddings provide a closely related statement in conditional form: with Bernoulli observations, Bernoulli EFE with an identity link aligns with skip-gram/negative sampling when zeros are subsampled and downweighted, and the paper shows that negative sampling corresponds to a biased stochastic gradient of a Bernoulli EFE objective. This weakens any simple opposition between “counting” and “predicting” [1901.09069] [1608.00778].

Interpretability is a persistent strength, but it appears in different forms across subfamilies. In matrix models, PMI and PPMI retain a direct link to observable corpus statistics. In Hellinger models, the coordinates derive from $\sqrt{p(c\mid w)}$, and unseen words or phrases can be encoded immediately from counts through a fixed projection, without retraining [1412.4930]. In process mining, the rows of $AA$ or $AC$ are treated as direct interpretable representations, and the paper argues for “simplicity in the modeling of distributional similarity of activities” precisely because count-based embeddings avoid a complex training process [2509.09440]. In graph learning, each coordinate is the count of a specific rooted pattern around a node, optionally weighted by a feature channel, making local and global explanations explicit [2308.15283].

The trade-offs are equally recurrent. Global count models are sensitive to corpus composition, stopwords, and window design; overly large windows can blur syntactic signals, whereas overly small windows can miss semantic relations [1901.09069]. Hellinger-based models can lose semantic information after compression, even though raw distributions can be unexpectedly strong on semantic analogies [1412.4930]. Poisson EFE requires designing the context function and maintaining two parameter sets $(\rho,\alpha)$, while downweighting zeros introduces bias even when it improves predictive performance; the additive Poisson embedding captures only positive correlations [1608.00778]. Activity–context representations can suffer context explosion as $n$ grows, and graph homomorphism counts grow rapidly with template size and can time out on large graphs, especially for cycles [2509.09440] [2308.15283].

Empirical comparisons in the literature do not yield a single winner. The survey reports that GloVe gains over previous count-based models and over SGNS on multiple tasks, but also that Baroni et al. found prediction-based vectors often outperform context-counting on a broad suite of tasks. The overall picture is explicitly described as mixed: modern count-based methods, especially PMI-weighted $+$ SVD and GloVe, are highly competitive and sometimes superior, with advantages depending on corpus, domain, and task [1901.09069].

## 6. Count-based embeddings as latent numerical states in LLMs and LVLMs

A more recent line of work uses the phrase “count-based embeddings” in a different sense: not embeddings learned from count statistics, but token or visual embeddings that themselves encode latent count information. The paper “Understanding Counting Mechanisms in Large Language and Vision-Language Models” studies controlled textual and visual counting tasks with Qwen2.5, Llama3, Qwen2.5-VL, and InternVL3.5, and introduces CountScope because Logit Lens and Tuned Lens were too noisy for decoding digits, especially in LVLMs [2511.17699].

The central finding is stated directly: “individual tokens or visual features encode latent positional count information that can be extracted and transferred across contexts.” CountScope constructs a minimal target context with placeholder tokens or visual regions followed by a counting question, transfers internal activations from a source item, token, or region into that target by minimal activation patching, and reads out softmax probabilities over digits. The method localizes count information by token or region and by layer, using token probability, indirect effect, logit difference, KL divergence, and the paper’s causal influence score
$$
\mathrm{CI} = \tfrac{1}{2}\Big[
\big(\mathbf{P}(\tilde r\mid C^\ast)-\mathbf{P}(\tilde r\mid C')\big)
+
\big(\mathbf{P}(r'\mid C')-\mathbf{P}(r'\mid C^\ast)\big)
\Big].
$$
Layerwise analyses show that lower layers encode small counts and higher layers are required to represent larger counts; in both LLM and LVLM settings, layerwise decoding progressively reveals counts $1\!\to\!9$ as the layer cutoff increases from $5$ to $16$ [2511.17699].

Localization is modality-dependent. In text, numerical information is primarily localized to context tokens rather than the question, and the final token carries maximal causal weight. Separators such as commas carry strong positional and count signals and act as shortcuts; causal patching all separators to the first-separator activation produces a performance drop of $0.75\pm0.39$ in monotypic lists and $0.97\pm0.05$ in polytypic lists. In vision, count information appears in both foreground and background visual embeddings, and the most informative region is not always the final object. The reported foreground versus background CountScope probabilities are $0.46\pm0.01$ versus $0.58\pm0.02$ for a $6\times6$ grid and $0.42\pm0.01$ versus $0.61\pm0.01$ for a $10\times10$ grid, with background patches increasingly dominating at higher visual resolutions and sparser layouts [2511.17699].

The paper argues for an internal counter mechanism that updates with each item and is mainly stored in the final token or region in text and in the last-plus-adjacent regions in vision. Position-difference vectors are approximately linearly additive in embedding space and can shift an item’s decoded count, with average CI scores $0.85\pm0.11$ for the LLM setting with $k=3$ and $0.62\pm0.07$ for the LVLM setting with $k=2$. Cross-context transfer is strong when patching context segments, with CI approximately $0.61\pm0.02$ in LLMs and $0.57\pm0.12$ in LVLMs, whereas patching question segments has no significant effect, with CI approximately $0.02\pm0.01$ and $0.03\pm0.04$. This suggests a terminological bifurcation: “count-based embeddings” now refers both to embeddings built from counts and to embeddings that internally represent counts [2511.17699].

Source: https://www.emergentmind.com/topics/count-based-embeddings