Count-Based Embeddings

• Count-based embeddings are techniques that generate dense vector representations from raw count statistics, leveraging measures like PMI and PPMI for semantic similarity.
• They encompass methods from classical matrix factorization and exponential family modeling to higher-order tensor and graph analyses, applying to NLP, market basket data, and visual tasks.
• While offering interpretability and competitive performance relative to prediction-based models, these methods require careful design of context windows and statistical transformations.

Searching arXiv for the cited papers and related work on count-based embeddings. Count-based embeddings are representations induced directly from observed counts or co-occurrence statistics rather than solely from a local predictive objective. In the classical distributional setting, they operationalize the hypothesis that words occurring in similar contexts tend to have similar meanings by aggregating word–context statistics and transforming them into dense vectors via PMI/PPMI, SVD, Hellinger geometry, or weighted regression; later work extends the same count-centric logic to conditional Poisson models for sparse transactional data, to higher-order tensors, to event logs, and to structural graph features. A distinct recent usage applies the phrase to latent numerical information carried by token or visual embeddings during counting in large language and vision-LLMs (Almeida et al., 2019, Lebret et al., 2014, Rudolph et al., 2016, Bailey et al., 2017, Kirchmann et al., 11 Sep 2025, Wolf et al., 2023, Hasani et al., 21 Nov 2025).

1. Distributional foundations and classical matrix constructions

The classical theory of count-based embeddings is grounded in the distributional hypothesis of Harris and Firth: entities occurring in similar contexts are similar. In this formulation, one builds a global word–context matrix and derives embeddings from its statistics rather than by directly training a context-prediction model. The survey literature explicitly distinguishes this family from prediction-based models such as CBOW and Skip-gram, while also noting that both families can become closely related after suitable transformations of co-occurrence counts (Almeida et al., 2019).

The canonical object is a sparse matrix $X$ with entries

$$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$

From $X$, one defines empirical probabilities

$$P(w) = \frac{\sum_j X_{w j}}{\sum_{i,j} X_{i j}}, \qquad P(c) = \frac{\sum_i X_{i c}}{\sum_{i,j} X_{i j}}, \qquad P(w,c) = \frac{X_{w c}}{\sum_{i,j} X_{i j}}.$$

The standard association score is pointwise mutual information,

$$\mathrm{PMI}(w,c) = \log \frac{P(w,c)}{P(w)\,P(c)},$$

with the clipped variant

$\mathrm{PPMI}(w,c) = \max(\mathrm{PMI}(w,c), 0),$

and the shifted version

$$\mathrm{SPPMI}(w,c) = \max\big(\mathrm{PMI}(w,c) - \log k,\, 0\big).$$

These transformations suppress the dominance of ubiquitous contexts and expose statistically informative associations.

Several influential constructions are matrix-based. Latent Semantic Analysis applies SVD to a term–document matrix. HAL scans a corpus and collects word–word co-occurrence counts within a fixed window, often distance-weighted by $1/d$, and reported an optimal window of $8$. COALS normalizes co-occurrence to emphasize conditional co-occurrence that is “more likely than random.” LR-MVL uses CCA between left and right contexts, Hellinger PCA applies spectral reduction to square-rooted conditional distributions, and GloVe fits word and context vectors to the log of global co-occurrence counts (Almeida et al., 2019).

For low-rank compression, one typically factorizes a weighted matrix $M$, often a PPMI or SPPMI matrix, via truncated SVD,

$$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$0

and defines embeddings by reweighting the singular values,

$$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$1

with $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$2. The survey literature notes common choices such as $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$3 or $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$4, and emphasizes that the resulting vectors are tied to global corpus statistics rather than to a purely local predictive loss (Almeida et al., 2019).

2. Geometric formulations and weighted factorization objectives

A major rehabilitation of count-based models replaces raw count geometry with conditional distributions and the Hellinger distance. Let $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$5 denote the number of occurrences of context word $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$6 in a window around target word $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$7. The conditional co-occurrence distribution is

$$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$8

and the word–context probability matrix is

$$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$9

The square-root map

$X$0

places each word on the unit sphere because $X$1, and Euclidean distances between these vectors correspond directly to Hellinger distances between the underlying distributions (Lebret et al., 2014).

This geometry yields a count-based analogue of PCA. Writing $X$2, one seeks a truncated factorization

$X$3

or, in the stochastic low-rank approximation used by Lebret and Collobert,

$X$4

The same framework provides an explicit encoding function for unseen words or phrases: compute counts for the new item, normalize to $X$5, apply the square-root map, and project with the learned linear operator. That encoding function is presented as a clear advantage over predictive models, which must train new words (Lebret et al., 2014).

GloVe represents a different count-based route. It learns word vectors $X$6 and context vectors $X$7 with biases $X$8 by minimizing

$X$9

where the weighting function downweights extremely rare and extremely frequent co-occurrences. In the survey formulation, a common choice is

$$P(w) = \frac{\sum_j X_{w j}}{\sum_{i,j} X_{i j}}, \qquad P(c) = \frac{\sum_i X_{i c}}{\sum_{i,j} X_{i j}}, \qquad P(w,c) = \frac{X_{w c}}{\sum_{i,j} X_{i j}}.$$0

Typical settings reported are $$P(w) = \frac{\sum_j X_{w j}}{\sum_{i,j} X_{i j}}, \qquad P(c) = \frac{\sum_i X_{i c}}{\sum_{i,j} X_{i j}}, \qquad P(w,c) = \frac{X_{w c}}{\sum_{i,j} X_{i j}}.$$1 and $$P(w) = \frac{\sum_j X_{w j}}{\sum_{i,j} X_{i j}}, \qquad P(c) = \frac{\sum_i X_{i c}}{\sum_{i,j} X_{i j}}, \qquad P(w,c) = \frac{X_{w c}}{\sum_{i,j} X_{i j}}.$$2 in the range $$P(w) = \frac{\sum_j X_{w j}}{\sum_{i,j} X_{i j}}, \qquad P(c) = \frac{\sum_i X_{i c}}{\sum_{i,j} X_{i j}}, \qquad P(w,c) = \frac{X_{w c}}{\sum_{i,j} X_{i j}}.$$3–$$P(w) = \frac{\sum_j X_{w j}}{\sum_{i,j} X_{i j}}, \qquad P(c) = \frac{\sum_i X_{i c}}{\sum_{i,j} X_{i j}}, \qquad P(w,c) = \frac{X_{w c}}{\sum_{i,j} X_{i j}}.$$4 (Almeida et al., 2019).

Empirically, the Hellinger-based study reports that symmetric windows consistently outperform asymmetric ones; window size $$P(w) = \frac{\sum_j X_{w j}}{\sum_{i,j} X_{i j}}, \qquad P(c) = \frac{\sum_i X_{i c}}{\sum_{i,j} X_{i j}}, \qquad P(w,c) = \frac{X_{w c}}{\sum_{i,j} X_{i j}}.$$5 favors syntactic similarity, whereas window size $$P(w) = \frac{\sum_j X_{w j}}{\sum_{i,j} X_{i j}}, \qquad P(c) = \frac{\sum_i X_{i c}}{\sum_{i,j} X_{i j}}, \qquad P(w,c) = \frac{X_{w c}}{\sum_{i,j} X_{i j}}.$$6 is preferred for semantic analogies. On English Wikipedia from January 2014, with about $$P(w) = \frac{\sum_j X_{w j}}{\sum_{i,j} X_{i j}}, \qquad P(c) = \frac{\sum_i X_{i c}}{\sum_{i,j} X_{i j}}, \qquad P(w,c) = \frac{X_{w c}}{\sum_{i,j} X_{i j}}.$$7B tokens and a target vocabulary of $$P(w) = \frac{\sum_j X_{w j}}{\sum_{i,j} X_{i j}}, \qquad P(c) = \frac{\sum_i X_{i c}}{\sum_{i,j} X_{i j}}, \qquad P(w,c) = \frac{X_{w c}}{\sum_{i,j} X_{i j}}.$$8 words, the scenario using mid-rare contexts with frequency between $$P(w) = \frac{\sum_j X_{w j}}{\sum_{i,j} X_{i j}}, \qquad P(c) = \frac{\sum_i X_{i c}}{\sum_{i,j} X_{i j}}, \qquad P(w,c) = \frac{X_{w c}}{\sum_{i,j} X_{i j}}.$$9 and $$\mathrm{PMI}(w,c) = \log \frac{P(w,c)}{P(w)\,P(c)},$$0 and $$\mathrm{PMI}(w,c) = \log \frac{P(w,c)}{P(w)\,P(c)},$$1 with symmetric window $$\mathrm{PMI}(w,c) = \log \frac{P(w,c)}{P(w)\,P(c)},$$2 yields semantic analogy accuracy $$\mathrm{PMI}(w,c) = \log \frac{P(w,c)}{P(w)\,P(c)},$$3, while stochastic low-rank approximation reports reconstruction error $$\mathrm{PMI}(w,c) = \log \frac{P(w,c)}{P(w)\,P(c)},$$4 versus $$\mathrm{PMI}(w,c) = \log \frac{P(w,c)}{P(w)\,P(c)},$$5 for PCA in a reduction from $$\mathrm{PMI}(w,c) = \log \frac{P(w,c)}{P(w)\,P(c)},$$6 to $$\mathrm{PMI}(w,c) = \log \frac{P(w,c)}{P(w)\,P(c)},$$7 dimensions (Lebret et al., 2014).

3. Conditional count-based embeddings and exponential-family modeling

Count-based embeddings need not be derived from a single global matrix. Exponential Family Embeddings generalize the idea by modeling each observation conditionally on a context in an appropriate exponential family. For observations $$\mathrm{PMI}(w,c) = \log \frac{P(w,c)}{P(w)\,P(c)},$$8 and contexts $$\mathrm{PMI}(w,c) = \log \frac{P(w,c)}{P(w)\,P(c)},$$9, the conditional distribution is

$\mathrm{PPMI}(w,c) = \max(\mathrm{PMI}(w,c), 0),$0

The model uses two sets of vectors, item embeddings $\mathrm{PPMI}(w,c) = \max(\mathrm{PMI}(w,c), 0),$1 and context embeddings $\mathrm{PPMI}(w,c) = \max(\mathrm{PMI}(w,c), 0),$2, defines the context summary

$\mathrm{PPMI}(w,c) = \max(\mathrm{PMI}(w,c), 0),$3

and sets the natural parameter to

$\mathrm{PPMI}(w,c) = \max(\mathrm{PMI}(w,c), 0),$4

Here $\mathrm{PPMI}(w,c) = \max(\mathrm{PMI}(w,c), 0),$5 parameterizes how the target item responds to context, while $\mathrm{PPMI}(w,c) = \max(\mathrm{PMI}(w,c), 0),$6 parameterizes how item $\mathrm{PPMI}(w,c) = \max(\mathrm{PMI}(w,c), 0),$7 contributes to the context signal; the magnitude of $\mathrm{PPMI}(w,c) = \max(\mathrm{PMI}(w,c), 0),$8 weights that contribution, so multi-counts $\mathrm{PPMI}(w,c) = \max(\mathrm{PMI}(w,c), 0),$9 amplify the influence of $$\mathrm{SPPMI}(w,c) = \max\big(\mathrm{PMI}(w,c) - \log k,\, 0\big).$$0 proportionally (Rudolph et al., 2016).

For count data, the central instance is the Poisson embedding. The conditional model is

$$\mathrm{SPPMI}(w,c) = \max\big(\mathrm{PMI}(w,c) - \log k,\, 0\big).$$1

The conditional log-likelihood for observation $$\mathrm{SPPMI}(w,c) = \max\big(\mathrm{PMI}(w,c) - \log k,\, 0\big).$$2 is

$$\mathrm{SPPMI}(w,c) = \max\big(\mathrm{PMI}(w,c) - \log k,\, 0\big).$$3

and with $$\mathrm{SPPMI}(w,c) = \max\big(\mathrm{PMI}(w,c) - \log k,\, 0\big).$$4 regularization the objective becomes

$$\mathrm{SPPMI}(w,c) = \max\big(\mathrm{PMI}(w,c) - \log k,\, 0\big).$$5

The gradients have the residual form

$$\mathrm{SPPMI}(w,c) = \max\big(\mathrm{PMI}(w,c) - \log k,\, 0\big).$$6

which matches the generalized linear model score: residual times the context design vector for $$\mathrm{SPPMI}(w,c) = \max\big(\mathrm{PMI}(w,c) - \log k,\, 0\big).$$7, and residual times item embedding times the context count for $$\mathrm{SPPMI}(w,c) = \max\big(\mathrm{PMI}(w,c) - \log k,\, 0\big).$$8 (Rudolph et al., 2016).

In market basket data, the context for item $$\mathrm{SPPMI}(w,c) = \max\big(\mathrm{PMI}(w,c) - \log k,\, 0\big).$$9 in trip $1/d$0 is all other items in the same basket,

$1/d$1

Larger counts increase the weight of $1/d$2 in $1/d$3, so frequent or multi-unit purchases exert stronger contextual influence. Exponential Family Embeddings are trained with stochastic gradient descent and Adagrad step-size adaptation; to handle sparsity, gradients are split into contributions from non-zero entries and zeros, the non-zero part is computed exactly, and zeros are subsampled. For shopping data, artificially downweighting the zeros by a factor such as $1/d$4 can improve predictive performance, and mini-batching with negative-sampling-style subsampling such as $1/d$5 negative samples per positive is used to scale to large item vocabularies (Rudolph et al., 2016).

The empirical profile is explicitly count-oriented. On market basket data with $1/d$6 trips and $1/d$7 items after filtering rare items, Poisson EFE outperforms hierarchical Poisson factorization and Poisson PCA on held-out data; for $1/d$8, $1/d$9-emb achieves the best log-likelihood, and downweighting zeros further improves performance. The learned geometry is asymmetric: nearest neighbors in embedding space are semantically coherent, while the inner products $8$0 expose complementarity and substitution. Large positive values indicate complements, such as chips with beer or pizza, whereas large negative values indicate substitutes or items rarely co-purchased, such as competing pasta sauces or soups. The same Poisson framework is also applied to movie ratings treated as nonnegative counts (Rudolph et al., 2016).

4. Higher-order, sequential, and structural generalizations

A first generalization of matrix count models is tensor factorization. Instead of pairwise co-occurrence, the tensor approach constructs a $8$1-way PPMI tensor $8$2 with entries

$8$3

where $8$4 is the probability that all $8$5 words occur within a fixed-length window irrespective of order. The symmetric CP model solves

$8$6

and in minibatch form minimizes

$8$7

The joint model JCP-S couples second- and third-order structure with a shared factor $8$8:

$8$9

Because

$M$0

multiplicative composition $M$1 acts as a context-sensitive meaning vector and is used to analyze polysemy. On the outlier-detection tasks reported in the paper, CP-S obtains OD2 OPP/accuracy $M$2 and OD3 OPP/accuracy $M$3, exceeding the corresponding CBOW values $M$4 and $M$5 (Bailey et al., 2017).

A second generalization replaces text corpora with event logs. In this setting, a trace is a finite sequence $M$6, contexts are extracted with a sliding window of length $M$7, and a padding symbol PAD is added at trace boundaries. Two matrix families are defined: an activity–activity matrix $M$8 counting how often activities share contexts, and an activity–context matrix $M$9 with entries $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$00. From any nonnegative count matrix $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$01 one defines

$$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$02

followed by

$$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$03

The embeddings are simply the rows of the resulting matrix, and similarity is computed with cosine distance. No dimensionality reduction is used in the reported experiments. The best intrinsic results are obtained by $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$04/sequence $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$05 PMI $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$06 $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$07, with $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$08, $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$09, $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$10, and $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$11, whereas next-activity prediction is strongest for $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$12/frequency with PMI or PPMI and $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$13, yielding accuracy $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$14 overall and $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$15 on complex logs (Kirchmann et al., 11 Sep 2025).

A third generalization defines count-based embeddings on graphs by counting small structural patterns. For a rooted pattern $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$16 and node $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$17, the node-anchored homomorphism count is

$$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$18

Given a family $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$19 of rooted patterns, the embedding of $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$20 is

$$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$21

The framework supports weighted and labeled variants, uses paths, cycles, trees, and full binary trees as templates, and exploits the fact that counting homomorphisms from bounded-treewidth templates is polynomial-time. Paths connect the method to walk counts, trees characterize $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$22-WL color refinement equivalence, and cycles capture spectral information. In the reported experiments, Tensor Cycles reaches $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$23 on Cora, Paths+Features reaches $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$24 on Citeseer, and Tensor Binary Trees reaches $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$25 on OGBN-Arxiv; the paper emphasizes that each feature has a direct combinatorial meaning and supports end-to-end explainability with classical models such as Random Forests or SVMs (Wolf et al., 2023).

5. Interpretability, trade-offs, and relation to predictive models

A recurring distinction inside the literature is between embeddings that reconstruct global statistics and embeddings that optimize conditional prediction. Classic count-based NLP methods such as PPMI$$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$26SVD and GloVe are built by constructing and factorizing global co-occurrence matrices or their transformations. Exponential Family Embeddings, by contrast, optimize a conditional likelihood at the level of individual observations: each target count is predicted from its current context by a GLM-like intensity, and the inner product $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$27 is not a symmetric similarity score but a learned intensity function (Almeida et al., 2019, Rudolph et al., 2016).

The boundary between count-based and predictive models is therefore porous rather than absolute. A key link reported in the survey literature is that Skip-gram with Negative Sampling implicitly factorizes a shifted PMI matrix, so that the learned dot products approximate $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$28. Exponential Family Embeddings provide a closely related statement in conditional form: with Bernoulli observations, Bernoulli EFE with an identity link aligns with skip-gram/negative sampling when zeros are subsampled and downweighted, and the paper shows that negative sampling corresponds to a biased stochastic gradient of a Bernoulli EFE objective. This weakens any simple opposition between “counting” and “predicting” (Almeida et al., 2019, Rudolph et al., 2016).

Interpretability is a persistent strength, but it appears in different forms across subfamilies. In matrix models, PMI and PPMI retain a direct link to observable corpus statistics. In Hellinger models, the coordinates derive from $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$29, and unseen words or phrases can be encoded immediately from counts through a fixed projection, without retraining (Lebret et al., 2014). In process mining, the rows of $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$30 or $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$31 are treated as direct interpretable representations, and the paper argues for “simplicity in the modeling of distributional similarity of activities” precisely because count-based embeddings avoid a complex training process (Kirchmann et al., 11 Sep 2025). In graph learning, each coordinate is the count of a specific rooted pattern around a node, optionally weighted by a feature channel, making local and global explanations explicit (Wolf et al., 2023).

The trade-offs are equally recurrent. Global count models are sensitive to corpus composition, stopwords, and window design; overly large windows can blur syntactic signals, whereas overly small windows can miss semantic relations (Almeida et al., 2019). Hellinger-based models can lose semantic information after compression, even though raw distributions can be unexpectedly strong on semantic analogies (Lebret et al., 2014). Poisson EFE requires designing the context function and maintaining two parameter sets $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$32, while downweighting zeros introduces bias even when it improves predictive performance; the additive Poisson embedding captures only positive correlations (Rudolph et al., 2016). Activity–context representations can suffer context explosion as $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$33 grows, and graph homomorphism counts grow rapidly with template size and can time out on large graphs, especially for cycles (Kirchmann et al., 11 Sep 2025, Wolf et al., 2023).

Empirical comparisons in the literature do not yield a single winner. The survey reports that GloVe gains over previous count-based models and over SGNS on multiple tasks, but also that Baroni et al. found prediction-based vectors often outperform context-counting on a broad suite of tasks. The overall picture is explicitly described as mixed: modern count-based methods, especially PMI-weighted $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$34 SVD and GloVe, are highly competitive and sometimes superior, with advantages depending on corpus, domain, and task (Almeida et al., 2019).

6. Count-based embeddings as latent numerical states in LLMs and LVLMs

A more recent line of work uses the phrase “count-based embeddings” in a different sense: not embeddings learned from count statistics, but token or visual embeddings that themselves encode latent count information. The paper “Understanding Counting Mechanisms in Large Language and Vision-LLMs” studies controlled textual and visual counting tasks with Qwen2.5, Llama3, Qwen2.5-VL, and InternVL3.5, and introduces CountScope because Logit Lens and Tuned Lens were too noisy for decoding digits, especially in LVLMs (Hasani et al., 21 Nov 2025).

The central finding is stated directly: “individual tokens or visual features encode latent positional count information that can be extracted and transferred across contexts.” CountScope constructs a minimal target context with placeholder tokens or visual regions followed by a counting question, transfers internal activations from a source item, token, or region into that target by minimal activation patching, and reads out softmax probabilities over digits. The method localizes count information by token or region and by layer, using token probability, indirect effect, logit difference, KL divergence, and the paper’s causal influence score

$$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$35

Layerwise analyses show that lower layers encode small counts and higher layers are required to represent larger counts; in both LLM and LVLM settings, layerwise decoding progressively reveals counts $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$36 as the layer cutoff increases from $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$37 to $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$38 (Hasani et al., 21 Nov 2025).

Localization is modality-dependent. In text, numerical information is primarily localized to context tokens rather than the question, and the final token carries maximal causal weight. Separators such as commas carry strong positional and count signals and act as shortcuts; causal patching all separators to the first-separator activation produces a performance drop of $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$39 in monotypic lists and $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$40 in polytypic lists. In vision, count information appears in both foreground and background visual embeddings, and the most informative region is not always the final object. The reported foreground versus background CountScope probabilities are $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$41 versus $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$42 for a $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$43 grid and $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$44 versus $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$45 for a $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$46 grid, with background patches increasingly dominating at higher visual resolutions and sparser layouts (Hasani et al., 21 Nov 2025).

The paper argues for an internal counter mechanism that updates with each item and is mainly stored in the final token or region in text and in the last-plus-adjacent regions in vision. Position-difference vectors are approximately linearly additive in embedding space and can shift an item’s decoded count, with average CI scores $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$47 for the LLM setting with $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$48 and $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$49 for the LVLM setting with $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$50. Cross-context transfer is strong when patching context segments, with CI approximately $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$51 in LLMs and $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$52 in LVLMs, whereas patching question segments has no significant effect, with CI approximately $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$53 and $$X_{ij} = \text{number of times word } i \text{ co-occurs with context } j \text{ within the window}.$$54. This suggests a terminological bifurcation: “count-based embeddings” now refers both to embeddings built from counts and to embeddings that internally represent counts (Hasani et al., 21 Nov 2025).