---
title: Feature Co-occurrence Probabilities
url: https://www.emergentmind.com/topics/feature-co-occurrence-probabilities
type: topic
---

# Feature Co-occurrence Probabilities

Feature co-occurrence probabilities quantify how often combinations of features appear together within structured data domains. This concept forms a central analytic primitive across imaging, natural language, high-dimensional probability, and modern machine learning. Formally, feature co-occurrence probability generalizes joint probability to the case where features are indicators or categorical variables over complex domains, and may refer to empirical estimates, conditional statistics, or model-based associations. These probabilities underpin algorithms in boundary-preserving filtering, lexical association, embedding learning, graphical models, and discriminative neural networks.

## 1. Mathematical Definition and Estimation

The most foundational definition of feature co-occurrence is the empirical probability that a set of features $(f_1,\dots,f_k)$ occur together in a given sampling unit. For binary (indicator) features, given $N$ i.i.d. samples $x^{(n)} \in \{0,1\}^D$, the empirical pairwise co-occurrence probability is
$$
\hat{P}(i,j) = \frac{1}{N} \sum_{n=1}^N x^{(n)}_i x^{(n)}_j
$$
which generalizes to $k$-way co-occurrence as $\hat{P}(i_1,\dots,i_k) = \frac{1}{N} \sum_{n=1}^N x^{(n)}_{i_1}\dots x^{(n)}_{i_k}$ [1504.02824]. For count data, co-occurrence matrices may encode the number of times two features are observed together in a sliding window, document, or spatial context [1709.01199].

Conditional co-occurrence probabilities, as in $P(y_n=1 \mid y_m=1)$ for multi-label classification, are typically estimated by normalizing the joint count $c_{mn}$ by the marginal $c_{mm}$:
$$
a_{mn} = \hat{P}(y^n = 1\mid y^m=1) = \frac{c_{mn}}{c_{mm}}
$$
forming an $N \times N$ matrix $A$ of conditional co-occurrence probabilities across all feature pairs [2404.16193].

Measure-theoretic treatments give $P[\bigcap_{i \in I} A_i]$ for arbitrary collections of events or feature subsets, and regular conditional co-occurrence probabilities $P[X_J \in A_J \mid X_I \in A_I]$ [2210.09913].

## 2. Core Roles in Models and Inference

Feature co-occurrence probabilities serve as statistical building blocks for several families of models:

- **Edge/boundary-aware image filters:** Co-occurrence matrices $C(i,j)$ summarize how often pixel intensities $i$ and $j$ co-occur in spatial neighborhoods, enabling edge- and texture-preserving smoothing via normalized or pointwise mutual information (PMI)–style weights in convolutional kernels [1703.04111].
- **Topic and Markov models:** In Hidden Markov Models, the observed pairwise emission co-occurrence matrix $P_2(n,\ell) = P(Y_t = n, Y_{t+1} = \ell)$ directly constrains the identifiability and recovery of both transition and emission distributions under mild conditions, even when higher-order statistics are unavailable [1802.06894].
- **Embedding learning:** Co-occurrence statistics underlie classical matrix factorization, deep energy-based models, and Tweedie GLMs, where the aim is to encode feature (item, word, label) relationships such that dot products or distances in the embedding space reflect the log-probability of observed co-occurrence [1504.02824, 2501.00623].
- **Co-occurrence-based feature selection:** In text processing, conditional co-occurrence distributions $p_c^t(j)$ are used to measure term specificity by Shannon entropy, with terms that co-occur very selectively (low entropy) being labeled as specific [1403.2004].
- **Statistical association testing:** Pairwise co-occurrence probabilities are core to tests of lexical or item association, using measures such as PMI, mutual information, log-likelihood ratios, significant co-occurrence testing, and marginal-invariant statistics (e.g., Yule’s $Y$) [2203.08089, 1008.5287, 1307.0596].

## 3. Normalization and Statistical Measures

Several standard normalization and information-theoretic transformations exist for raw co-occurrence probabilities:

- **PMI:** $\mathrm{PMI}(i, j) = \log \frac{P(i, j)}{P(i)P(j)}$ accentuates unexpectedly high (or low) co-occurrence beyond independence [2203.08089, 1307.0596].
- **Mutual information (MI):** MI is the expected PMI over all joint outcomes, quantifying dependence between two features.
- **Yule’s Y:** Yule's $Y = (\sqrt{\lambda} - 1)/(\sqrt{\lambda} + 1)$, with $\lambda$ the odds ratio, is invariant to the marginals and bounded in $[-1,1]$ [2203.08089].
- **Corpus- and document-level significance tests:** Measures such as cPMI [1307.0596] and CSR [1008.5287] incorporate statistical significance corrections for observed counts relative to random expectation, countering bias toward rare events.

The choice of normalization reflects the analytic goal: For detecting "suspicious coincidences" among rare features, unnormalized PMI may overweight low-probability events; for marginal-invariant association, Yule's $Y$ is preferred; corpus-level significance affords robust discovery of strong associations despite data sparsity.

## 4. Learning, Modeling, and Embedding with Co-occurrence Probabilities

Energy-based and embedding models typically use observed co-occurrence probabilities (or counts) to learn parameter matrices, feature vectors, or neural networks that can generalize behavior beyond the observed data:

- **Deep Embedding Models (DEM):** These are energy-based latent variable models that learn to estimate $p_\theta(x)$ and conditionals $p_\theta(x_i=1 \mid x_{-i})$ matching observed co-occurrence patterns, optimized via pseudo-likelihood and backpropagation [1504.02824].
- **Factorization and Tweedie models:** In high-dimensional count domains, co-occurrence matrices Y are factorized using Poisson-Gamma (Tweedie) GLMs parameterized by embeddings $w_i, \tilde w_j$ so that $\log \mu_{ij} = w_i^\top \tilde w_j + b_i + \tilde b_j$, with the fitted means $\mu_{ij}$ normalized into co-occurrence probabilities [2501.00623].
- **Higher-order (k-way) embeddings:** The relationship $\log p(w_1,\dots, w_k) = \sum_i \|w_i\|^2/2d - k\log Z$ holds exactly under certain neural generative models, enabling learning with joint $k$-tuple probabilities [1709.01199]. Empirically, $k=2$ or $3$ balances signal and data sparsity.

In multi-label classification, empirical co-occurrence probabilities are encoded as adjacency matrices for graph convolutional networks, refining independent predictions by enforcing observed label co-occurrence statistics [2404.16193].

## 5. Co-occurrence Matrices in Images and Spatial Data

In imaging, co-occurrence probabilities encode spatial dependencies:

- **Gray Level Co-occurrence Matrices (GLCM):** For discrete images, GLCMs $P_d(i,j)$ are normalized counts of gray-level pairs separated by displacement $d$; GLCMs approximate $P[I(x)=i,\, I(x+d)=j]$ [1205.4831]. They underlie Haralick features and other texture descriptors, as well as new measures such as the Trace (the sum of diagonal joint probabilities).
- **Co-occurrence filtering:** The CoF uses learned co-occurrence matrices—normalized either to joint probabilities $P(i,j)$ or PMI approximations—to provide weights for edge- and boundary-aware smoothing kernels [1703.04111].
- **Deep co-occurrence tensors:** In deep networks, cross-channel and local-spatial co-occurrence tensors derived from convolutional feature maps improve global image descriptors for retrieval, with both fixed and learnable filter variants [2003.13827].

These mechanisms leverage the difference between frequent (within-region or within-texture) and rare (cross-boundary, cross-class) co-occurrences to selectively smooth, segment, or differentiate image regions.

## 6. Theoretical Results, Identifiability, and Measure-Theoretic Foundations

Recent theoretical developments clarify fundamental aspects of co-occurrence probabilities:

- **Identifiability in latent variable models:** In HMMs, pairwise co-occurrence probabilities $P_2(n,\ell)$ are sufficient for unique recovery of transition and emission matrices under “sufficiently scattered” conditions—a geometric, non-degeneracy property on the emission matrix [1802.06894].
- **Sufficiency of co-occurrence features:** For $X$ classified using co-occurrence probabilities $P(C|X)$, the correlation between $P(Y=1|X)$ and any function $f(P(C|X))$ is upper-bounded by the “information” already present in $P(C|X)$. Using multiple context features strictly enlarges this upper bound, explaining the empirical success of distributed and multi-context representations [1707.04218].
- **Measure-theoretic co-occurrence:** In settings with combinatorial feature sets and extreme sparsity, measure-theoretic formalism underpins conditional and marginal co-occurrence probabilities, expectations (E-integrals), and the algebra of conditional inference, enabling rigor and extensibility in high-dimensional contexts [2210.09913].

## 7. Practical Considerations, Limitations, and Applications

Feature co-occurrence probability estimation is subject to sparsity, sample-size, and normalization trade-offs:

- **Sparsity and significance:** For high-cardinality domains, empirical co-occurrence estimates are highly sparse. Shrinkage (e.g., corpus-level significance in cPMI), Bayesian estimation, or matrix factorization mitigates variance and prevents rare-event overweighting [1307.0596].
- **Computational scalability:** Large-scale co-occurrence matrices require chunked storage, batched estimation (e.g., row-wise factorization for Tweedie models), and efficient embedding learning (e.g., negative sampling, regularization) [1504.02824, 2501.00623].
- **Domain-specific integration:** Co-occurrence probabilities inform multi-label recognition (through GCN adjacency), keyphrase extraction (via entropy of conditional co-occurrence), topic coherence and perplexity, and fine-grained retrieval (image or multi-modal) [1403.2004, 2404.16193, 2003.13827].

The selection of co-occurrence probability formulations and metrics is dictated by application goals (association discovery, modeling, retrieval, boundary detection), data sparsity, and theoretical guarantees on identifiability and consistency.

---

In sum, feature co-occurrence probabilities constitute a unifying statistical primitive that enables robust, expressive modeling of structure, association, and interaction in complex, high-dimensional systems, provided careful attention to normalization, statistical significance, and domain constraints [1703.04111, 1802.06894, 1504.02824, 2404.16193, 2203.08089, 1403.2004, 1008.5287, 1307.0596, 1707.04218, 1205.4831, 1709.01199, 2501.00623, 2210.09913, 2003.13827].

Source: https://www.emergentmind.com/topics/feature-co-occurrence-probabilities