Feature Co-occurrence Probabilities

• Feature co-occurrence probabilities are statistical measures that define how frequently specific feature combinations occur in high-dimensional data.
• They underpin diverse methods such as edge-aware filtering, embedding learning, and graphical models through normalization techniques like PMI and mutual information.
• Effective application demands addressing sparsity and computational scaling using strategies like matrix factorization and Bayesian shrinkage.

Feature co-occurrence probabilities quantify how often combinations of features appear together within structured data domains. This concept forms a central analytic primitive across imaging, natural language, high-dimensional probability, and modern machine learning. Formally, feature co-occurrence probability generalizes joint probability to the case where features are indicators or categorical variables over complex domains, and may refer to empirical estimates, conditional statistics, or model-based associations. These probabilities underpin algorithms in boundary-preserving filtering, lexical association, embedding learning, graphical models, and discriminative neural networks.

1. Mathematical Definition and Estimation

The most foundational definition of feature co-occurrence is the empirical probability that a set of features $(f_1,\dots,f_k)$ occur together in a given sampling unit. For binary (indicator) features, given $N$ i.i.d. samples $x^{(n)} \in \{0,1\}^D$, the empirical pairwise co-occurrence probability is

$$\hat{P}(i,j) = \frac{1}{N} \sum_{n=1}^N x^{(n)}_i x^{(n)}_j$$

which generalizes to $k$-way co-occurrence as $$\hat{P}(i_1,\dots,i_k) = \frac{1}{N} \sum_{n=1}^N x^{(n)}_{i_1}\dots x^{(n)}_{i_k}$$ (Shen et al., 2015). For count data, co-occurrence matrices may encode the number of times two features are observed together in a sliding window, document, or spatial context (Bollegala et al., 2017).

Conditional co-occurrence probabilities, as in $P(y_n=1 \mid y_m=1)$ for multi-label classification, are typically estimated by normalizing the joint count $c_{mn}$ by the marginal $c_{mm}$:

$$a_{mn} = \hat{P}(y^n = 1\mid y^m=1) = \frac{c_{mn}}{c_{mm}}$$

forming an $N$0 matrix $N$1 of conditional co-occurrence probabilities across all feature pairs (Rawlekar et al., 2024).

Measure-theoretic treatments give $N$2 for arbitrary collections of events or feature subsets, and regular conditional co-occurrence probabilities $N$3 (Wang et al., 2022).

2. Core Roles in Models and Inference

Feature co-occurrence probabilities serve as statistical building blocks for several families of models:

• Edge/boundary-aware image filters: Co-occurrence matrices $N$4 summarize how often pixel intensities $N$5 and $N$6 co-occur in spatial neighborhoods, enabling edge- and texture-preserving smoothing via normalized or pointwise mutual information (PMI)–style weights in convolutional kernels (Jevnisek et al., 2017).
• Topic and Markov models: In Hidden Markov Models, the observed pairwise emission co-occurrence matrix $N$7 directly constrains the identifiability and recovery of both transition and emission distributions under mild conditions, even when higher-order statistics are unavailable (Huang et al., 2018).
• Embedding learning: Co-occurrence statistics underlie classical matrix factorization, deep energy-based models, and Tweedie GLMs, where the aim is to encode feature (item, word, label) relationships such that dot products or distances in the embedding space reflect the log-probability of observed co-occurrence (Shen et al., 2015, Kim et al., 2024).
• Co-occurrence-based feature selection: In text processing, conditional co-occurrence distributions $N$8 are used to measure term specificity by Shannon entropy, with terms that co-occur very selectively (low entropy) being labeled as specific (Stewart, 2014).
• Statistical association testing: Pairwise co-occurrence probabilities are core to tests of lexical or item association, using measures such as PMI, mutual information, log-likelihood ratios, significant co-occurrence testing, and marginal-invariant statistics (e.g., Yule’s $N$9) (Williams, 2022, Chaudhari et al., 2010, Damani, 2013).

3. Normalization and Statistical Measures

Several standard normalization and information-theoretic transformations exist for raw co-occurrence probabilities:

• PMI: $x^{(n)} \in \{0,1\}^D$0 accentuates unexpectedly high (or low) co-occurrence beyond independence (Williams, 2022, Damani, 2013).
• Mutual information (MI): MI is the expected PMI over all joint outcomes, quantifying dependence between two features.
• Yule’s Y: Yule's $x^{(n)} \in \{0,1\}^D$1, with $x^{(n)} \in \{0,1\}^D$2 the odds ratio, is invariant to the marginals and bounded in $x^{(n)} \in \{0,1\}^D$3 (Williams, 2022).
• Corpus- and document-level significance tests: Measures such as cPMI (Damani, 2013) and CSR (Chaudhari et al., 2010) incorporate statistical significance corrections for observed counts relative to random expectation, countering bias toward rare events.

The choice of normalization reflects the analytic goal: For detecting "suspicious coincidences" among rare features, unnormalized PMI may overweight low-probability events; for marginal-invariant association, Yule's $x^{(n)} \in \{0,1\}^D$4 is preferred; corpus-level significance affords robust discovery of strong associations despite data sparsity.

4. Learning, Modeling, and Embedding with Co-occurrence Probabilities

Energy-based and embedding models typically use observed co-occurrence probabilities (or counts) to learn parameter matrices, feature vectors, or neural networks that can generalize behavior beyond the observed data:

• Deep Embedding Models (DEM): These are energy-based latent variable models that learn to estimate $x^{(n)} \in \{0,1\}^D$5 and conditionals $x^{(n)} \in \{0,1\}^D$6 matching observed co-occurrence patterns, optimized via pseudo-likelihood and backpropagation (Shen et al., 2015).
• Factorization and Tweedie models: In high-dimensional count domains, co-occurrence matrices Y are factorized using Poisson-Gamma (Tweedie) GLMs parameterized by embeddings $x^{(n)} \in \{0,1\}^D$7 so that $x^{(n)} \in \{0,1\}^D$8, with the fitted means $x^{(n)} \in \{0,1\}^D$9 normalized into co-occurrence probabilities (Kim et al., 2024).
• Higher-order (k-way) embeddings: The relationship $$\hat{P}(i,j) = \frac{1}{N} \sum_{n=1}^N x^{(n)}_i x^{(n)}_j$$0 holds exactly under certain neural generative models, enabling learning with joint $$\hat{P}(i,j) = \frac{1}{N} \sum_{n=1}^N x^{(n)}_i x^{(n)}_j$$1-tuple probabilities (Bollegala et al., 2017). Empirically, $$\hat{P}(i,j) = \frac{1}{N} \sum_{n=1}^N x^{(n)}_i x^{(n)}_j$$2 or $$\hat{P}(i,j) = \frac{1}{N} \sum_{n=1}^N x^{(n)}_i x^{(n)}_j$$3 balances signal and data sparsity.

In multi-label classification, empirical co-occurrence probabilities are encoded as adjacency matrices for graph convolutional networks, refining independent predictions by enforcing observed label co-occurrence statistics (Rawlekar et al., 2024).

5. Co-occurrence Matrices in Images and Spatial Data

In imaging, co-occurrence probabilities encode spatial dependencies:

• Gray Level Co-occurrence Matrices (GLCM): For discrete images, GLCMs $$\hat{P}(i,j) = \frac{1}{N} \sum_{n=1}^N x^{(n)}_i x^{(n)}_j$$4 are normalized counts of gray-level pairs separated by displacement $$\hat{P}(i,j) = \frac{1}{N} \sum_{n=1}^N x^{(n)}_i x^{(n)}_j$$5; GLCMs approximate $$\hat{P}(i,j) = \frac{1}{N} \sum_{n=1}^N x^{(n)}_i x^{(n)}_j$$6 (V et al., 2012). They underlie Haralick features and other texture descriptors, as well as new measures such as the Trace (the sum of diagonal joint probabilities).
• Co-occurrence filtering: The CoF uses learned co-occurrence matrices—normalized either to joint probabilities $$\hat{P}(i,j) = \frac{1}{N} \sum_{n=1}^N x^{(n)}_i x^{(n)}_j$$7 or PMI approximations—to provide weights for edge- and boundary-aware smoothing kernels (Jevnisek et al., 2017).
• Deep co-occurrence tensors: In deep networks, cross-channel and local-spatial co-occurrence tensors derived from convolutional feature maps improve global image descriptors for retrieval, with both fixed and learnable filter variants (Forcen et al., 2020).

These mechanisms leverage the difference between frequent (within-region or within-texture) and rare (cross-boundary, cross-class) co-occurrences to selectively smooth, segment, or differentiate image regions.

6. Theoretical Results, Identifiability, and Measure-Theoretic Foundations

Recent theoretical developments clarify fundamental aspects of co-occurrence probabilities:

• Identifiability in latent variable models: In HMMs, pairwise co-occurrence probabilities $$\hat{P}(i,j) = \frac{1}{N} \sum_{n=1}^N x^{(n)}_i x^{(n)}_j$$8 are sufficient for unique recovery of transition and emission matrices under “sufficiently scattered” conditions—a geometric, non-degeneracy property on the emission matrix (Huang et al., 2018).
• Sufficiency of co-occurrence features: For $$\hat{P}(i,j) = \frac{1}{N} \sum_{n=1}^N x^{(n)}_i x^{(n)}_j$$9 classified using co-occurrence probabilities $k$0, the correlation between $k$1 and any function $k$2 is upper-bounded by the “information” already present in $k$3. Using multiple context features strictly enlarges this upper bound, explaining the empirical success of distributed and multi-context representations (Li, 2017).
• Measure-theoretic co-occurrence: In settings with combinatorial feature sets and extreme sparsity, measure-theoretic formalism underpins conditional and marginal co-occurrence probabilities, expectations (E-integrals), and the algebra of conditional inference, enabling rigor and extensibility in high-dimensional contexts (Wang et al., 2022).

7. Practical Considerations, Limitations, and Applications

Feature co-occurrence probability estimation is subject to sparsity, sample-size, and normalization trade-offs:

• Sparsity and significance: For high-cardinality domains, empirical co-occurrence estimates are highly sparse. Shrinkage (e.g., corpus-level significance in cPMI), Bayesian estimation, or matrix factorization mitigates variance and prevents rare-event overweighting (Damani, 2013).
• Computational scalability: Large-scale co-occurrence matrices require chunked storage, batched estimation (e.g., row-wise factorization for Tweedie models), and efficient embedding learning (e.g., negative sampling, regularization) (Shen et al., 2015, Kim et al., 2024).
• Domain-specific integration: Co-occurrence probabilities inform multi-label recognition (through GCN adjacency), keyphrase extraction (via entropy of conditional co-occurrence), topic coherence and perplexity, and fine-grained retrieval (image or multi-modal) (Stewart, 2014, Rawlekar et al., 2024, Forcen et al., 2020).

The selection of co-occurrence probability formulations and metrics is dictated by application goals (association discovery, modeling, retrieval, boundary detection), data sparsity, and theoretical guarantees on identifiability and consistency.

---

In sum, feature co-occurrence probabilities constitute a unifying statistical primitive that enables robust, expressive modeling of structure, association, and interaction in complex, high-dimensional systems, provided careful attention to normalization, statistical significance, and domain constraints (Jevnisek et al., 2017, Huang et al., 2018, Shen et al., 2015, Rawlekar et al., 2024, Williams, 2022, Stewart, 2014, Chaudhari et al., 2010, Damani, 2013, Li, 2017, V et al., 2012, Bollegala et al., 2017, Kim et al., 2024, Wang et al., 2022, Forcen et al., 2020).