Hierarchical Co-occurrence Model

• Hierarchical Co-occurrence Model is a framework that decomposes feature interactions into distinct hierarchical orders using ANOVA-like techniques for clear variance partitioning.
• It employs orthogonality constraints and regularization methods, such as sparse autoencoder penalties, to minimize feature overlap and improve model interpretability.
• The model is applied in feature selection, concept erasure, and neural representation analysis, offering actionable insights for modularity and hierarchical sensitivity assessments.

A Hierarchical Co-occurrence Model refers, in contemporary machine learning literature, to any conceptual or formal structure that organizes and quantifies the relationships, interactions, or modular contributions among multiple features, units, or "concepts," with explicit attention to hierarchical (multi-scale) and orthogonality (interference, disentanglement) constraints. Such models manifest in domains including feature selection, neural representation analysis, and model interpretability, and are critically relevant for tasks requiring the separation of effects across nested or overlapping concept sets, as well as the disambiguation of interactive signals.

1. Functional, Statistical, and Conceptual Decompositions

Hierarchical co-occurrence analysis is closely linked to orthogonal functional and variance decompositions. In supervised learning, a function $f : X \rightarrow Y$ with $X = (X_1, ..., X_d)$ admits an ANOVA-like decomposition:

$$f(X) = f_0 + \sum_i f_i(X_i) + \sum_{i < j} f_{ij}(X_i, X_j) + \ldots + f_{1...d}(X_1, ..., X_d),$$

where each term $f_u(X_u)$ captures the isolated and interactive contribution of feature subset $u$ ($u \subset \{1,...,d\}$), defined recursively via conditional expectations. Orthogonality of this decomposition, i.e.,

$$\forall u\ne v,\quad \int f_u(X_u) f_v(X_v) dX = 0,$$

guarantees an exact partition of variance:

$$\text{Var}(Y) = \sum_u V_u, \quad V_u = \text{Var}[f_u(X_u)].$$

This framework is intrinsically hierarchical in that it separates contributions by order—main effects, pairwise, higher-order—clarifying how feature co-occurrence (joint presence) influences the output and enabling hierarchical sensitivity or attribution analysis (Kamalov, 2019).

2. Hierarchical Orthogonality in Modular and Concept Erasure Models

The DyME framework for multi-concept erasure in diffusion models exemplifies hierarchical co-occurrence modelling in complex neural architectures. Erasure tasks are structured along a hierarchy—e.g., brand $\supset$ series $\supset$ character—requiring the modular suppression of various, potentially overlapping, concepts. DyME enforces bi-level orthogonality:

• Feature-level: For concept-specific LoRA adapters, disjointness is measured in the prompt-conditioned activation shifts (cross-attention outputs) via Frobenius-normalized cosine similarity between adapter-induced feature shift matrices, $X = (X_1, ..., X_d)$0. High $X = (X_1, ..., X_d)$1 signals minimal co-occurrence/interference.
• Parameter-level: The adapters' update matrices are required to be algebraically orthogonal via the symmetrized Gram condition,

$X = (X_1, ..., X_d)$2

where $X = (X_1, ..., X_d)$3 is a bilinear function of adapter LoRA parameters. This ensures the subspaces underlying different concept modules are hierarchically non-overlapping, robustly disentangling their influence.

Empirically, enforcing such orthogonality constraints increases modularity and minimises crosstalk, particularly as more or higher-level concepts are composed at inference; this structure is validated on ErasureBench-H, a benchmark reflecting hierarchical concept scopes (Liu et al., 25 Sep 2025).

3. Orthogonality Regularization and Intervenable Hierarchical Features

Sparse autoencoder (SAE) approaches demonstrate hierarchical co-occurrence modelling at the level of dictionary learning and neural representation analysis. Here, the columns of the decoder matrix, $X = (X_1, ..., X_d)$4, are interpreted as learned "concept-level features." To prevent superposition (non-orthogonal, overlapping features) and promote intervention capability, an orthogonality penalty is imposed:

$X = (X_1, ..., X_d)$5

where the off-diagonal Gram-structure quantifies feature co-occurrence and $X = (X_1, ..., X_d)$6 tunes the strictness. Stronger orthogonality yields:

• Lower self-coherence within the feature dictionary;
• Increased semantic diversity among natural language explanations for the features;
• Isolated, non-overlapping responses when individually intervened upon.

By analogy to the Independent Causal Mechanisms (ICM) principle, this quantifiably modularizes the causes (features), supporting hierarchical manipulation and interpretation of compositional neural phenomena (Miller et al., 4 Feb 2026).

4. Diagnostic Metrics for Hierarchical Co-occurrence and Orthogonality

Across these areas, a repertoire of diagnostic metrics is essential for quantifying hierarchical co-occurrence and disentanglement:

• Feature-cosine matrix $X = (X_1, ..., X_d)$7: Empirically measures pairwise (dis)similarity in concept- or feature-induced activation shifts;
• Parameter-Gram matrix $X = (X_1, ..., X_d)$8: Algebraic similarity for adapter parameter updates;
• Principal subspace angle $X = (X_1, ..., X_d)$9: SVD-based subspace angle between low-rank feature representations;
• Self-coherence $$f(X) = f_0 + \sum_i f_i(X_i) + \sum_{i < j} f_{ij}(X_i, X_j) + \ldots + f_{1...d}(X_1, ..., X_d),$$0: Maximal off-diagonal inner product in a feature dictionary;
• Semantic explanation similarity: Cosine distance between embedded LLM-generated feature summaries.

Hierarchical benchmarks such as ErasureBench-H enable systematic evaluation of these metrics at multiple granularities (character/series/brand), directly linking them to erasure effectiveness and utility preservation. Table 1 summarizes select diagnostic metrics and their domains:

Metric	Level	Primary Domain
Feature-cosine matrix	Feature	Adapter/intervention
Parameter-Gram matrix	Parameter	Adapter/intervention
Self-coherence	Decoder	SAE/dictionary
Semantic explanation sim	Latent/concept	SAE/dictionary

(Kamalov, 2019, Liu et al., 25 Sep 2025, Miller et al., 4 Feb 2026)

5. Hierarchical Co-occurrence in RKHS: Orthogonal Representation Analysis

Finite-dimensional orthogonal polynomial kernels enable exact expansion of SVM decision functions within a tensor-product orthonormal basis, yielding a direct, hierarchical quantification of co-occurrence effects via Orthogonal Representation Contribution Analysis (ORCA). The normalized Orthogonal Kernel Contribution (OKC) indices partition the squared RKHS norm according to interaction order and polynomial degree:

$$f(X) = f_0 + \sum_i f_i(X_i) + \sum_{i < j} f_{ij}(X_i, X_j) + \ldots + f_{1...d}(X_1, ..., X_d),$$1

and the normalized OKC by block $$f(X) = f_0 + \sum_i f_i(X_i) + \sum_{i < j} f_{ij}(X_i, X_j) + \ldots + f_{1...d}(X_1, ..., X_d),$$2 is

$$f(X) = f_0 + \sum_i f_i(X_i) + \sum_{i < j} f_{ij}(X_i, X_j) + \ldots + f_{1...d}(X_1, ..., X_d),$$3

Interaction-order profiles $$f(X) = f_0 + \sum_i f_i(X_i) + \sum_{i < j} f_{ij}(X_i, X_j) + \ldots + f_{1...d}(X_1, ..., X_d),$$4 and degree profiles $$f(X) = f_0 + \sum_i f_i(X_i) + \sum_{i < j} f_{ij}(X_i, X_j) + \ldots + f_{1...d}(X_1, ..., X_d),$$5 diagnose whether the decision boundary is dominated by marginal, pairwise, or higher-order co-occurrence structure, as well as by low- or high-degree complexity. This mechanistically unpacks hierarchical influence of features on learned function complexity, revealing both explicit and implicit co-occurrences (Soto-Larrosa et al., 16 Apr 2026).

6. Applications, Limitations, and Extensions

Hierarchical co-occurrence models serve interpretability, intervention, and modularity goals across domains:

• Feature selection: Orthogonal ANOVA frameworks optimize for minimal feature overlap.
• Concept erasure: Hierarchical, modular adapters support selective, multi-granularity suppression.
• Dictionary learning: Orthogonality regularization yields more interpretable, intervenable concept features.

Limitations include the exponential complexity of complete higher-order decompositions, explicit independence assumptions (in classical ANOVA), and kernel-specificity in orthogonal basis methods. Extensions encompass surrogate orthogonal expansions for general kernels, replacement of independence assumptions via Shapley- or copula-based variance decompositions, and semantic/causal extension of hierarchical concepts to learned representations and interventions (Kamalov, 2019, Liu et al., 25 Sep 2025, Miller et al., 4 Feb 2026, Soto-Larrosa et al., 16 Apr 2026).