Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrence

Abstract: We propose a distributional theory of how hypernymy -- the ``is-a'' relation between general and specific concepts -- is encoded geometrically in language representations. Starting from the empirically verified assumption that words closer on the WordNet hypernym graph co-occur more often, we characterize theoretically the spectrum of the resulting embedding Gram matrix of word2vec embeddings. Under mild positivity and decay conditions on the co-occurrence kernel, we prove that the leading eigenvectors first separate broad taxonomic branches and then progressively finer sub-branches, producing a \emph{hierarchical splitting geometry} with a coarse-to-fine spectral organization that mirrors the tree. We confirm these predictions in word2vec embeddings across many sampled WordNet subtrees, and show that the same signature extends strikingly well to Gemma 2B unembeddings. Our results indicate that hierarchical concept geometry in LLMs need not reflect a hierarchy-specific functional mechanism, but emerges from the spectral structure of pairwise word statistics.

• The paper demonstrates that hierarchical geometry in language models emerges from spectral properties of word co-occurrence statistics.
• It employs spectral decomposition on theoretical and empirical Gram matrices to show that principal components encode coarse-to-fine taxonomic splits.
• The study validates the model in both word2vec and LLM embeddings, yielding quantitative and qualitative predictions of hierarchical structure in representations.

Hierarchical Geometry in LLM Representations: A Distributional Account

Introduction and Motivation

The paper "Hierarchical Concept Geometry in LLMs Emerges from Word Co-occurrence" (2605.23821) explores the geometric encoding of hypernymy—the "is-a" relationship between general and specific concepts—in LLM embeddings. It formalizes how hierarchical concept geometry, observed both in classical embeddings and modern LLMs, arises not from explicit inductive bias or functional mechanisms, but from spectral properties of word co-occurrence statistics in text corpora. By grounding the analysis in WordNet's taxonomic structure and leveraging spectral decomposition theory, the authors provide a mechanistic account for how successive principal components encode taxonomy branches, offering both qualitative and quantitative predictions for the structure of concept representations.

Theoretical Framework: Co-occurrence Induces Hierarchical Structure

The foundation of this study is the empirical assumption that words located more closely in the WordNet hypernym graph tend to co-occur more frequently in corpora. Leveraging this, the authors posit a hierarchical co-occurrence model, formalized as:

$$P_{ij} = P_i P_j [ \widetilde{C}(\mathrm{dist}(i,j)) + \epsilon_{ij} ],$$

where $P_{ij}$ is the co-occurrence probability, $P_i$ the unigram probability, and $\widetilde{C}$ a monotone function of graph distance. This induces a normalized co-occurrence matrix $M^\star_{ij} = f(\mathrm{dist}(i,j))$, which, under spectral decomposition, yields word2vec-style embeddings constructed from its leading eigenvectors.

Importantly, under mild assumptions—positivity and monotonic decay of $f$ with semantic distance—the spectral decomposition organizes embedding geometry so that principal components (PCs) encode coarse-to-fine taxonomic splits. The leading PCs separate major branches (e.g., plant vs animal), while subsequent PCs resolve finer distinctions (e.g., bird vs fish, flower vs tree).

Figure 1: Visualization of the WordNet taxonomy, co-occurrence decay with hierarchy distance, and projection of taxonomy nodes onto hierarchical principal components.

The paper provides detailed theoretical analysis demonstrating that in idealized settings (e.g., symmetric binary trees), the Gram matrix decomposes into scaling modes (constant on each depth) and wavelet modes (contrast between subtrees), and that eigenvectors align with these hierarchical splits. Degenerate eigenspaces are attributable to symmetry across subtrees of identical height.

Empirical Validation in Classical and LLM Embeddings

word2vec

Quantitative evidence is presented through sampling numerous binary subtrees from contracted WordNet hierarchies and comparing theoretical Gram matrices (parameterized by fitted exponential decay kernels) to empirical Gram matrices derived from word2vec embeddings. Eigenspace alignment is measured using the top-$k$ alignment metric:

$g(k) = \frac{1}{k} \|U_k^\top V_k\|_F^2,$

where $U_k$ and $V_k$ contain the top $P_{ij}$0 eigenvectors from empirical and theoretical Gram matrices, respectively. Empirical subspaces consistently align above shuffled-label baselines, demonstrating that the hierarchical splitting geometry is not merely an artifact of random structure.

Figure 2: Empirical mean co-occurrence statistic decays with semantic distance, confirming the monotonicity assumption.

Figure 3: Direct comparison of Gram matrix structure and leading eigenvectors in theory, word2vec, and Gemma; showcasing hierarchical splitting geometry.

Figure 4: Top-$P_{ij}$1 eigenspace alignment between theoretical and empirical Gram matrices for sampled WordNet binary trees; alignment is statistically significant relative to baselines.

LLMs (Gemma 2B)

The study extends the analysis to LLMs—specifically, Gemma 2B unembeddings. After centering and whitening vectors to mitigate global anisotropy, hierarchical splitting geometry is observed in the LLM representations. The eigenspace alignment metric shows robust correspondence to the theoretical spectral predictions, and analysis reveals the same coarse-to-fine spectral organization. Robustness checks demonstrate that the effect is invariant to kernel parameterization, subtree depth, and representation controls (centered embeddings, internal residual-stream activations), and it persists after conditioning on lowest-common-ancestor depth.

Figure 5: Alignment across centered/unwhitened Gemma and Llama internal activations illustrates that the hierarchical signal is not explained by global modes.

Concept-Level Functionality: Orthogonal Innovations and Concept Vectors

Building on prior work (Park et al.), the paper analyzes whether the co-occurrence-driven spectral structure reproduces the observed concept-level orthogonality diagnostics for parent-child innovations in hierarchical concept vectors. For concept vector $P_{ij}$2 associated with concept $P_{ij}$3, the functional postulate is that child vectors $P_{ij}$4 decompose into parent $P_{ij}$5 plus an innovation orthogonal to the parent. The diagnostic $P_{ij}$6 concentrates near zero for parent-child pairs in both Gemma and synthetic co-occurrence embeddings, while shuffled-parent baselines are significantly displaced.

Figure 6: Parent-child concept-vector innovations are nearly orthogonal in both Gemma and co-occurrence embeddings, matching functional predictions.

Robustness and Extensions

A suite of robustness checks bolsters the findings:

• The hierarchical distance decay structure persists in the positive spectral component of $P_{ij}$7 (i.e., what can be embedded in Euclidean space).
• Conditioning on lowest-common-ancestor depth does not nullify the monotonic distance decay.
• Alignment remains statistically significant across alternative kernel families, tree depths, and representation choices.
• The hierarchical eigenspace alignment is not explained by global mean modes or by unordered vector content within sampled trees.

Figure 7: Distance decay in low-rank PSD truncations of $P_{ij}$8, confirming hierarchical structure in Gram-realizable components.

Figure 8: Distance decay persists after conditioning on lowest-common-ancestor depth, reinforcing the validity of the distance model.

Implications and Future Directions

The strong empirical and theoretical results support the claim that hierarchical geometry in LLM representations arises from generic properties of word co-occurrence statistics. This account clarifies prior observations of concept vector orthogonality and hierarchy-aligned PCs: such features do not imply explicit hierarchy-specific functional mechanisms (e.g., inductive bias or targeted training), but follow from the spectral organization induced by pairwise statistics.

Practically, this insight informs interpretability and representation engineering: hierarchical information is robustly encoded in LLMs without explicit architectural bias. Theoretically, it motivates further investigation into how LLMs reconcile context-driven ambiguity and whether the spectral framework can be extended to contextual representations or to learning dynamics in transformer architectures. Future research will likely expand on the "attribute-driven geometry" paradigm, mapping discrete, continuous, and hierarchical linguistic variables to geometric structure in model representations.

Conclusion

This paper provides a mechanistic, distributional theory for hierarchical semantic geometry in LLMs. Co-occurrence statistics, decaying with semantic graph distance, induce coarse-to-fine spectral organization in word embeddings and LLM representations, encoding taxonomic splits in principal components and concept vectors. Hierarchical concept geometry thus emerges spontaneously from the statistics of language, offering quantitative, model-agnostic predictions for representation structure and challenging interpretations based purely on functional orthogonality. This framework enhances understanding of representation learning and will influence future research in interpretability and structure exploitation in AI.