Categorical & Hierarchical Geometry

• Geometry of categorical and hierarchical concepts is a framework that embeds taxonomies into geometric, algebraic, and topological spaces to enable interpretable models across applications such as language models, computer vision, and knowledge graphs.
• Methodologies employ causal embeddings in Minkowski space, convex polytope representations in high-dimensional language models, and spectral decompositions to accurately model and retrieve hierarchical structures with high precision.
• Advanced algorithms integrating iterative local adjustments, manifold learning, and category-theoretic approaches demonstrate the field’s potential to enhance classification, retrieval, and interpretability in complex concept systems.

Geometry of categorical and hierarchical concepts concerns the mathematical and representational structures that encode categories, taxonomies, and their interrelations as objects in geometric, algebraic, or topological spaces. The modern field synergizes perspectives from representation learning, category theory, algebraic geometry, spectral graph theory, and convex analysis, unified by the pursuit of interpretable, computationally tractable models of conceptual structure. Applications span LLMs, computer vision, knowledge graphs, and the theoretical foundations of cognition.

1. Geometric Representations of Hierarchies: Classical and Causal

A key breakthrough is the realization that hierarchies and taxonomies can be embedded in geometric spaces where categorical and parent–child (“is-a”) links correspond to geometric or causal relations. In Minkowski space, every hierarchical structure can be perfectly represented as a partial order of causally related events. Given tokens $p$ and $c$ assigned coordinates $x_p, x_c \in \mathbb{R}^{1+2}$ with $x_p = (t_p, x_p^1, x_p^2)$ and $x_c = (t_c, x_c^1, x_c^2)$, one sets

$$\Delta s^2 = - (t_p - t_c)^2 + \|\mathbf{x}_p - \mathbf{x}_c\|^2$$

and imposes the constraint $t_p - t_c > \|\mathbf{x}_p - \mathbf{x}_c\|$ for every $(p, c)$ in the “is-a” relation, ensuring a causal ordering. Iterative algorithms correct local violations by adjusting only time coordinates, driving all known hierarchy pairs to lie within the causal cone. Retrieval of ancestry is then recast as a light-cone membership test: $q$ is an ancestor of $v$ iff $c$0 and $c$1, with minimal $c$2 proper time selecting the closest ancestor. This approach yields perfect recovery of WordNet hierarchies, including ambiguous nodes, using only local signals and no global symbolic structure, and outperforms earlier Poincaré-ball embeddings in accuracy and mean average precision (Anabalon et al., 7 May 2025).

2. Linear and Polytope Models in LLM Geometry

In the high-dimensional embedding spaces of LLMs, categorical and hierarchical concepts are realized as convex polytopes and direct-sum decompositions. Binary features $c$3 are assigned vectors $c$4 such that the inner product $c$5 reflects membership for token $c$6. Categorical (multi-way) concepts are represented by the convex hull $c$7 of their feature vectors. Empirically, clusters of feature vectors for hyponyms and hypernyms in WordNet create simplices whose barycentric coordinates encode probability assignments across fine-grained categories.

Hierarchical relations translate to orthogonality constraints between difference vectors of the form $c$8 for subordination $c$9, with disjoint subtrees yielding orthogonal representation subspaces. Simplicial structure is confirmed empirically: categorical vectors for sibling concepts (e.g., “mammal,” “bird,” “fish”) form affinely independent sets spanning lower-dimensional simplices nested within the larger space. Quantitative containment (barycentric) tests show that ≳95% of tokens in subordinate categories are correctly placed inside the parent polytope. This geometric organization in LMs supports concept editing, steerability, and model interpretability (Park et al., 2024).

3. Spectral and Distributional Origins of Hierarchical Geometry

The geometry of hierarchical concepts in word and LLM embeddings can emerge from distributional statistics, independent of explicit hierarchical supervision. When the Gram matrix $x_p, x_c \in \mathbb{R}^{1+2}$0 of inner products is constructed by applying a monotonically decreasing kernel $x_p, x_c \in \mathbb{R}^{1+2}$1 to hypernym path distances in a taxonomy, its spectral decomposition yields principal components that sequentially partition the space into broad branches (scaling modes) and progressively finer subbranches (wavelet modes). Specifically, on binary trees, the eigenvectors correspond to Haar-like splitting functions that partition categories from root to leaf in a coarse-to-fine order. This structure is observed in both word2vec and Gemma 2B unembedding vectors, where empirical eigenspaces align with theoretical predictions: scaling blocks distinguish top-level splits (e.g., “animal” vs. “plant”), and subsequent modes recursively resolve subcategories (Nava et al., 22 May 2026).

Orthogonality between parent and child contrast vectors also arises in this purely spectral setting, matching the functional orthogonality patterns found in LMs. As a result, hierarchical structure need not be imposed as a functional mechanism; it emerges directly from statistical regularities in pairwise co-occurrence (Nava et al., 22 May 2026).

4. Manifold Geometry: Hyperbolic, Spherical, and Polar Representations

Embedding hierarchies in manifolds with non-Euclidean geometry exploits curvature to efficiently encode structure. Hyperbolic space (Poincaré ball) possesses exponential volume growth, enabling the embedding of trees such that depth is mapped to radial coordinate and category separation is managed by angular position. Distance metrics in hyperbolic space allow deep branches to remain far apart, reflecting the combinatorial explosion of descendants, and empirical results show superior encoding and retrieval accuracy for both seen and unseen classes relative to Euclidean or spherical embeddings (Liu et al., 8 Apr 2025).

Polar/spherical frameworks, such as Polaris, decouple semantic direction (unit vectors on $x_p, x_c \in \mathbb{R}^{1+2}$2) from hierarchical level (normalized radius), enforcing meaning-structure separation. Tangent-space projections and manifold-consistent operations preserve geometry, while geometric and probabilistic triplet losses (e.g., angular Welsch loss, von Mises–Fisher KL divergence) encourage proper semantic clustering and containment. Structure-aware retrieval leverages radial “orbital” gates to constrain ancestry queries efficiently (Mishra et al., 30 Apr 2026).

5. Algebraic, Topological, and Category-Theoretic Encodings

Category theory formalizes concepts and their relationships as objects (points) and morphisms (arrows) within a discrete or topological structure. The “metacategory as discrete geometric space” encodes objects as points in a plane and morphisms as directed (possibly non-composable) 3D paths. The associated Cat-vector space $x_p, x_c \in \mathbb{R}^{1+2}$3 implements non-commutative, partially defined addition (composition), inner product (existence and length of composites), and wedge product (orthogonality). The resulting Cat-algebra inherits Clifford-geometric properties, blending combinatorial and algebraic perspectives (Majkic, 2024).

Within the algebraic-geometric tradition, tropical geometry provides a layered categorical setting. Layered semirings assign a “ghost” layer to each element, capturing degeneracies and multiplicities, crucial for modeling combinatorial hierarchies and their degenerations. Categorical/tropicalization functors map classical varieties and their valuations to layered semirings, preserving the Zariski correspondence and endowing varieties with a hierarchy of multiplicity “layers” invisible in the coarse max-plus structure (Izhakian et al., 2012).

Category-theoretic generalizations also appear in models of conceptual spaces: convex-algebraic frameworks factorize domains (shape, color, etc.) via monoidal categories, with hierarchical inclusion and projection realized by string diagrams. These constructions naturally extend to quantum conceptual spaces, where classical convexity is subsumed by density operators and quantum effects, capturing non-separable structure unavailable in classical settings (Tull et al., 2023).

6. Empirical and Algorithmic Methodologies

Estimation of geometric representations draws from a variety of algorithms:

• Iterative causal embedding for spacetime models—local time “pushes” enforce causal constraints across all parent–child pairs until no violations remain, with light-cone retrieval for hierarchy navigation (Anabalon et al., 7 May 2025).
• Feature extraction in LMs combines whitening of embedding/unembedding spaces, Fisher LDA for feature vector estimation, and barycentric containment testing for polytope membership (Park et al., 2024).
• Spectral analysis leverages SVD/eigendecomposition of Gram matrices, subspace alignments, and topological data analysis (e.g., persistent homology, Wasserstein distances) to assess the coarse-to-fine structure and cross-domain transfer of hierarchy subspaces (Nava et al., 22 May 2026, Sakata et al., 9 Apr 2026).
• Neural and manifold learning methods for polar and hyperbolic embeddings employ tangent-space projections, exponential maps, norm constraints, spherical/hyperbolic layers, and specialized loss functions to capture both symmetry and asymmetric containment (Mishra et al., 30 Apr 2026, Liu et al., 8 Apr 2025).
• Joint geometry–topology pipelines for hierarchical datasets integrate local manifold learning (e.g., diffusion geometry) with topological data analysis (filtration, persistent diagrams) to provide stable, multiscale summaries of categorical and hierarchical structure (Aloni et al., 2021).

7. Implications and Future Directions

Current research shows that the geometry of categorical and hierarchical concepts is deeply grounded in the properties of both the data and the learning algorithms. Hierarchical splitting in vector spaces, emergent from distributional statistics, links classical and deep-learning paradigms. Embeddings that cleanly separate semantic content from structural level (e.g., in polar or hyperbolic representations) enable more robust classification, retrieval, and interpretable manipulation of complex knowledge graphs and taxonomies. Category-theoretic and algebraic approaches provide formal universality, compositionality, and a framework to integrate classical, tropical, and quantum concept geometry.

The geometry of meaning is thus realized across Minkowski, Euclidean, hyperbolic, spherical, and categorical settings, each eliciting distinct but often mathematically unified mechanisms by which discrete hierarchical relations acquire geometric, algebraic, or topological structure. This synthesis has far-reaching implications for the interpretability, transferability, and expansion of artificial and cognitive models of knowledge (Anabalon et al., 7 May 2025, Park et al., 2024, Nava et al., 22 May 2026, Mishra et al., 30 Apr 2026, Majkic, 2024, Izhakian et al., 2012, Aloni et al., 2021, Tull et al., 2023).