Explain representation geometry for hierarchical attributes in language models

Develop a theoretical framework that explains the emergence of geometric structure in model representations for hierarchical attributes in large language models, complementing existing theories for continuous attributes (e.g., time and space) and binary attributes that yield linear analogies.

Background

This work provides a symmetry-based explanation for representation manifolds arising from continuous latent variables and connects to prior theory on binary attributes that explain linear analogies. However, the representational geometry observed for hierarchical attributes in LLMs lacks a comparable theoretical account.

The authors explicitly note that while continuous and binary cases are theoretically addressed, the hierarchical case remains unresolved, and a unified framework that covers all these properties would be desirable.

References

have identified the emergence of remarkable geometric structure for hierarchical attributes that currently remains unexplained - providing a global framework for all these properties would be desirable.

— Symmetry in language statistics shapes the geometry of model representations  (2602.15029 - Karkada et al., 16 Feb 2026) in Limitations

Although the full nature of LLM behavior is not yet fully understood, studies have shown that LLMs partially encode conceptual knowledge in vectorized form, allowing semantic domains to be probed using linear methods .

— Action Sequence Transfer via LLMs for Heterogeneous Environments  (2609.34730 - Chung et al., 28 Sep 2026) in Section II, Related Works, subsection “LLM Comprehension of Hierarchical Concepts”