Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spaces

Abstract: The polysemantic nature of synthetic neurons in artificial intelligence LLMs is currently understood as the result of a necessary superposition of distributed features within the latent space. We propose an alternative approach, geometrically defining a neuron in layer n as a categorical vector space with a non-orthogonal basis, composed of categorical sub-dimensions extracted from preceding neurons in layer n-1. This categorical vector space is structured by the activation space of each neuron and enables, via an intra-neuronal attention process, the identification and utilization of a critical categorical zone for the efficiency of the LLM - more homogeneous and located at the intersection of these different categorical sub-dimensions.