Hebbian learning the local structure of language (2503.02057v1)

Abstract: Learning in the brain is local and unsupervised (Hebbian). We derive the foundations of an effective human LLM inspired by these microscopic constraints. It has two parts: (1) a hierarchy of neurons which learns to tokenize words from text (whichiswhatyoudowhenyoureadthis); and (2) additional neurons which bind the learned symanticless patterns of the tokenizer into a symanticful token (an embedding). The model permits continuous parallel learning without forgetting; and is a powerful tokenizer which performs renormalization group. This allows it to exploit redundancy, such that it generates tokens which are always decomposable into a basis set (e.g an alphabet), and can mix features learned from multiple languages. We find that the structure of this model allows it to learn a natural language morphology WITHOUT data. The language data generated by this model predicts the correct distribution of word-forming patterns observed in real languages, and further demonstrates why microscopically human speech is broken up into words. This model provides the basis for understanding the microscopic origins of language and human creativity.

Analysis of "Hebbian Learning the Local Structure of Language"

The paper "Hebbian Learning the Local Structure of Language," authored by P. Myles Eugenio, offers a model of human language grounded in Hebbian learning, which is characterized by local and unsupervised processes. The research integrates neuronal hierarchies to tokenize written text and subsequently binds syntactic patterns into semantically rich tokens, termed embeddings. The collective framework contributes to an understanding of language development from minimalistic forms, similar to cases of spontaneous language emergence, such as Nicaraguan Sign Language.

This approach contrasts starkly with current LLMs, which necessitate colossal datasets for training. In contrast, the Hebbian LLM elucidates long-term correlations intrinsic to language, without reliance on substantial pre-existing data. The model posits a hierarchical, unsupervised reinforcement of correlations, beginning with basic symbolic correlations and building more complex structures through successive layers, similar to biological neuron hierarchies.

Structural and Theoretical Considerations

1. Hierarchical Hebbian Model: The model leverages a hierarchy of neurons which learn word tokenization via Hebbian rules. It starts from basic correlations like bigrams and extrapolates to n-grams across higher hierarchy levels. The progression reflects a constrained retokenization process, inherent to the formalism of unsupervised neurological learning.
2. Natural Language Mimicking: As the model undergoes training with random strings, it surfaces a tokenizable "morphology" that mimics natural languages. This suggests underlying neural encoding mechanisms may be imprinted as such a linguistic structure, proposing an endogenous reason for the hierarchy seen in human language.
3. Smoothness and Tokenization: It is significant that the proposed model requires learned n-grams to be derivations of smaller learned units, enforcing a scaling constraint analogous to the hierarchical chunking seen in human languages.
4. Practical Learnability and Replay Mechanisms: The capability for learning is hampered at scale by the dimension explosion of projection matrices. The paper mitigates this computational burden via a neural replay mechanism, fostering continuous learning without forgetting. This effectively creates embeddings through Hebbian replay cycles that tie detected patterns into a cohesive framework, explaining the empirical tendencies of localized language patterns.
5. Simulation and Complexity Management: With the replay mechanism and parallel independence of embeddings, the model achieves a significant compression and disentanglement of memory networks, enabling the practical learnability of increasingly complex structures. This compression arises from disentangled hierarchical encoding into the synaptic networks of added neurons.

Potential Implications and Future Directions

These insights afford a new direction for understanding the micromechanical origins of language within the neural system. The model is inherently scalable and draws parallels between biological learning processes and computational language acquisition, suggesting theoretical convergence on integrated memory models characterized via key-value memory frameworks. There is anticipation of deciphering linguistic morphology not just as a cultural artifact but as an imprint of fundamental brain physiology, which might have broader implications across cognitive and neurobiological studies of language.

For future work, empirical testing of biological plausibility remains critical. There is potential for the adaptation of such models to practical AI settings, offering a neural network architecture that more closely resembles the efficient learning observed in organic neural systems. The correlation between language data distributions and synaptic learning parameters should also be clarified to better characterize constraints that manifest in human speech parameters.

Overall, this paper provides a framework for reinterpreting language through a lattice of nested and partially independent neural computations, spotlighting a reconceptualization of theories about the emergence and chronology of linguistic data.