Phase structure of the Random Language Model

Abstract: Context-free grammars are minimal models of hierarchical structure in human language, generating structured text from recursive production rules. The Random LLM (RLM) [De Giuli, PRL 2019], an ensemble of such grammars with random rule weights, exhibits a cross-over from gibberish-like output to structured text as a function of a "temperature", but the location and nature of this transition remained unclear. Here, we show that the RLM exhibits a hierarchy of phase transitions in a double-scaling limit where the grammar temperature $\tildeε_d \to 0$ and the number of hidden symbols $N \to \infty$ at fixed $x = \tildeε_d \log N$. By identifying the relation between RLM and the Random Energy Model, we identify a series of transitions where correlations between symbols emerge, single-symbol marginals become non-uniform, and rule use freezes in a glassy phase. A semi-annealed approximation yields nontrivial scaling laws for rule usage, entropy, and energy, consistent with Heaps' law and context-length scaling observed in LLMs.