Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
9 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Robustness of the Random Language Model (2309.14913v2)

Published 26 Sep 2023 in cond-mat.dis-nn and cs.CL

Abstract: The Random LLM (De Giuli 2019) is an ensemble of stochastic context-free grammars, quantifying the syntax of human and computer languages. The model suggests a simple picture of first language learning as a type of annealing in the vast space of potential languages. In its simplest formulation, it implies a single continuous transition to grammatical syntax, at which the symmetry among potential words and categories is spontaneously broken. Here this picture is scrutinized by considering its robustness against extensions of the original model, and trajectories through parameter space different from those originally considered. It is shown here that (i) the scenario is robust to explicit symmetry breaking, an inevitable component of learning in the real world; and (ii) the transition to grammatical syntax can be encountered by fixing the deep (hidden) structure while varying the surface (observable) properties. It is also argued that the transition becomes a sharp thermodynamic transition in an idealized limit. Moreover, comparison with human data on the clustering coefficient of syntax networks suggests that the observed transition is equivalent to that normally experienced by children at age 24 months. The results are discussed in light of theory of first-language acquisition in linguistics, and recent successes in machine learning.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (14)
  1. E. L. Post, American journal of mathematics 65, 197 (1943).
  2. N. Chomsky, Syntactic structures (Walter de Gruyter, Berlin, 2002).
  3. D. B. Searls, Nature 420, 211 (2002).
  4. B. Knudsen and J. Hein, Nucleic acids research 31, 3423 (2003).
  5. J. G. Escudero, in Symmetries in Science IX (Springer, Boston, 1997) pp. 139–152.
  6. E. DeGiuli, Phys. Rev. Lett. 122, 128301 (2019).
  7. E. De Giuli, Journal of Physics A: Mathematical and Theoretical 52, 504001 (2019).
  8. E. De Giuli, Journal of Physics A: Mathematical and Theoretical 55, 489501 (2022).
  9. G. Parisi, Statistical field theory (Addison-Wesley, 1988).
  10. K. Nakaishi and K. Hukushima, Physical Review Research 4, 023156 (2022).
  11. D. Imagawa and H. Kawamura, Journal of the Physical Society of Japan 71, 127 (2002).
  12. N. Chomsky, Lectures on government and binding: The Pisa lectures, 9 (Walter de Gruyter, 1993).
  13. T. A. Chang and B. K. Bergen, arXiv preprint arXiv:2303.11504  (2023).
  14. M. Mézard, arXiv preprint arXiv:2309.06947  (2023).
Citations (1)

Summary

We haven't generated a summary for this paper yet.