Human-scale data sufficiency for emergent compositional abilities

Ascertain whether in-context compositional generalization capabilities observed in large language models pretrained on internet-scale corpora can emerge when models are trained on developmentally plausible, human-lifetime amounts of data.

Background

The authors argue that large pretrained LLMs exhibit emergent in-context learning and compositional behaviors despite training on unstructured text. However, they note that these models are trained on orders of magnitude more data than humans receive. Establishing whether similar capabilities arise with human-scale data would directly bear on debates about the necessity of massive data for compositional generalization and on the plausibility of LLMs as models of human development.