Limits of Learning from Text-Only Corpora

Determine which competencies large language models can and cannot learn solely from large text corpora, without multi-modal data or embodied interaction, in light of their surprising pattern of successes and failures.

Background

In the Day Two discussion, participants asked what abilities LLMs can acquire from text alone. Andrew Lampinen noted the unpredictable success-failure profile of such models and expressed uncertainty about the boundaries of text-only learning.

Clarifying these limits would inform whether additional modalities, embodiment, or structured feedback are necessary to achieve specific forms of generalization and understanding.

References

The open discussion began with a request for Lampinen's own opinion about what can and can't be learned from large language corpora. Lampinen was unsure, but noted the often surprising pattern of success and failure in LLMs: these systems often fail in some seemingly easy tasks while succeeding in some seemingly difficult tasks.

— Embodied, Situated, and Grounded Intelligence: Implications for AI  (2210.13589 - Millhouse et al., 2022) in General Discussion — Day Two

Since LLMs aren't people, we don't exactly know what they could have learned while acquiring the level of linguistic competence they exhibit, but we can consider people as a kind of outer bound: tasks that no human acquires through language exposure alone are unlikely to have been learned by PTGA models.

— LLMs are not stochastic parrots: Evidence for meaning-mediated abstraction from conlang-like tasks  (2609.34187 - Zimmerman et al., 28 Sep 2026) in Section 2.4, “Rule design”

Although we have ruled out, by negation, other explanations for the gaps (e.g. too few criteria, criteria stated badly, weak judges, weak aggregators, etc., App.~\ref{sec:res-frontier}), it is unclear whether LLMs can acquire all forms of human tacit knowledge with current learning mechanisms.

— Verifiable, Articulable, and Tacit Components of Preference  (2610.03025 - Spangher et al., 2 Oct 2026) in Discussion, Section 7 (paragraph beginning “Humans accommodate for underspecification”)

And what if such approaches are simply too data inefficient to enable such rich capabilities to emerge as \citet{polanyi1958personal} depicts? There are other areas of tacit knowledge acquisition that are poorly understood even in humans, and might yield novel approaches.

— Verifiable, Articulable, and Tacit Components of Preference  (2610.03025 - Spangher et al., 2 Oct 2026) in Recommendations and Future Work #2 (Learning)