Transparency Helps Reveal When Language Models Learn Meaning (2210.07468v3)

Abstract: Many current NLP systems are built from LLMs trained to optimize unsupervised objectives on large amounts of raw text. Under what conditions might such a procedure acquire meaning? Our systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations (i.e., languages with strong transparency), both autoregressive and masked LLMs successfully learn to emulate semantic relations between expressions. However, when denotations are changed to be context-dependent with the language otherwise unmodified, this ability degrades. Turning to natural language, our experiments with a specific phenomenon -- referential opacity -- add to the growing body of evidence that current LLMs do not represent natural language semantics well. We show this failure relates to the context-dependent nature of natural language form-meaning mappings.