Overcoming the Reversal Curse in Knowledge Generalization

Determine mechanisms that enable autoregressive language models to generalize from training on facts of the form "A is B" to correctly infer their reversals "B is A" (and analogous inverses), without requiring exhaustive data augmentation.

Background

The Reversal Curse describes a failure of autoregressive LLMs to generalize bidirectionally from asymmetric training data, e.g., from "A is B" to "B is A." The authors present it as a core open problem tied to the unidirectionality of conditional probability learning, which may impede simple logical generalization. Although some mitigation strategies exist, they are costly and do not fundamentally address the issue.