Transformative schema creation for RL-trained LLMs

Establish whether reinforcement learning can enable large language models to achieve transformative generalization by creating new solution schemas—"schema creation"—for qualitatively novel cases, such as discovering invariants needed to solve perfectly periodic or degenerate dynamics in the BouncingSim benchmark.

Background

The paper evaluates three axes of generalization—exploratory, compositional, and transformative—on BouncingSim. While RL training yields strong transfer within families and in compositional settings, performance on transformative cases remains near zero, indicating difficulty with qualitatively new dynamics that require novel invariants or solution schemas.

This persistent gap suggests that, unlike structural composition, the creation of new reasoning schemas for transformative shifts is still unresolved, motivating targeted methods capable of discovering and applying such schemas.