From Kepler to Newton: Inductive Biases Guide Learned World Models in Transformers
This presentation explores a fundamental question in AI: can Transformers learn true physical laws, or do they merely fit curves? The researchers investigate whether general-purpose Transformer architectures can internalize Newtonian mechanics when modeling planetary motion, revealing that three minimal inductive biases—spatial smoothness, spatial stability, and temporal locality—determine whether these models become genuine physicists or sophisticated curve-fitters. Through systematic experiments, the work demonstrates that predictive accuracy alone does not guarantee mechanistic understanding, and that architectural choices critically shape whether models learn geometric patterns or causal physical principles.Script
Can a Transformer truly understand Newton's laws of motion, or does it simply memorize planetary trajectories? This distinction between curve-fitting and mechanistic understanding lies at the heart of whether AI can become a tool for scientific discovery.
Building on that question, the researchers tackle a crucial problem: high predictive performance doesn't guarantee that a model has learned the underlying physics. Prior studies found that even with impressive accuracy on planetary motion, Transformers failed to internalize Newton's force-based dynamics.
The authors identify three minimal architectural biases that determine whether understanding emerges.
First, spatial smoothness: when you break continuous coordinates into discrete tokens, nearby points in physical space become randomly scattered in the model's representation. The authors found that even exhaustive training can't fully recover a smooth spatial map, creating an inherent tension between geometric fidelity and representational quality.
Second, spatial stability: regression models that predict continuous coordinates tend to spiral into chaos as errors compound. But here's the key insight—by adding moderate noise to the input history during training, the researchers made regression consistently outperform the tokenized classification approach that previous work favored.
Third and most crucial is temporal locality. The authors discovered that context window length fundamentally controls which type of model emerges: long contexts let the Transformer fit global geometric patterns like Kepler, while short contexts of just 2 timesteps force it to internalize Newtonian forces.
This creates a striking dichotomy: with long contexts, the model becomes Kepler, memorizing orbital shapes with impressive smoothness. With short contexts, it becomes Newton, computing forces with nearly perfect internal representations that generalize far better to new situations.
Between these extremes lies a fascinating phase transition: as you dial context length up or down, the model's internal world model smoothly shifts from mechanistic to geometric, revealing that architecture choices don't just affect performance—they determine the very nature of what the model learns.
These findings carry profound implications: they demonstrate that we cannot evaluate world models on prediction alone. For AI to truly discover scientific principles rather than memorize data, we must deliberately architect systems with the right inductive biases—biases that enforce locality, stability, and continuity.
Of course, challenges remain: this work uses controlled planetary motion, and the models don't autonomously output symbolic equations. Scaling these insights to messy, real-world physics and building systems that discover laws end-to-end without human guidance are the next frontiers.
This research reveals a fundamental truth: the path from Kepler to Newton isn't just about more data or bigger models—it's about choosing the right constraints that let genuine understanding emerge. Visit EmergentMind.com to explore more cutting-edge research at the intersection of AI and scientific discovery.
