Role of data coverage and model capacity in spatial map emergence

Establish whether and how training data coverage across the token vocabulary and model capacity, specifically the embedding dimension, determine the emergence of a spatial map in transformer token embeddings for tokenized coordinate prediction, and identify the conditions under which such a spatial map reliably emerges.

Background

The paper investigates why transformers trained on tokenized planetary motion fail to learn mechanistic world models, identifying the lack of an accurate spatial map in token embeddings as a key obstacle. The authors analyze how the emergence of a spatial map—in the sense of linear directions in the embedding space that correspond to true spatial coordinates—depends on tokenization choices and training setup.

In Section 2.3, they posit that both adequate coverage of the token vocabulary by the training data (data size relative to vocabulary size) and the model’s embedding capacity are crucial determinants of whether such a spatial map emerges. They then study this dependence using a simplified 1D sine-wave dataset and report empirical scaling trends, motivating a formal determination of the precise roles and conditions involved.