Establish pretraining frequency–omission association beyond numbers (e.g., non-standard dialects)

Establish whether the frequency–omission association observed for numerical tokens—namely, that higher omission rates in reconstructions from pretrained Mamba state space language models correlate with lower counts per unique token in the pretraining corpus—generalizes to other types of information loss, including tokens from non-standard English dialects such as African American Vernacular English, using appropriate dialect taggers applied to The Pile (or Pile Uncopyrighted).

Background

The paper analyzes selective memory in pretrained Mamba LLMs by training auto-encoders to reconstruct inputs from hidden states and measuring omissions. It finds that numerical tokens and certain named entities are disproportionately forgotten. A training corpus analysis on a Pile Uncopyrighted sample shows an association between omission rates and pretraining frequencies for numbers (NUM), suggesting rarity contributes to forgetting. However, due to the lack of dialect taggers, the authors were unable to test whether similar frequency–omission relationships hold for other information types, particularly non-standard English dialects such as AAVE. Establishing this connection would clarify whether the observed pattern for numbers extends to broader categories of information loss.