Layer-wise and training-dynamics analysis of morpho-syntactic representations in BLOOM models

Analyze, across layers and over the course of pretraining, how morpho-syntactic representations are formed in the 176B-parameter BLOOM model and the 1.7B-parameter BLOOM-1B7 model to characterize representational development and property acquisition dynamics.

Background

The reported probing results focus on averaged representations and end-of-training checkpoints. Prior work suggests that different layers and training stages can encode different types of information, implying that BLOOM models might exhibit developmental trajectories in representing morpho-syntactic features.

A systematic layer-wise and temporal analysis would clarify where and when specific linguistic properties emerge, providing insights into model interpretability and guiding architectural or training improvements.