Ensemble structure of middle layers in large language models

Characterize, within the ensemble-averaging interpretation of Transformer-based large language models trained for next-token prediction, whether the middle layers constitute a single ensemble approximating one function group or a fixed number of ensembles/function groups, and determine how increasing total depth allocates layers among these ensembles.

Background

The authors conclude that ensemble averaging best explains the observed inverse-depth scaling and hidden-state dynamics across most layers and tokens.

They note, however, that multiple ensemble configurations could be consistent with the observed phenomena (e.g., a single ensemble versus multiple fixed ensembles), and their current analyses cannot distinguish between these possibilities.