Defining concrete boundaries for “intermediate” Transformer blocks

Develop a clear methodology to define concrete boundaries for the “intermediate” blocks in decoder-only Transformer models such as GPT-2 Large and LLaMa-7B, so that analyses targeting middle-layer behaviors can be conducted on consistently identified layer ranges.

Background

The paper investigates layerwise behavior and often focuses on “intermediate” blocks to examine how representations evolve in the model’s middle stages. However, the authors note that there is no established method to delineate the intermediate portion of a Transformer, making their chosen ranges somewhat arbitrary.

A standardized methodology for identifying intermediate blocks would improve reproducibility and comparability across studies analyzing latent geometry and component behavior in different models and configurations.