Variation in encoded information across GPT-OSS layers

Investigate how the information encoded by GPT-OSS representations varies across its 25 layers when approximating symbolic-task inputs with DISCOVER, rather than assuming that all layers encode the same role-filler information.

Background

The GPT-OSS experiments train one DISCOVER model for each of the model’s 25 layers. However, the analysis uses the same hypothesized role-filler structure for corresponding tokens at every layer, while allowing the learned DISCOVER parameters to differ by layer. This uniform structural assumption produces effective approximations, but it may not accurately reflect the information represented at each stage of processing.

The unresolved issue is therefore whether different GPT-OSS layers encode different subsets or types of information, and how those layer-specific representational differences affect the emergent Tensor Product Representation structure identified by DISCOVER.

References

It is likely that different layers in fact differ in what information they encode \citep[e.g.,][]{tenney2019bert}, but we leave investigation of such variation for future work, as the assumption of uniformity---flawed though it might be---produces effective approximations in our experiments.

— The Emergent Symbolic Structure of Artificial Neural Networks  (2608.29530 - McCoy et al., 30 Aug 2026) in Appendix, Section "Technical details for experiments on GPT-OSS in symbolic domains," subsection "DISCOVER"