Can metadata enhance post-training?

Determine whether incorporating document-level metadata during post-training procedures for large language models (e.g., instruction tuning or related post-training stages) enhances performance or training efficiency compared to post-training without metadata.

Background

The study focuses on pretraining and demonstrates acceleration effects from various metadata types and positions. However, the authors explicitly flag uncertainty about benefits in post-training stages, which are operationally distinct from pretraining and include procedures like instruction tuning or preference optimization.

This question is highlighted in the conclusion as an unresolved direction, indicating potential applicability of metadata beyond pretraining but without current evidence.