Combine ALRA with Sequence-Level Distillation and Other Training Stages

Investigate how Adaptive Local Relational Alignment (ALRA) can be combined with sequence-level distillation or with other stages of language-model training.

Background

Adaptive Local Relational Alignment (ALRA) is presented as a token-level, white-box logit-distillation framework for autoregressive LLMs. Its reported experiments focus on pre-training students using teacher and student output distributions over a shared vocabulary.

The paper explicitly identifies the integration of ALRA with sequence-level distillation and with other stages of language-model training as unresolved future work, leaving open how ALRA's adaptive local-token and pairwise-relational objectives would interact with these broader distillation or training settings.

References

Combining ALRA with sequence-level distillation or other stages of language-model training remains an open direction for future work.

— ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Models  (2609.03355 - Trung et al., 3 Sep 2026) in Conclusion, Section 5