Evaluate OAL at larger model scales and context lengths

Determine the performance of Outlier-Aware Attention Linearization on models larger than the evaluated 0.5B–1B backbones and on models with longer context lengths, thereby assessing whether its reported attention-approximation and downstream-performance benefits persist at larger scales and contexts.

Background

The paper evaluates Outlier-Aware Attention Linearization (OAL) on Qwen2.5-0.5B and Llama-3.2-1B, reporting improvements in frozen-attention fidelity and in language-modeling and downstream-task performance after LoRA adaptation. The authors explicitly limit the scope of the experiments to these relatively small backbones and state that performance at larger scales has not been verified. They identify evaluation on larger and longer-context models as future work, leaving the scalability and generality of the reported results unresolved.