2000 character limit reached
Position Interpolation Improves ALiBi Extrapolation (2310.13017v1)
Published 18 Oct 2023 in cs.CL, cs.AI, and cs.LG
Abstract: Linear position interpolation helps pre-trained models using rotary position embeddings (RoPE) to extrapolate to longer sequence lengths. We propose using linear position interpolation to extend the extrapolation range of models using Attention with Linear Biases (ALiBi). We find position interpolation significantly improves extrapolation capability on upstream LLMling and downstream summarization and retrieval tasks.