Evaluation methodologies for long-context abilities in LLMs

Develop rigorous and standardized evaluation methodologies for assessing the long-context abilities of large language models when processing input sequences that exceed their pretraining context window lengths.

Background

The paper introduces SelfExtend, an inference-time method that remaps unseen large relative positions to seen positions to extend the effective context window of LLMs without fine-tuning. While experimental results on perplexity, synthetic passkey retrieval, and real-world benchmarks (LongBench and L-Eval) show improvements, the authors emphasize that perplexity alone is not a reliable indicator of long-context capabilities and that simple synthetic tasks can be insufficient.

Given these observations, the authors explicitly state that methodologies for evaluating long-context abilities are still unresolved. Establishing principled, reliable, and generalizable evaluation frameworks remains necessary to accurately measure how well models understand and utilize information distributed across long sequences.