LongGenBench: Long-context Generation Benchmark (2410.04199v3)

Abstract: Current long-context benchmarks primarily focus on retrieval-based tests, requiring LLMs to locate specific information within extensive input contexts, such as the needle-in-a-haystack (NIAH) benchmark. Long-context generation refers to the ability of a LLM to generate coherent and contextually accurate text that spans across lengthy passages or documents. While recent studies show strong performance on NIAH and other retrieval-based long-context benchmarks, there is a significant lack of benchmarks for evaluating long-context generation capabilities. To bridge this gap and offer a comprehensive assessment, we introduce a synthetic benchmark, LongGenBench, which allows for flexible configurations of customized generation context lengths. LongGenBench advances beyond traditional benchmarks by redesigning the format of questions and necessitating that LLMs respond with a single, cohesive long-context answer. Upon extensive evaluation using LongGenBench, we observe that: (1) both API accessed and open source models exhibit performance degradation in long-context generation scenarios, ranging from 1.2% to 47.1%; (2) different series of LLMs exhibit varying trends of performance degradation, with the Gemini-1.5-Flash model showing the least degradation among API accessed models, and the Qwen2 series exhibiting the least degradation in LongGenBench among open source models.

• The paper demonstrates that LLMs suffer significant performance degradation in long-context generation, with declines ranging from 1.2% to 47.1%.
• It employs a synthetic benchmark to evaluate generation coherence and model-specific trends, revealing that larger models and robust architectures exhibit lesser drops.
• The study correlates initial baseline performance with resilience, providing actionable insights for enhancing long-context generation capabilities in future LLMs.

Overview of "LongGenBench: Long-context Generation Benchmark"

The paper "LongGenBench: Long-context Generation Benchmark" presents an innovative benchmark designed to address the gap in assessing long-context generation capabilities of LLMs. While existing benchmarks predominantly focus on retrieval tasks, such as the needle-in-a-haystack (NIAH) benchmark, LongGenBench shifts the focus towards evaluating the coherence and contextual accuracy of LLM-generated text across extended passages or documents.

The authors introduce LongGenBench as a synthetic benchmark, which is configured to scrutinize the performance of LLMs in generating logical and consistent responses over long contexts. The research identifies a paucity in benchmarks specifically evaluating long-context generation, as current tools primarily assess retrieval skills rather than the generation of expansive, contextually linked responses. LongGenBench necessitates LLMs to deliver a single cohesive answer that spans several questions, thus simulating a long-answer generation environment.

Key Findings

In conducting this evaluation, the paper outlines a series of methodologically robust experiments and presents several findings:

1. Performance Degradation: Both API-accessed and open-source LLMs show significant performance degradation when tasked with long-context generation. The degradation rates ranged from as low as 1.2% to a high of 47.1%.
2. Model-Specific Trends: Models exhibit varying levels of degradation. Among API-specific models, Gemini-1.5-Flash exhibited the smallest performance decline, while the Qwen2 series showed minimal decline among open-source models.
3. Correlation with Baseline Performance: The paper identifies a correlation between initial baseline performances and the extent of degradation under long-context scenarios, where higher baseline scores generally equate to lesser performance drops.
4. Impact of Model Size: Larger models within series—such as in the LLaMA-3 and Qwen2—tend to suffer lesser degradation, illustrating the impact of model size on performance resilience.
5. Architectural Variance: Different model architectures display distinct trends in performance degradation, underscoring the importance of architectural design in the robustness of LLMs for long-context tasks. For example, LLaMA-3-8B-Instruct degraded by 47.1% in performance on the GSM8K dataset, while ChatGLM4-9B-Chat showed a smaller decline despite similar baselines.

Implications and Future Directions

The insights from LongGenBench emphasize the importance of devising models with enhanced long-context generation abilities, which remain robust across diverse data sequences. The analysis uncovers the critical relationships and factors affecting model performance, providing a foundational basis for future improvements in the design and training of LLMs.

The benchmark proposed in this paper lays the groundwork for further research into the structural and algorithmic enhancements needed to support long-context generation. The nuanced understanding of performance degradation offers valuable guidance for optimizing transformer architecture and long-context mechanisms.

As the field of natural language processing continues to evolve, frameworks like LongGenBench will be pivotal in steering the direction of LLM improvements and adaptations. By transparently elucidating model limitations and coherency challenges, benchmarks such as LongGenBench promise to refine model development strategies and ultimately enhance the capability of LLMs to generate logically consistent long-form textual content across various applications.