LongIns Benchmark: Long-Context LLM Evaluation

• LongIns Benchmark is a testing suite designed for evaluating long-context reasoning and instruction-following in LLMs via concatenated QA pairs across set token lengths.
• It implements three evaluation settings—GIST, LIST, and LIMT—to measure context retention, positional instruction effects, and multi-task generalization.
• Results reveal significant F1 score degradation with increased token lengths, exposing instruction-distance failures and the gap between theoretical and effective context windows.

The LongIns Benchmark is a suite for evaluating the long-context capabilities of LLMs, specifically targeting instruction-based multi-task and multi-hop reasoning under extended context window constraints. It is constructed to expose not only retrieval but also reasoning and comprehension failures that standard retrieval-centric benchmarks do not reveal, offering granular insights into effective and claimed context window sizes for commercial and open-source LLMs (Gavin et al., 2024).

1. Benchmark Construction and Formal Specification

LongIns is constructed atop question–answer pools from Super-NaturalInstructions (SNI) and BIG-Bench, with additional synthetic augmentation to ensure comprehensive coverage across seven NLP task types: question answering (QA), classification (Classif), reading comprehension (RC), natural language inference (NLI), machine translation (MT), named entity recognition (NER), and commonsense reasoning (CSR). Each “paper” in the evaluation corpus is created by concatenating question–answer pairs until a predetermined context window length $L$ is met. Lengths considered are $L \in \{256,512,1024,2048,4096,8192,16384\}$ tokens, with $N=1409$ such examples for each $L$.

To create more demanding reading comprehension demands, approximately 10% of answers in every task-specific block are deliberately set to be incorrect. This obviates pattern-matching shortcuts that could be exploited in predominantly correct-item datasets and forces explicit content inspection distributed throughout long contexts. The average question density per 100 tokens is around 1.4, maintaining consistency in information distribution across lengths. Using formal notation: the context for each benchmark instance is defined over $L$, the set of instructions $\mathcal{I}$, and the set of concatenated QA items $\mathcal{Q}_L$.

2. Evaluation Settings: GIST, LIST, LIMT

LongIns introduces three progressively complex prompting regimes designed to isolate context retention, instruction-following under positional degradation, and multi-task generalization:

• Global Instruction Single Task (GIST): The instruction precedes the entire block of questions, which are all of the same task type. Let $i$ be the instruction and $q_j$ the $j$th QA pair of the same type; then the context is $C_{\text{GIST}} = (i, q_1, \ldots, q_m)$, with $I_s = \{i\}$.
• Local Instruction Single Task (LIST): The same instruction is repeated before every QA pair of a single task, i.e., $C_{\text{LIST}} = \bigcup_{j=1}^m (i\Vert q_j)$.
• Local Instruction Multiple Tasks (LIMT): Each question may belong to a different task, with its own specific instruction; $$C_{\text{LIMT}} = \bigcup_{j=1}^m (i_j\Vert q_j), I_s = \{i_1, \dots, i_m\}$$.

These paradigms explicitly test the sensitivity of models to instruction proximity, the difficulty of instruction/task switching mid-document, and the overall robustness across diverse compositional scenarios.

3. Experimental Methodology and Metrics

Models are evaluated on their ability to enumerate indices of all incorrectly answered questions within the constructed “paper.” The primary metric is F1 over predicted versus gold index sets, with precision $P$ and recall $R$ defined respectively as $$P = \frac{|\text{predicted} \cap \text{ground-truth}|}{|\text{predicted}|}$$ and $$R = \frac{|\text{predicted} \cap \text{ground-truth}|}{|\text{ground-truth}|}$$. F1 score is $2PR/(P+R)$.

A wide array of both closed-source (e.g. GPT-4o, GPT-4-Turbo, ERNIE-Speed, Yi-Large-Turbo, Deepseek-Chat, Qwen-Long, Moonshot-v1, GLM-4) and open-source models (e.g. ChatGLM2-6B, MAP-Neo-Ins-v0.1, Llama3-8B-Ins, Yi-34B) spanning 4k to 200k context tokens were evaluated, covering both SOTA and resource-constrained architectures.

4. Results: Effective vs. Advertised Context Windows

GIST results reveal a steep degradation of F1 with increasing context length, far more severe than raw window-size claims suggest. For instance, GPT-4o (advertised 128k) yields:

$L$ (tokens)	F1 (GPT-4o)	F1 (GPT-4-Turbo)	F1 (GPT-3.5-Turbo)
256	70.9	69.6	54.6
4096	52.9	57.5	26.3
16384	31.5	40.9	12.2

Open-source 7B–13B models universally fall below 10% F1 beyond 8192 tokens. Repeating instructions locally (LIST) increases F1 by 5–10 points at all lengths, highlighting acute “instruction-distance” failures; i.e., prompt prefixing is strongly positional and content far from the head is often ignored. Allowing per-item instructions with multi-tasking (LIMT) is easier than GIST but remains below LIST, with F1 at 16k typically 5–8 points higher than GIST.

Analysis further indicates that detection accuracy for “bad answers” drops with increasing “depth” (distance from the context prefix), even for GPT-4o, which loses more than 50% of its early-window F1 when errors appear near the end of a 16k segment. Higher key-information (question) density at fixed $L$ results in catastrophic F1 drop for all but the best GPT-4 variants, underscoring that dense reasoning, not just length, is a major limiting factor.

5. Failure Modes and Qualitative Behavior

Qualitative inspection reveals that smaller models frequently omit target error indices or hallucinate spurious ones in the output set. The effectiveness of instruction-repetition or granularity (LIST) demonstrates that current LLM memory and attention decay rapidly over long contexts. Some models (e.g. Yi-Spark 7B) substantially outperform size-peers at intermediate lengths, but most models collapse well before reaching their token-window theoretical limit. Several large-window open-source models (Mistral-7B-Ins @ 32k) fail entirely above 4k, emphasizing that implementation, not just parameter count or window size, is determinant. Refusal-to-answer behavior (e.g., ERNIE-Speed) deflates F1 and signals interaction between safety tuning and comprehension scoring.

6. Comparison to Existing Long-context Benchmarks

Previous benchmarks such as LongBench, Long-Range Arena, and L-Eval primarily emphasize retrieval from a long context, permitting models to exploit local passage matching rather than demanding global multi-hop reasoning. LongIns, in contrast, is specifically constructed to measure understanding and reasoning over distributed, infrequently signaled, and error-mined information in the presence of instructions at varying positions. While other suites like $\infty$Bench (Zhang et al., 2024) push further in absolute context length (average 200k tokens, multi-domain and bilingual), LongIns uniquely diagnoses instruction-following and non-local judgment, identifying failure thresholds rarely revealed by other evaluation methods.

7. Limitations and Future Directions

Key limitations of LongIns include its synthetic concatenation-based context, which may diverge from natural language documents, codebases, or multi-document aggregations encountered in real-world use. The error-detection formulation does not fully probe generative abilities, abstract summarization, or creative composition. Authors recommend future work towards: (a) higher-heterogeneity, naturally occurring long contexts, (b) evaluation tasks that cover generative and summarization challenges, and (c) architectural innovations in positional encodings and memory systems that preserve information uniformly beyond 100k tokens.

LongIns stands out as a rigorous, discriminative metric for the actual effective context window and reasoning capacity of modern LLMs, establishing a baseline for both theoretical studies of attention decay and practical benchmarks for next-generation model design (Gavin et al., 2024).