LongICLBench: Evaluating Long In-Context Learning

• LongICLBench is a benchmark that tests long in-context learning by requiring models to induce tasks from extensive, multi-class demonstrations.
• It employs extreme-label classification across six datasets with prompt lengths up to 50K tokens, challenging models to integrate full prompt context.
• Empirical results reveal that even advanced models like GPT4-turbo struggle on tasks such as Discovery, highlighting limitations in current long-context integration.

LongICLBench is a benchmark for evaluating long in-context learning in LLMs through extreme-label classification. It was introduced in “Long-context LLMs Struggle with Long In-context Learning” to test whether a model can infer a task from a long prompt containing many labeled demonstrations, rather than merely retrieve a local fact or exploit short-range cues. The benchmark comprises six datasets with 28 to 174 classes and prompt lengths ranging from roughly 0.8K/1K to 50K tokens, depending on the number of demonstration rounds. Its central claim is that existing long-context evaluations such as perplexity measurements, synthetic retrieval tests, and many long-document QA or summarization settings do not force a model to integrate information across the full prompt in order to determine the complete label space; LongICLBench is designed to do so (Li et al., 2024).

1. Conceptual objective

LongICLBench targets a specific capability: many-class task induction from long demonstrations. In the benchmark’s framing, conventional long-context evaluations are inadequate for three different reasons. First, long-document perplexity is treated as only a weak proxy for understanding. Second, synthetic retrieval tests such as passkey retrieval or needle-in-a-haystack are described as too easy and unrealistic, with many models scoring above 99%. Third, long-document QA and summarization often admit shortcuts, because answers can sometimes be recovered from a short span or because summarization can exploit strong positional bias, especially toward leading sentences (Li et al., 2024).

The benchmark therefore reformulates long-context evaluation as extreme-label classification via in-context learning. A model is given a natural-language instruction, a long sequence of labeled demonstrations, and a final unlabeled test instance. To succeed, it must infer the relevant label inventory and the distinctions among labels from the demonstrations themselves. This is especially demanding when the label space is large. The most extreme dataset in the suite, Discovery, has 174 classes and about 61 tokens per example, so even 1 shot per label already exceeds 10K tokens (Li et al., 2024).

This design makes LongICLBench distinct from long-context benchmarks centered on retrieval, summarization, or open-ended document QA. A plausible implication is that the benchmark measures not only whether a model can maintain a long context window, but whether it can use that window as a task specification substrate. In that sense, the benchmark is closer to testing prompt-induced category formation than document comprehension in the ordinary extractive sense.

2. Benchmark construction and task format

Prompt construction is organized in rounds. A 1-round (1R) prompt contains one example for each label or class in the dataset; 2R contains two examples per label; and so on through 5R. Demonstrations are sampled evenly across labels, so the prompt is balanced rather than reflecting natural class frequency. The evaluation set for each dataset consists of 500 test examples, again sampled with even label distribution (Li et al., 2024).

The six datasets span several task families, but all are cast as classification or structured prediction from demonstrations:

Dataset	Classes	Prompt length regime
GoEmotion	28	0.8K/1K to 4K
BANKING77	77	2K to 14K
TacRED	41	4K to 18K
Few-NERD	66	5K to 24K
DialogRE	36	8K to 32K
Discovery	174	10K to 50K

The benchmark summary table reports prompt ranges such as [1K, 4K] for GoEmotion, [2K, 11K] for BANKING77, [4K, 18K] for TacRED, [5K, 23K] for Few-NERD, [8K, 32K] for DialogRE, and [10K, 50K] for Discovery; the appendix provides more precise per-round values for several datasets, including 0.8K, 1.6K, 2.4K, 3.2K, 4K for GoEmotion and 5K, 9K, 14K, 19K, 24K for Few-NERD (Li et al., 2024).

Per-example length varies substantially across datasets. GoEmotion and BANKING77 average 28 tokens per shot; TacRED averages 80; Few-NERD averages 61; DialogRE averages 226; and Discovery averages 61. This means difficulty is driven by both label-space size and tokens per shot. DialogRE is hard despite having only 36 labels, because each demonstration is long. Discovery is hardest because it combines a very large label space with long total prompts (Li et al., 2024).

The output format depends on the dataset. GoEmotion requires one emotion label, BANKING77 one intent, TacRED one relation between a subject and object, Few-NERD named entities and their types in the required format, DialogRE relation outputs for entity pairs in dialogue, and Discovery the discourse marker or conjunction connecting two sentences. Dataset-specific prompt templates are used rather than a single universal template (Li et al., 2024).

3. Evaluation protocol and model suite

LongICLBench uses standard task metrics rather than introducing a new benchmark-specific scoring rule. Accuracy is used for GoEmotion, BANKING77, and Discovery; F1 is used for TacRED, Few-NERD, and DialogRE. The benchmark also reports task-specific supervised reference systems as upper-bound task baselines, including BERT = 58.9 on GoEmotion, RoBERTA + ICDA = 94.4 on BANKING77, DeepStruct = 76.8 on TacRED, PL-Marker = 70.9 on Few-NERD, HiDialog = 77.1 on DialogRE, and MTL = 87.4 on Discovery (Li et al., 2024).

The provided model tables list 13 named long-context LLMs:

1. Gemma-7B-base
2. LLaMA-2-7B-32K
3. ChatGLM3-6B-32K
4. Qwen-1.5-7B-base
5. Mistral-7B-v0.2-base
6. LLaMA-2-7B-LongLora
7. Yi-6B-200K
8. InternLM2-7B-base
9. Long-LLaMA-code-7B
10. RWKV-5-World
11. Mamba-2.8B
12. Gemini-1.0-Pro
13. GPT4-turbo

The abstract, however, states that the benchmark evaluates 15 long-context LLMs, whereas the model list in the supplied tables contains 13. The context-window settings reported in the tables range from 8K for Gemma-7B-base to 256K for Long-LLaMA-code-7B, with RWKV-5-World and Mamba-2.8B described as effectively infinite. Most open-source systems are around 7B parameters, and the paper states that it uses base models before instruction tuning, except for Gemini and GPT4-turbo (Li et al., 2024).

For reproducibility, the benchmark fixes the demonstration sample set across models for each dataset and round. Open-source models are loaded from HuggingFace weights, while proprietary models are accessed through the official OpenAI and Google Vertex AI APIs. The project code is released at https://github.com/TIGER-AI-Lab/LongICLBench (Li et al., 2024).

4. Empirical findings

The principal empirical result is that long advertised context windows do not imply strong long in-context learning. On relatively manageable tasks such as BANKING77, several models benefit from increasing prompt length. GPT4-turbo rises from 73.5 at 1R to 84.4 at 5R. Among open-source models, LLaMA-2-7B-32K improves from 30.2 to 77.2, Qwen-1.5-7B-base from 21.6 to 67.8, and Mistral-7B-v0.2-base reaches the high 60s before declining slightly. The paper interprets this as evidence that easier long-ICL tasks can benefit from additional demonstrations, especially when prompt length grows from around 2K to 4K tokens (Li et al., 2024).

On TacRED, the pattern is less favorable. GPT4-turbo improves from 74.4 to 84.2, and Gemini-1.0-Pro remains competitive in the 71.4–78.2 range, but many open models peak at intermediate prompt lengths and then deteriorate. The benchmark explicitly notes that longer context does not uniformly help; several long-context extension models remain near zero or low single digits (Li et al., 2024).

On DialogRE, where each shot is long, the task becomes substantially harder. GPT4-turbo increases from 43.5 to 60.9 across 8K–32K prompts, but most open-source models remain far lower, often below the mid-20s. On Few-NERD, open models perform more respectably—Mistral reaches 50.0, InternLM2 48.3, and Qwen 47.8—but they still trail the supervised PL-Marker = 70.9 reference (Li et al., 2024).

The most consequential result concerns Discovery. The paper’s narrative states that all models fail on this dataset. Representative values underscore the point: GPT4-turbo records 1.5, 0.5, 0.5, 0.5, 0.5 across rounds, and many other models are at 0 throughout. The paper notes that some table entries appear anomalous, but interprets the overall outcome unambiguously: no evaluated model can reliably understand the long demonstration prompt for the 174-label Discovery task (Li et al., 2024).

Across the benchmark, transformer-based models outperform RWKV and Mamba, and GPT4-turbo is the strongest overall system. Yet even GPT4-turbo fails on the hardest setting. This suggests that the bottleneck is not merely raw context length support. The paper repeatedly emphasizes that performance often degrades as task complexity rises, and that some models improve only up to around 20K tokens before plateauing or declining (Li et al., 2024).

5. Diagnostic analyses and failure modes

A distinguishing feature of LongICLBench is its analysis of label-position effects. The benchmark performs a controlled comparison on TacRED, 3 rounds, with 41 labels, 3 examples per label, and prompt length around 10K tokens. In the default scattered condition, instances of the same label are distributed across the prompt. In the grouped condition, all examples for a label are placed next to each other. This intervention directly probes whether the model can aggregate multiple pieces of evidence for the same class while maintaining a global representation of the overall label space (Li et al., 2024).

Grouping often hurts performance sharply. The reported scattered-to-grouped changes include ChatGLM3-6B-32K: 38.9 → 35.6, Qwen-1.5-7B-base: 45.2 → 33.0, Mistral-7B-v0.2-base: 51.6 → 5.1, InternLM2-7B-base: 15.5 → 4.8, and GPT4-turbo: 79.5 → 59.2. Only a few models improve slightly, such as LLaMA-2-7B-32K: 0.4 → 3.0 and RWKV-5-World: 1.0 → 3.6, and these improvements occur from very low baselines (Li et al., 2024).

The paper links these results to “Lost in the Middle”-style positional effects and concludes that some models exhibit a marked recency bias toward later-presented labels. It specifically states that prompt label position can “dramatically influence” performance, and that some systems, including InternLM2-7B-base, effectively handle only labels positioned near the end of the prompt. This is important because LongICLBench is intended to require full-prompt integration; later-label bias is therefore evidence that many models are not maintaining a uniform usable representation across the entire demonstration sequence (Li et al., 2024).

The grouped-versus-scattered results also indicate a broader failure to aggregate multiple supporting examples into a stable class representation. Simply presenting all evidence for a class together does not help and often makes performance much worse. The benchmark interprets this as a failure of long-range reasoning over demonstrations rather than a simple memory-capacity problem. A plausible implication is that current long-context models can often retain local associations, yet still fail to synthesize a coherent many-class decision boundary from a very long prompt.

6. Position in the benchmark landscape

LongICLBench occupies a specific niche within long-context evaluation. It differs from LongBench-style long-context understanding suites, which emphasize broad task coverage across QA, summarization, code, retrieval, and synthetic tasks. MiniLongBench later showed that LongBench can be compressed to 237 samples while preserving model rankings with average Spearman correlation around 0.97; that work is explicitly a benchmark-compression method rather than a new long-context task formulation, whereas LongICLBench is a purpose-built stress test for demonstration-based long-context ICL (Huang et al., 26 May 2025).

It also differs from 100-LongBench, which argues that many de facto long-context benchmarks conflate baseline task strength with genuine long-context robustness and proposes LongScore on a length-controllable benchmark to separate those factors (Yang et al., 25 May 2025). LongICLBench already varies prompt length through demonstration rounds, but it does not decompose results into baseline ability versus normalized long-context retention. This suggests, though does not prove, that a length-controlled or normalized extension of LongICLBench could further sharpen cross-model interpretation.

Relative to ICLEval, LongICLBench is more realistic as a long-prompt stress test but less mechanistically isolated. ICLEval focuses on exact copying and rule learning in mostly synthetic short-to-moderate contexts, with strong contamination controls and exact-match scoring (Chen et al., 2024). By contrast, LongICLBench uses real NLP datasets and long prompts, but it evaluates one specific task family—extreme-label classification—rather than core ICL primitives in isolation.

Relative to LIFBench, LongICLBench is demonstration-based rather than instruction-execution-centered. LIFBench evaluates long-context instruction following and stability across paraphrases, variables, and lengths, but it is explicitly zero-shot and does not use few-shot task induction from demonstrations (Wu et al., 2024). Relative to LifeBench, LongICLBench is a prompt-bounded ICL benchmark rather than a long-horizon multi-source memory benchmark for agents operating over year-scale personal traces (Cheng et al., 4 Mar 2026).

The closest adjacent benchmark is MIR-Bench, which explicitly positions itself as going beyond LongICLBench’s classification-only many-shot setting. MIR-Bench evaluates latent function induction from up to 2048 shots and argues that the main failure mode in many-shot reasoning is not pure retrieval but distraction when aggregating too much information (Yan et al., 14 Feb 2025). Taken together, the two benchmarks trace a progression: LongICLBench shows that long-context models struggle to infer large label spaces from long demonstrations, while MIR-Bench extends the same concern from classification to general input-output pattern induction.

LongICLBench’s enduring significance lies in its diagnosis of a gap between nominal context-window size and effective prompt-level learning. Its results suggest that long-context competence cannot be established solely through retrieval tests or fixed-context QA. A model may accept a 32K-, 100K-, or 200K-token input and still fail to learn a difficult task from demonstrations distributed across that input. Within the long-context literature, LongICLBench therefore functions as a benchmark for whether a model can use a long prompt not merely as storage, but as a coherent source of supervision (Li et al., 2024).