---
title: 'LongICLBench: Evaluating Long In-Context Learning'
url: https://www.emergentmind.com/topics/longiclbench
type: topic
---

# LongICLBench: Evaluating Long In-Context Learning

LongICLBench is a benchmark for evaluating **long in-context learning** in large language models through **extreme-label classification**. It was introduced in “Long-context LLMs Struggle with Long In-context Learning” to test whether a model can infer a task from a long prompt containing many labeled demonstrations, rather than merely retrieve a local fact or exploit short-range cues. The benchmark comprises six datasets with **28 to 174 classes** and prompt lengths ranging from roughly **0.8K/1K to 50K tokens**, depending on the number of demonstration rounds. Its central claim is that existing long-context evaluations such as perplexity measurements, synthetic retrieval tests, and many long-document QA or summarization settings do not force a model to integrate information across the full prompt in order to determine the complete label space; LongICLBench is designed to do so [2404.02060].

## 1. Conceptual objective

LongICLBench targets a specific capability: **many-class task induction from long demonstrations**. In the benchmark’s framing, conventional long-context evaluations are inadequate for three different reasons. First, long-document perplexity is treated as only a weak proxy for understanding. Second, synthetic retrieval tests such as passkey retrieval or needle-in-a-haystack are described as too easy and unrealistic, with many models scoring above 99%. Third, long-document QA and summarization often admit shortcuts, because answers can sometimes be recovered from a short span or because summarization can exploit strong positional bias, especially toward leading sentences [2404.02060].

The benchmark therefore reformulates long-context evaluation as **extreme-label classification via in-context learning**. A model is given a natural-language instruction, a long sequence of labeled demonstrations, and a final unlabeled test instance. To succeed, it must infer the relevant label inventory and the distinctions among labels from the demonstrations themselves. This is especially demanding when the label space is large. The most extreme dataset in the suite, **Discovery**, has **174 classes** and about **61 tokens per example**, so even **1 shot per label** already exceeds **10K tokens** [2404.02060].

This design makes LongICLBench distinct from long-context benchmarks centered on retrieval, summarization, or open-ended document QA. A plausible implication is that the benchmark measures not only whether a model can maintain a long context window, but whether it can use that window as a **task specification substrate**. In that sense, the benchmark is closer to testing prompt-induced category formation than document comprehension in the ordinary extractive sense.

## 2. Benchmark construction and task format

Prompt construction is organized in **rounds**. A **1-round (1R)** prompt contains **one example for each label or class** in the dataset; **2R** contains two examples per label; and so on through **5R**. Demonstrations are sampled **evenly across labels**, so the prompt is balanced rather than reflecting natural class frequency. The evaluation set for each dataset consists of **500 test examples**, again sampled with **even label distribution** [2404.02060].

The six datasets span several task families, but all are cast as classification or structured prediction from demonstrations:

| Dataset | Classes | Prompt length regime |
|---|---:|---|
| GoEmotion | 28 | 0.8K/1K to 4K |
| BANKING77 | 77 | 2K to 14K |
| TacRED | 41 | 4K to 18K |
| Few-NERD | 66 | 5K to 24K |
| DialogRE | 36 | 8K to 32K |
| Discovery | 174 | 10K to 50K |

The benchmark summary table reports prompt ranges such as **[1K, 4K]** for GoEmotion, **[2K, 11K]** for BANKING77, **[4K, 18K]** for TacRED, **[5K, 23K]** for Few-NERD, **[8K, 32K]** for DialogRE, and **[10K, 50K]** for Discovery; the appendix provides more precise per-round values for several datasets, including **0.8K, 1.6K, 2.4K, 3.2K, 4K** for GoEmotion and **5K, 9K, 14K, 19K, 24K** for Few-NERD [2404.02060].

Per-example length varies substantially across datasets. **GoEmotion** and **BANKING77** average **28 tokens per shot**; **TacRED** averages **80**; **Few-NERD** averages **61**; **DialogRE** averages **226**; and **Discovery** averages **61**. This means difficulty is driven by both **label-space size** and **tokens per shot**. DialogRE is hard despite having only **36 labels**, because each demonstration is long. Discovery is hardest because it combines a very large label space with long total prompts [2404.02060].

The output format depends on the dataset. **GoEmotion** requires one emotion label, **BANKING77** one intent, **TacRED** one relation between a subject and object, **Few-NERD** named entities and their types in the required format, **DialogRE** relation outputs for entity pairs in dialogue, and **Discovery** the discourse marker or conjunction connecting two sentences. Dataset-specific prompt templates are used rather than a single universal template [2404.02060].

## 3. Evaluation protocol and model suite

LongICLBench uses standard task metrics rather than introducing a new benchmark-specific scoring rule. **Accuracy** is used for **GoEmotion**, **BANKING77**, and **Discovery**; **F1** is used for **TacRED**, **Few-NERD**, and **DialogRE**. The benchmark also reports task-specific supervised reference systems as upper-bound task baselines, including **BERT = 58.9** on GoEmotion, **RoBERTA + ICDA = 94.4** on BANKING77, **DeepStruct = 76.8** on TacRED, **PL-Marker = 70.9** on Few-NERD, **HiDialog = 77.1** on DialogRE, and **MTL = 87.4** on Discovery [2404.02060].

The provided model tables list **13 named long-context LLMs**:

1. Gemma-7B-base  
2. LLaMA-2-7B-32K  
3. ChatGLM3-6B-32K  
4. Qwen-1.5-7B-base  
5. Mistral-7B-v0.2-base  
6. LLaMA-2-7B-LongLora  
7. Yi-6B-200K  
8. InternLM2-7B-base  
9. Long-LLaMA-code-7B  
10. RWKV-5-World  
11. Mamba-2.8B  
12. Gemini-1.0-Pro  
13. GPT4-turbo  

The abstract, however, states that the benchmark evaluates **15 long-context LLMs**, whereas the model list in the supplied tables contains **13**. The context-window settings reported in the tables range from **8K** for Gemma-7B-base to **256K** for Long-LLaMA-code-7B, with **RWKV-5-World** and **Mamba-2.8B** described as effectively **infinite**. Most open-source systems are around **7B parameters**, and the paper states that it uses **base models before instruction tuning**, except for **Gemini** and **GPT4-turbo** [2404.02060].

For reproducibility, the benchmark fixes the demonstration sample set across models for each dataset and round. Open-source models are loaded from **HuggingFace** weights, while proprietary models are accessed through the official **OpenAI** and **Google Vertex AI** APIs. The project code is released at `https://github.com/TIGER-AI-Lab/LongICLBench` [2404.02060].

## 4. Empirical findings

The principal empirical result is that **long advertised context windows do not imply strong long in-context learning**. On relatively manageable tasks such as **BANKING77**, several models benefit from increasing prompt length. **GPT4-turbo** rises from **73.5** at **1R** to **84.4** at **5R**. Among open-source models, **LLaMA-2-7B-32K** improves from **30.2** to **77.2**, **Qwen-1.5-7B-base** from **21.6** to **67.8**, and **Mistral-7B-v0.2-base** reaches the high **60s** before declining slightly. The paper interprets this as evidence that easier long-ICL tasks can benefit from additional demonstrations, especially when prompt length grows from around **2K to 4K** tokens [2404.02060].

On **TacRED**, the pattern is less favorable. **GPT4-turbo** improves from **74.4** to **84.2**, and **Gemini-1.0-Pro** remains competitive in the **71.4–78.2** range, but many open models peak at intermediate prompt lengths and then deteriorate. The benchmark explicitly notes that longer context does not uniformly help; several long-context extension models remain near zero or low single digits [2404.02060].

On **DialogRE**, where each shot is long, the task becomes substantially harder. **GPT4-turbo** increases from **43.5** to **60.9** across **8K–32K** prompts, but most open-source models remain far lower, often below the mid-20s. On **Few-NERD**, open models perform more respectably—**Mistral** reaches **50.0**, **InternLM2** **48.3**, and **Qwen** **47.8**—but they still trail the supervised **PL-Marker = 70.9** reference [2404.02060].

The most consequential result concerns **Discovery**. The paper’s narrative states that **all models fail** on this dataset. Representative values underscore the point: **GPT4-turbo** records **1.5, 0.5, 0.5, 0.5, 0.5** across rounds, and many other models are at **0** throughout. The paper notes that some table entries appear anomalous, but interprets the overall outcome unambiguously: no evaluated model can reliably understand the long demonstration prompt for the **174-label** Discovery task [2404.02060].

Across the benchmark, **transformer-based models outperform RWKV and Mamba**, and **GPT4-turbo** is the strongest overall system. Yet even GPT4-turbo fails on the hardest setting. This suggests that the bottleneck is not merely raw context length support. The paper repeatedly emphasizes that performance often degrades as task complexity rises, and that some models improve only up to around **20K** tokens before plateauing or declining [2404.02060].

## 5. Diagnostic analyses and failure modes

A distinguishing feature of LongICLBench is its analysis of **label-position effects**. The benchmark performs a controlled comparison on **TacRED, 3 rounds**, with **41 labels**, **3 examples per label**, and prompt length around **10K** tokens. In the default **scattered** condition, instances of the same label are distributed across the prompt. In the **grouped** condition, all examples for a label are placed next to each other. This intervention directly probes whether the model can aggregate multiple pieces of evidence for the same class while maintaining a global representation of the overall label space [2404.02060].

Grouping often hurts performance sharply. The reported scattered-to-grouped changes include **ChatGLM3-6B-32K: 38.9 → 35.6**, **Qwen-1.5-7B-base: 45.2 → 33.0**, **Mistral-7B-v0.2-base: 51.6 → 5.1**, **InternLM2-7B-base: 15.5 → 4.8**, and **GPT4-turbo: 79.5 → 59.2**. Only a few models improve slightly, such as **LLaMA-2-7B-32K: 0.4 → 3.0** and **RWKV-5-World: 1.0 → 3.6**, and these improvements occur from very low baselines [2404.02060].

The paper links these results to **“Lost in the Middle”**-style positional effects and concludes that some models exhibit a marked **recency bias toward later-presented labels**. It specifically states that prompt label position can “dramatically influence” performance, and that some systems, including **InternLM2-7B-base**, effectively handle only labels positioned near the end of the prompt. This is important because LongICLBench is intended to require full-prompt integration; later-label bias is therefore evidence that many models are not maintaining a uniform usable representation across the entire demonstration sequence [2404.02060].

The grouped-versus-scattered results also indicate a broader failure to **aggregate multiple supporting examples into a stable class representation**. Simply presenting all evidence for a class together does not help and often makes performance much worse. The benchmark interprets this as a failure of long-range reasoning over demonstrations rather than a simple memory-capacity problem. A plausible implication is that current long-context models can often retain local associations, yet still fail to synthesize a coherent many-class decision boundary from a very long prompt.

## 6. Position in the benchmark landscape

LongICLBench occupies a specific niche within long-context evaluation. It differs from **LongBench**-style long-context understanding suites, which emphasize broad task coverage across QA, summarization, code, retrieval, and synthetic tasks. **MiniLongBench** later showed that LongBench can be compressed to **237** samples while preserving model rankings with average Spearman correlation around **0.97**; that work is explicitly a benchmark-compression method rather than a new long-context task formulation, whereas LongICLBench is a purpose-built stress test for demonstration-based long-context ICL [2505.19959].

It also differs from **100-LongBench**, which argues that many de facto long-context benchmarks conflate baseline task strength with genuine long-context robustness and proposes **LongScore** on a **length-controllable** benchmark to separate those factors [2505.19293]. LongICLBench already varies prompt length through demonstration rounds, but it does not decompose results into baseline ability versus normalized long-context retention. This suggests, though does not prove, that a length-controlled or normalized extension of LongICLBench could further sharpen cross-model interpretation.

Relative to **ICLEval**, LongICLBench is more realistic as a long-prompt stress test but less mechanistically isolated. ICLEval focuses on **exact copying** and **rule learning** in mostly synthetic short-to-moderate contexts, with strong contamination controls and exact-match scoring [2406.14955]. By contrast, LongICLBench uses real NLP datasets and long prompts, but it evaluates one specific task family—extreme-label classification—rather than core ICL primitives in isolation.

Relative to **LIFBench**, LongICLBench is demonstration-based rather than instruction-execution-centered. LIFBench evaluates long-context **instruction following** and **stability** across paraphrases, variables, and lengths, but it is explicitly **zero-shot** and does not use few-shot task induction from demonstrations [2411.07037]. Relative to **LifeBench**, LongICLBench is a prompt-bounded ICL benchmark rather than a **long-horizon multi-source memory** benchmark for agents operating over year-scale personal traces [2603.03781].

The closest adjacent benchmark is **MIR-Bench**, which explicitly positions itself as going beyond LongICLBench’s **classification-only** many-shot setting. MIR-Bench evaluates **latent function induction** from up to **2048 shots** and argues that the main failure mode in many-shot reasoning is not pure retrieval but **distraction when aggregating too much information** [2502.09933]. Taken together, the two benchmarks trace a progression: LongICLBench shows that long-context models struggle to infer large label spaces from long demonstrations, while MIR-Bench extends the same concern from classification to general input-output pattern induction.

LongICLBench’s enduring significance lies in its diagnosis of a gap between **nominal context-window size** and **effective prompt-level learning**. Its results suggest that long-context competence cannot be established solely through retrieval tests or fixed-context QA. A model may accept a 32K-, 100K-, or 200K-token input and still fail to learn a difficult task from demonstrations distributed across that input. Within the long-context literature, LongICLBench therefore functions as a benchmark for whether a model can use a long prompt not merely as storage, but as a coherent source of supervision [2404.02060].

Source: https://www.emergentmind.com/topics/longiclbench