---
title: Oolong-Real Benchmark
url: https://www.emergentmind.com/topics/oolong-real-a3b59f73-9d43-46fa-92d7-b89bf7da5bf0
type: topic
---

# Oolong-Real Benchmark

Oolong-real is a benchmark for evaluating long-context reasoning and aggregation capabilities of large language models (LLMs) on truly unscripted, multi-episode conversational data. It assesses whether models can reliably extract, classify, and aggregate atomic events embedded in extensive transcripts that span up to 175,000 tokens. Distinct from retrieval-focused long-context benchmarks, Oolong-real requires end-to-end analysis of real-world conversational sessions, emphasizing fine-grained event tracking, cross-temporal reasoning, and robust aggregation under substantial contextual noise [2511.02817].

## 1. Task Formulation and Objectives

Oolong-real is designed to probe models’ ability to perform detailed extraction and aggregation over extended conversational archives. Core requirements include:

- In-situ classification of each relevant atomic event (e.g., dice roll, spell cast) within the transcript.
- Accurate attribution of event metadata, specifying which actor (character or player) performed the action and the precise episode index.
- Aggregation of these event-level analyses to answer narrowly defined statistical and indexing queries.

These objectives are operationalized through question templates that direct the model to perform tasks such as counting occurrences, computing percentages of event types, enumerating events, and computing cumulative sums across episodes. Unlike traditional benchmarks favoring "needle-in-a-haystack" retrieval, Oolong-real necessitates holistic reasoning over the entire context, requiring models to sift through dialogue, tangents, retcons, and commentary to extract relevant signals.

## 2. Source Dataset and Annotation Pipeline

The data foundation for Oolong-real is the Critical Role Dungeons & Dragons campaign archive from the CRD3 dataset (Rameshkumar & Bailey, 2020), comprising the first 115 episodes of live-play sessions. Each session is transcribed, preserving speaker tags and out-of-character utterances, resulting in transcripts averaging approximately 55,000 tokens per episode.

Gold standard event annotations stem from the CritRoleStats project, which provides exhaustive logs for dice rolls (including type, natural value, actor) and spell-casting events (name, level, caster). These annotations undergo verification by multiple independent curators to ensure correctness.

To enable tests of aggregation across temporally delimited events, context windows are formed by concatenating single, double, or triple-episode transcripts, with explicit delimiters marking episode boundaries. This structuring supports both intra-episode and cross-episode aggregation and reasoning.

## 3. Question Templates and Aggregation Operations

Oolong-real utilizes a collection of 29 single-episode and 40 multi-episode question templates, systematically instantiated across all possible context windows, to yield an extensive QA corpus (approximately 8,000 QA pairs).

Templates encompass:

- **Counting and Percentages**: "How many Crits in this episode?", "What percentage of rolls were natural 20s?"
- **Enumeration and Indexing**: "What is the second spell cast in episode 2?", "List the rolls made by Percy in the final 10 minutes."
- **Cumulative Sums and Cross-Episode Aggregation**: "What is the cumulative total of rolls by the end of episode 3?"

Each query $q \in Q$ requires extraction of the relevant event set $\mathcal{E}$ and associated labels $L(e)$ (e.g., roll type, spell name) and metadata (actor, episode index). Processing steps are:

1. Locate and classify each event $e \in \mathcal{E}$ to infer the correct label.
2. Filter $\mathcal{E}$ according to the attributes specified in $q$, such as actor, episode, event type, or value.
3. Aggregate the filtered set $\mathcal{E}'$ using the requested operation:

    - **Count**: $N=|\mathcal{E}'|$
    - **Percentage**: $p = \frac{|\mathcal{E}'|}{|\mathcal{E}|} \times 100$
    - **Order-indexing**: Identify the $k$th event in sorted $\mathcal{E}'$
    - **Cumulative Sum**: $\sum_{i=1}^{m} |\{e: \mathrm{episode}(e) \leq i \}|$

Example QA instances include:

| Question                                         | Ground-Truth Answer |
|--------------------------------------------------|---------------------|
| “Total number of rolls in this episode?”         | Answer: 137         |
| “What is the second spell cast in episode 2?”    | Answer: Disintegrate|

All gold answers derive from CritRoleStats verified event logs.

## 4. Evaluation Protocol and Scoring

Oolong-real employs a rigorous evaluation protocol at three context lengths: 55K, 118K, and 175K tokens. Each model’s output for a given query is parsed and converted into one of three canonical answer types: numeric, single string, or comma-separated list.

Scoring functions are as follows:

- **Single-string answers** demand exact match:
  $$
  \mathrm{Score}_{\mathrm{str}}(\hat{y}) = \begin{cases}
    1 & \hat{y} = y \\
    0 & \text{otherwise}
  \end{cases}
  $$
- **Numeric answers** permit partial credit:
  $$
  \mathrm{Score}_{\mathrm{num}}(\hat{y}) = 0.75^{|y - \hat{y}|}
  $$
  An error of one yields a score of approximately 0.75, two yields ≈0.56, etc.
- **List answers** use set-overlap F$_1$:
  $$
  \mathrm{Score}_{\mathrm{list}} = \frac{2|P \cap G|}{|P| + |G|}
  $$

Overall model accuracy is the unweighted average of all per-question scores. No elaborate prompt engineering is used beyond the standardized instruction format: "Below is the transcript of Episode X… Answer each question using \boxed{…}."

## 5. Quantitative Performance and Empirical Findings

Empirical evaluation reveals significant challenges for state-of-the-art LLMs. Table 1 summarizes leaderboard results across the three context lengths:

| Model                | Avg. | 55K   | 118K  | 175K  |
|----------------------|------|-------|-------|-------|
| Gemini-2.5-Pro       | 52.95| 60.12 | 50.81 | 47.93 |
| GPT-5                | 47.00| 58.74 | 45.72 | 36.53 |
| o3                   |36.71 | 50.57 |33.57  | 25.99 |
| Claude-Sonnet-4      |36.75 | 50.58 |32.98  | 26.70 |
| GPT-5-mini           |34.55 | 49.86 |29.90  | 23.89 |
| Deepseek-R1          |32.00 | 47.85 |27.35  | 20.81 |
| GPT-5-nano           |31.05 | 43.09 |26.82  | 23.23 |
| o4-mini              |27.13 | 41.69 |21.77  | 17.93 |
| Llama-4-Maverick     | 2.07 |  2.48 | 2.11  |  1.62 |

Key observations:

- Gemini-2.5-Pro and GPT-5 achieve the highest mean accuracies (≈53% and 47%, respectively).
- All models show substantial performance degradation as context length increases. For instance, GPT-5’s score drops 22 points (58.74% to 36.53%) from 55K to 175K tokens.
- Open-weight models (e.g., Llama-4, Deepseek-R1) substantially underperform API-scale systems, particularly for extended contexts.
- No evaluated model surpasses 61% accuracy at the short context setting.

These results indicate that even the most advanced current LLMs consistently fall short of robust, multi-step classification and aggregation over densely packed long-context data.

## 6. Significance and Research Implications

Oolong-real provides a unique, reproducible testbed challenging models to perform atomic event extraction, tracking, and aggregation at scales reflecting real downstream use cases. Its reliance on messy, unscripted data exposes failure modes absent from synthetic or highly structured evaluation settings and highlights persistent deficits in current LLM architectures’ context utilization capabilities.

A plausible implication is that progress on Oolong-real would directly benefit applications requiring timeline reasoning, longitudinal tracking, and robust summarization in conversational or log-based domains. The benchmark’s aggregation-centric design foregrounds the need to move beyond naïve retrieval, incentivizing research into models that integrate entity state tracking, fine-grained sequence modeling, and scalable labeling mechanisms.

By anchoring its evaluation in a well-annotated, human-curated event space and providing granular scoring metrics, Oolong-real enables transparent comparison of architectural, algorithmic, and training strategies aimed at long-context information aggregation [2511.02817].

## 7. Availability and Reproducibility

The Oolong-real dataset, gold annotation files, and official evaluation harness are made available to facilitate further research and reproducibility. All benchmark questions and contexts are derived from open-source transcripts and annotated event logs, with scoring scripts enforcing standardized answer formatting and comparison. This infrastructure enables efficient, community-wide assessment of emerging model variants against a rigorously controlled aggregation benchmark.

Researchers are thus equipped with a challenging, realistic environment for measuring and improving long-context reasoning and aggregation performance in next-generation LLMs.

Source: https://www.emergentmind.com/topics/oolong-real-a3b59f73-9d43-46fa-92d7-b89bf7da5bf0