Oolong-Real Benchmark

• Oolong-real is a benchmark that assesses long-context reasoning by evaluating LLMs' ability to extract, classify, and aggregate atomic events in unscripted conversational data.
• It utilizes real-world Dungeons & Dragons campaign transcripts, requiring fine-grained event tracking with detailed metadata over episodes spanning up to 175,000 tokens.
• The benchmark employs systematic question templates and aggregation metrics to reveal performance challenges and degradation in LLMs as context length increases.

Oolong-real is a benchmark for evaluating long-context reasoning and aggregation capabilities of LLMs on truly unscripted, multi-episode conversational data. It assesses whether models can reliably extract, classify, and aggregate atomic events embedded in extensive transcripts that span up to 175,000 tokens. Distinct from retrieval-focused long-context benchmarks, Oolong-real requires end-to-end analysis of real-world conversational sessions, emphasizing fine-grained event tracking, cross-temporal reasoning, and robust aggregation under substantial contextual noise (Bertsch et al., 4 Nov 2025).

1. Task Formulation and Objectives

Oolong-real is designed to probe models’ ability to perform detailed extraction and aggregation over extended conversational archives. Core requirements include:

• In-situ classification of each relevant atomic event (e.g., dice roll, spell cast) within the transcript.
• Accurate attribution of event metadata, specifying which actor (character or player) performed the action and the precise episode index.
• Aggregation of these event-level analyses to answer narrowly defined statistical and indexing queries.

These objectives are operationalized through question templates that direct the model to perform tasks such as counting occurrences, computing percentages of event types, enumerating events, and computing cumulative sums across episodes. Unlike traditional benchmarks favoring "needle-in-a-haystack" retrieval, Oolong-real necessitates holistic reasoning over the entire context, requiring models to sift through dialogue, tangents, retcons, and commentary to extract relevant signals.

2. Source Dataset and Annotation Pipeline

The data foundation for Oolong-real is the Critical Role Dungeons & Dragons campaign archive from the CRD3 dataset (Rameshkumar & Bailey, 2020), comprising the first 115 episodes of live-play sessions. Each session is transcribed, preserving speaker tags and out-of-character utterances, resulting in transcripts averaging approximately 55,000 tokens per episode.

Gold standard event annotations stem from the CritRoleStats project, which provides exhaustive logs for dice rolls (including type, natural value, actor) and spell-casting events (name, level, caster). These annotations undergo verification by multiple independent curators to ensure correctness.

To enable tests of aggregation across temporally delimited events, context windows are formed by concatenating single, double, or triple-episode transcripts, with explicit delimiters marking episode boundaries. This structuring supports both intra-episode and cross-episode aggregation and reasoning.

3. Question Templates and Aggregation Operations

Oolong-real utilizes a collection of 29 single-episode and 40 multi-episode question templates, systematically instantiated across all possible context windows, to yield an extensive QA corpus (approximately 8,000 QA pairs).

Templates encompass:

• Counting and Percentages: "How many Crits in this episode?", "What percentage of rolls were natural 20s?"
• Enumeration and Indexing: "What is the second spell cast in episode 2?", "List the rolls made by Percy in the final 10 minutes."
• Cumulative Sums and Cross-Episode Aggregation: "What is the cumulative total of rolls by the end of episode 3?"

Each query $q \in Q$ requires extraction of the relevant event set $\mathcal{E}$ and associated labels $L(e)$ (e.g., roll type, spell name) and metadata (actor, episode index). Processing steps are:

1. Locate and classify each event $e \in \mathcal{E}$ to infer the correct label.
2. Filter $\mathcal{E}$ according to the attributes specified in $q$, such as actor, episode, event type, or value.
3. Aggregate the filtered set $\mathcal{E}'$ using the requested operation:

- Count: $N=|\mathcal{E}'|$ - Percentage: $$p = \frac{|\mathcal{E}'|}{|\mathcal{E}|} \times 100$$ - Order-indexing: Identify the $k$th event in sorted $\mathcal{E}$0 - Cumulative Sum: $\mathcal{E}$1

Example QA instances include:

Question	Ground-Truth Answer
“Total number of rolls in this episode?”	Answer: 137
“What is the second spell cast in episode 2?”	Answer: Disintegrate

All gold answers derive from CritRoleStats verified event logs.

4. Evaluation Protocol and Scoring

Oolong-real employs a rigorous evaluation protocol at three context lengths: 55K, 118K, and 175K tokens. Each model’s output for a given query is parsed and converted into one of three canonical answer types: numeric, single string, or comma-separated list.

Scoring functions are as follows:

• Single-string answers demand exact match:

$\mathcal{E}$2

• Numeric answers permit partial credit:

$\mathcal{E}$3

An error of one yields a score of approximately 0.75, two yields ≈0.56, etc.

• List answers use set-overlap F$\mathcal{E}$4:

$\mathcal{E}$5

Overall model accuracy is the unweighted average of all per-question scores. No elaborate prompt engineering is used beyond the standardized instruction format: "Below is the transcript of Episode X… Answer each question using \boxed{…}."

5. Quantitative Performance and Empirical Findings

Empirical evaluation reveals significant challenges for state-of-the-art LLMs. Table 1 summarizes leaderboard results across the three context lengths:

Model	Avg.	55K	118K	175K
Gemini-2.5-Pro	52.95	60.12	50.81	47.93
GPT-5	47.00	58.74	45.72	36.53
o3	36.71	50.57	33.57	25.99
Claude-Sonnet-4	36.75	50.58	32.98	26.70
GPT-5-mini	34.55	49.86	29.90	23.89
Deepseek-R1	32.00	47.85	27.35	20.81
GPT-5-nano	31.05	43.09	26.82	23.23
o4-mini	27.13	41.69	21.77	17.93
Llama-4-Maverick	2.07	2.48	2.11	1.62

Key observations:

• Gemini-2.5-Pro and GPT-5 achieve the highest mean accuracies (≈53% and 47%, respectively).
• All models show substantial performance degradation as context length increases. For instance, GPT-5’s score drops 22 points (58.74% to 36.53%) from 55K to 175K tokens.
• Open-weight models (e.g., Llama-4, Deepseek-R1) substantially underperform API-scale systems, particularly for extended contexts.
• No evaluated model surpasses 61% accuracy at the short context setting.

These results indicate that even the most advanced current LLMs consistently fall short of robust, multi-step classification and aggregation over densely packed long-context data.

6. Significance and Research Implications

Oolong-real provides a unique, reproducible testbed challenging models to perform atomic event extraction, tracking, and aggregation at scales reflecting real downstream use cases. Its reliance on messy, unscripted data exposes failure modes absent from synthetic or highly structured evaluation settings and highlights persistent deficits in current LLM architectures’ context utilization capabilities.

A plausible implication is that progress on Oolong-real would directly benefit applications requiring timeline reasoning, longitudinal tracking, and robust summarization in conversational or log-based domains. The benchmark’s aggregation-centric design foregrounds the need to move beyond naïve retrieval, incentivizing research into models that integrate entity state tracking, fine-grained sequence modeling, and scalable labeling mechanisms.

By anchoring its evaluation in a well-annotated, human-curated event space and providing granular scoring metrics, Oolong-real enables transparent comparison of architectural, algorithmic, and training strategies aimed at long-context information aggregation (Bertsch et al., 4 Nov 2025).

7. Availability and Reproducibility

The Oolong-real dataset, gold annotation files, and official evaluation harness are made available to facilitate further research and reproducibility. All benchmark questions and contexts are derived from open-source transcripts and annotated event logs, with scoring scripts enforcing standardized answer formatting and comparison. This infrastructure enables efficient, community-wide assessment of emerging model variants against a rigorously controlled aggregation benchmark.

Researchers are thus equipped with a challenging, realistic environment for measuring and improving long-context reasoning and aggregation performance in next-generation LLMs.