Oolong-Synthetic Benchmark

• Oolong-Synthetic is a benchmark that evaluates models by requiring atomic text classification and subsequent aggregation over extremely long contexts.
• It employs controlled tasks for counting, user-based, and temporal subsets, using standard text-classification datasets and explicit token budgeting.
• Empirical results reveal that even state-of-the-art models degrade significantly with increasing context length, highlighting the need for recursive and external computation methods.

Oolong-Synthetic is the synthetic task set of the Oolong benchmark for long-context reasoning and aggregation. It is designed to test whether models can analyze individual chunks of text on an atomic level and then aggregate those analyses into a final answer, rather than relying on retrieval from one or a few salient spans. In the benchmark definition, Oolong is separated into two task sets, Oolong-synth and Oolong-real, and Oolong-synth provides a controlled setting in which counting, user-based, and temporal aggregation can be ablated and measured across contexts ranging from 1 K to 4 M tokens (Bertsch et al., 4 Nov 2025). In later work, the Oolong-Synthetic validation split was used as a controlled evaluation substrate for Recursive Agent Harnesses, with 199 samples stratified into 13 context-length buckets up to 4 M tokens (Lumer et al., 11 Jun 2026).

1. Benchmark definition and problem formulation

Oolong-Synthetic is a benchmark of long-context reasoning tasks that require analyzing individual chunks of text on an atomic level, and then aggregating these analyses to answer distributional questions (Bertsch et al., 4 Nov 2025). This design is explicitly contrasted with long-context evaluations that rely primarily on retrieval from one or more sections of the context, which allows nearly all of the context tokens to be disregarded as noise. The benchmark therefore targets a different failure mode: models may be able to locate isolated evidence, yet still fail when the task requires exhaustive traversal and aggregation over large numbers of examples.

The synthetic setting is constructed so that the underlying per-chunk operation is classification, while the end task is aggregation over many classified examples. In the formal notation used for Oolong-synth, the in-context examples are denoted

$D=\{x_i,y_i,t_i,u_i\}_{i=1}^N,$

where $y_i \in L$. The model performs per-chunk classification,

$\hat y_i=\text{ModelLabel}(x_i),$

and the benchmark then evaluates whether these local decisions are correctly pooled into global answers such as counts, comparisons, user-restricted tallies, or temporal summaries (Bertsch et al., 4 Nov 2025).

A common misconception is that Oolong-Synthetic is primarily a classification benchmark. The published analysis instead identifies the primary bottleneck as identification of relevant chunks plus aggregation, not per-chunk labeling (Bertsch et al., 4 Nov 2025). This is reinforced by the reported ablation in which providing gold labels in context yields only a modest +1 to +10 pt gain, confirming that classification is not the main bottleneck.

2. Synthetic construction and document structure

Oolong-synth uses ten standard text-classification datasets: Spam, TREC-QC, AGNews, App Reviews, Formality, IMDB, HiTZ Negation, Yahoo Topics, MultiNLI, and Metaphors (Bertsch et al., 4 Nov 2025). Each example is augmented with two metadata fields:

• date $\in$ 40-month range
• user $\in \{u_1,\ldots,u_M\}$ (80% of examples come from top 20% of users; heavy-tailed)

These document units, consisting of text, date, and user_id, are then concatenated to form a very long context. The chunk format is specified as:

$\hat y_i=\text{ModelLabel}(x_i),$4

Token budgeting is also defined explicitly. Average chunk length is measured at approximately 50 Llama2 tokens each, plus about 50 tokens for instructions and 100 tokens for the question. To target a context window of size $K$ tokens, the benchmark samples roughly

$\lfloor 0.95 \cdot K / \text{avg\_chunk} \rfloor$

examples (Bertsch et al., 4 Nov 2025).

Contexts range from 1 K up to 4 M tokens, in powers of two. For each $(\text{dataset}, \text{length})$, two independent windows are sampled and 25 distinct questions are issued per window, yielding 400 questions per length (Bertsch et al., 4 Nov 2025). In the separate Recursive Agent Harnesses evaluation, the reported slice comprises 199 samples drawn from the Oolong-Synthetic validation split and stratified into 13 bins by document length: 1 K, 2 K, 4 K, 8 K, 16 K, 32 K, 64 K, 131 K, 262 K, 524 K, 1 M, 2 M, and 4 M tokens, with roughly 14–16 documents per bin (Lumer et al., 11 Jun 2026).

This dual description reflects two uses of the same synthetic benchmark family: a broad benchmark construction in Oolong itself, and a fixed validation evaluation in the Recursive Agent Harnesses study.

3. Question families and aggregation operators

Oolong-Synthetic defines three question families: counting questions, user-subset questions, and temporal-subset (“timeline”) questions (Bertsch et al., 4 Nov 2025). All three require the same atomic operation—classification of each chunk—but differ in how the classified instances are pooled.

For counting questions, the aggregate statistic is

$c(\ell)=\sum_{i=1}^N 1[y_i=\ell].$

The benchmark then asks for derived quantities such as the most common label,

$\arg\max_{\ell \in L} c(\ell),$

the least common label,

$y_i \in L$0

the count of a specific label, or a comparison between two labels. An example prompt is: “In the above data, which label is most common? Format: label: ANSWER” (Bertsch et al., 4 Nov 2025).

For user-subset questions, the aggregation is restricted to users or user subsets. The benchmark defines

$y_i \in L$1

and poses queries such as “which user appears most often,” “which user has most examples of label $y_i \in L$2,” or “compare user A vs. B on label $y_i \in L$3” (Bertsch et al., 4 Nov 2025). These tasks require joint tracking of classification and user identity.

For temporal-subset questions, the aggregation is restricted by dates or calendar bins. The benchmark defines, for example,

$y_i \in L$4

with analogous forms for after a cutoff or per-month bins (Bertsch et al., 4 Nov 2025). Example tasks include determining whether a label was more common before or after a date, finding the first month where one label exceeds another, or counting how many months a label was the most frequent.

The question-family structure matters because the benchmark reports that timeline tasks cluster at the lowest scores across all models at 64 K context, user tasks are intermediate, and counting tasks are easiest (Bertsch et al., 4 Nov 2025). This suggests that temporal partitioning compounds the core aggregation difficulty with date comparison and binning.

4. Scoring, evaluation protocol, and answer types

The Oolong evaluation uses exact match for discrete answer types and a graded reward for numerical answers. In the Oolong-synth formal specification, discrete answers include label, user_id, date, and comparison outputs, all scored by exact match (Bertsch et al., 4 Nov 2025). Numerical answers, such as counts or percentages, are scored by

$y_i \in L$5

which gives partial credit for near misses (Bertsch et al., 4 Nov 2025).

The Recursive Agent Harnesses paper reports the Oolong-Synthetic validation evaluation in nearly the same terms, but at the level of per-instance scoring over 199 instances. For categorical answer types—LABEL, COMPARISON, USER, DATE—the score is

$y_i \in L$6

For NUMERIC answers,

$y_i \in L$7

The overall micro-averaged Oolong Score is then

$y_i \in L$8

That evaluation also reports 95 % bootstrap confidence intervals computed over 10 000 resamples of the per-instance scores (Lumer et al., 11 Jun 2026).

The benchmark additionally defines an analytic random-guess baseline: uniform random among valid answers for $y_i \in L$9-way discrete choice, $\hat y_i=\text{ModelLabel}(x_i),$0 for counts, and uniform random among those in context for user_id or date (Bertsch et al., 4 Nov 2025). The expected random-baseline score is computed analytically over all examples. This baseline is useful because Oolong-Synthetic mixes answer formats with different cardinalities and error profiles.

5. Empirical performance and scaling behavior

The main Oolong paper reports that even frontier models struggle on Oolong, with GPT-5, Claude-Sonnet-4, and Gemini-2.5-Pro all achieving less than 50% accuracy on both splits at 128K (Bertsch et al., 4 Nov 2025). On the combined Oolong-synth and Oolong-real evaluation, average synth scores and context-length performance are reported for multiple model families, and performance drops off sharply after approximately 32 K tokens; by 128 K no model exceeds 50 % on Oolong-synth (Bertsch et al., 4 Nov 2025).

A concise summary of the reported combined results is given below.

Model	Avg Synth	128K
GPT-5	70.8 %	46.4
Gemini-2.5-Pro	55.3 %	25.1
OpenAI o3	62.4 %	37.5
GPT-5-mini	63.7 %	40.9
Claude-Sonnet-4	58.2 %	48.0

At 64 K context, GPT-5 is reported at about 68 % on label outputs, about 63 % on comparisons, about 50 % on users/dates, and about 42 % on numeric percent answers (Bertsch et al., 4 Nov 2025). This answer-type decomposition is consistent with the benchmark’s design: discrete label decisions remain comparatively tractable, while user/date tracking and approximate numeric aggregation are weaker.

The same paper reports that reasoning effort (“low” vs. “high” chain-of-thought) helps only up to 16 K, with no benefit beyond 64 K (Bertsch et al., 4 Nov 2025). It also reports that the short-context variant from 1 K to 8 K is easier, but even at 8 K top models plateau at about 83 % accuracy. These observations indicate that merely extending context length or eliciting longer reasoning traces does not solve the aggregation problem.

6. Oolong-Synthetic in Recursive Agent Harnesses

In “Recursive Agent Harnesses,” Oolong-Synthetic serves as a controlled long-context reasoning evaluation for comparing full-context inference, recursive LLMs, coding agents, and recursive harness-based agents under a fixed backbone (Lumer et al., 11 Jun 2026). The study evaluates 199 samples from the Oolong-Synthetic validation split and emphasizes that each sample is a single document containing many labeled key–value entries, such as user IDs, labels, numeric fields, dates, and comparisons. A question asks the agent to aggregate or extract the correct value for one of these entries, requiring long-context navigation and per-entry reasoning.

The baseline results with a GPT-5 backbone are:

System	Oolong Score
Full-context	59.22 %
Recursive LLM (RLM)	64.38 %
Codex coding agent (no retriever)	71.75 %
RAH with GPT-5 backbone	81.36 %
RAH with Claude Sonnet 4.5 backbone	89.77 %

For the GPT-5-based RAH, the reported 95 % confidence interval is 76.0 %, 86.5 %. The paper also reports a controlled comparison in which the 71.75 % Codex baseline and the 81.36 % RAH result both use the identical GPT-5 checkpoint, gpt-5-2025-08-07. The quantitative lift is given as

$\hat y_i=\text{ModelLabel}(x_i),$1

with 95 % CI $\hat y_i=\text{ModelLabel}(x_i),$2 (Lumer et al., 11 Jun 2026). Relative to the RLM baseline, the improvement is +16.98 % with 95 % CI $\hat y_i=\text{ModelLabel}(x_i),$3.

The per-context-length breakdown shows that the recursive harness design remains competitive throughout the full range from 1 K to 4 M tokens (Lumer et al., 11 Jun 2026).

Context	GPT-5	Sonnet 4.5
1 K	100.0 %	93.8 %
2 K	93.8 %	90.6 %
4 K	94.1 %	94.1 %
8 K	94.5 %	96.9 %
16 K	93.0 %	90.6 %
32 K	86.7 %	90.4 %
64 K	92.3 %	100.0 %
131 K	73.3 %	87.6 %
262 K	57.1 %	92.0 %
524 K	80.0 %	86.7 %
1 M	53.3 %	80.0 %
2 M	66.7 %	86.7 %
4 M	66.7 %	76.7 %

The study interprets these gains as harness-driven rather than model-driven because the backbone is held fixed in the main comparison. It further argues that per-entry subagent reasoning, powered by filesystem, code, and parallelism, closes gaps that neither a single regex-based loop nor pure model recursion can (Lumer et al., 11 Jun 2026). A plausible implication is that Oolong-Synthetic is particularly sensitive to externalized intermediate computation, not only to raw language-model quality.

7. Limitations, failure modes, and research significance

The Oolong work identifies several failure modes in current long-context systems. The primary bottleneck is identification of relevant chunks plus aggregation, not per-chunk labeling, and temporal reasoning is particularly weak (Bertsch et al., 4 Nov 2025). Pathologies observed in model traces include running out of output budget mid-chain-of-thought, self-debates about tractability on very long inputs, and refusals when context size seems “too big.” These are not benchmark-specific quirks; they reflect broader issues in end-to-end reasoning over large quantities of text.

The benchmark also includes concrete recommendations for stronger long-context aggregation: hierarchical chunking, retrieval-augmented in-context processing, iterative labeling with external aggregation, specialized date-comparison modules or explicit temporal-logic prompts, and dynamic prompt routing (Bertsch et al., 4 Nov 2025). These recommendations are method classes rather than a single prescribed architecture. The later Recursive Agent Harnesses results provide one concrete instantiation of the broader recommendation to externalize intermediate computation and combine local analyses through a structured controller (Lumer et al., 11 Jun 2026).

Within the long-context literature, Oolong-Synthetic is significant because it operationalizes a regime in which nearly all context tokens may matter. It therefore complements retrieval-centric benchmarks by stressing exhaustive or near-exhaustive aggregation. This suggests that strong performance on document retrieval or short-span extraction should not be conflated with robust long-context reasoning. In the reported evidence, models that remain strong at moderate context still degrade sharply as the benchmark shifts from localized lookup to distributed counting, user-relation tracking, and timeline reasoning (Bertsch et al., 4 Nov 2025).

Oolong-Synthetic thus functions both as a benchmark and as a diagnostic instrument. In its original formulation, it isolates controlled aggregation challenges over synthetic but naturalistic documents. In subsequent use, it serves as a stringent evaluation of agent architectures that attempt to turn long-context reasoning into a recursive systems problem rather than a single forward pass (Lumer et al., 11 Jun 2026).