---
title: 'O-Bench: Overloaded Evaluation Benchmarks'
url: https://www.emergentmind.com/topics/o-bench
type: topic
---

# O-Bench: Overloaded Evaluation Benchmarks

“O-Bench” is not a single standardized benchmark name in the supplied literature. Instead, it appears as a field-dependent shorthand attached to multiple distinct evaluation suites. In contemporary LLM work, the label is used for **OR-Bench**, a benchmark for **over-refusal** in safety-aligned models [2405.20947], for **OIBench**, a private olympiad-level informatics benchmark for algorithmic reasoning [2506.10481], and for **OpaqueToolsBench**, a benchmark for learning the behavior of **opaque tools** through interaction [2602.15197]. By contrast, the earlier survey “Object Database Benchmarks” explicitly states that it does **not** mention any benchmark called “O-Bench” [1701.07739]. The term therefore denotes an overloaded naming convention rather than a canonical benchmark family.

## 1. Terminological status

Within the supplied sources, “O-Bench” functions as an informal or contextual abbreviation rather than a uniquely identified artifact. The OR-Bench material states that the query “O-Bench” refers to **OR-Bench**, while also noting that there is no separate benchmark called “O-Bench” in that paper [2405.20947]. The OIBench material states that OIBench is “sometimes abbreviated ‘O-Bench’ in discussion” [2506.10481]. The OpaqueToolsBench paper explicitly titles its benchmark overview “OpaqueToolsBench (O-Bench)” [2602.15197]. In contrast, the object-database survey states that it does **not** mention a benchmark called “O-Bench” and treats the phrase only as a possible generic label for “an object database benchmark” [1701.07739].

| Usage of “O-Bench” | Domain | Core evaluation target |
|---|---|---|
| OR-Bench | LLM safety alignment | Over-refusal vs toxic rejection |
| OIBench | Algorithmic code reasoning | Olympiad-level correctness and efficiency |
| OpaqueToolsBench | Tool-using LLM agents | Learning opaque tool behavior through interaction |

This terminological dispersion is consequential for interpretation. Any reference to “O-Bench” requires immediate disambiguation by domain, because the underlying task, metric, and benchmark construction differ substantially across these usages.

## 2. OR-Bench: over-refusal in safety-aligned language models

OR-Bench is a benchmark designed to quantify **over-refusal**, defined as the case where an LLM **refuses benign prompts** that appear risky at a surface level but are actually safe and socially acceptable [2405.20947]. Its central object of measurement is the trade-off between safety and helpfulness: aligned models should reject harmful prompts, but should not over-refuse benign prompts that merely look unsafe.

The benchmark contains three components. **OR-Bench-80K** comprises approximately **80,000 safe, seemingly toxic prompts** across **10 common rejection categories**. **OR-Bench-Hard-1K** is a subset of approximately **1,000 hard prompts** that remain challenging for strong aligned models. **OR-Bench-Toxic** contributes approximately **600 genuinely toxic prompts** used to measure whether a model properly rejects harmful content [2405.20947]. The ten categories are **deception, harassment, harmful, hate, illegal, privacy, self-harm, sexual, unethical,** and **violence**.

Its formal evaluation centers on two rates. If $\mathcal{S}$ is the safe prompt set, $\mathcal{T}$ the toxic prompt set, and $R_M(x)$ indicates whether model $M$ refuses prompt $x$, then the **over-refusal rate** is
$$
\text{ORR}(M) = \frac{1}{|\mathcal{S}|} \sum_{x \in \mathcal{S}} R_M(x),
$$
while the **toxic rejection rate** is
$$
\text{TRR}(M) = \frac{1}{|\mathcal{T}|} \sum_{x \in \mathcal{T}} R_M(x).
$$
The benchmark evaluates models in the $(\text{ORR}, \text{TRR})$ plane, where the ideal region is described as **top-left**: low rejection of safe prompts and high rejection of toxic prompts [2405.20947].

Methodologically, OR-Bench is built around a fully automatic prompt-generation pipeline. Toxic seeds are generated with **Mixtral 8×7B**, rewritten into “seemingly toxic” but benign prompts, then filtered with a **3-way LLM ensemble** consisting of **GPT-4-turbo-2024-04-09**, **Llama-3-70B**, and **Gemini-1.5-pro**. A second pass uses **Mistral-7B-Instruct-v0.3** to reduce false positives from over-sensitive moderation [2405.20947]. The paper reports a **Spearman rank correlation** of
$$
\rho_{\text{Spearman}}(\text{ORR}, \text{TRR}) = 0.878,
$$
quantifying a strong positive association between safety on toxic prompts and over-refusal on safe but subtle prompts.

As a benchmark interpretation, OR-Bench is notable because it treats refusal behavior itself as the measured phenomenon rather than as a hidden side effect of alignment. This makes it useful for comparing alignment policies that would otherwise appear strong under safety-only evaluation.

## 3. OIBench: olympiad-level informatics for strong reasoning models

OIBench is a **bilingual (Chinese/English), olympiad-level informatics benchmark of 250 algorithmic programming problems**, each accompanied by **large, rigorous test data** and **canonical C++ reference solutions** [2506.10481]. It is described as **private** because the problems were taken from coaches’ private repositories or newly authored, and verified to be unpublished before release. Its stated aims are to evaluate **algorithmic reasoning and competitive-programming–style coding**, to be **contamination-resistant**, to expose **time/space efficiency** differences, and to enable **direct human–model comparison**.

Each problem includes a full statement, I/O format, data range and constraints, at least one sample, full test data averaging **37.5M bytes per problem**, canonical C++ solutions averaging **75.2 lines**, and metadata such as difficulty and algorithm tags [2506.10481]. Difficulty is both coach-labeled and empirically calibrated. In the comparison table reported in the paper, **GPT-4O** achieves **2.6%** on OIBench, lower than on **HumanEval (87.2%)**, **CodeContests (34.7%)**, **USACO (8.3%)**, **CodeElo (16.8%)**, **LiveCodeBench (41.9%)**, and **EffiBench (50.8%)** [2506.10481]. OIBench is accordingly labeled with four stars of difficulty in that summary.

Its core metric is **AC (All Correct) rate**: a problem is solved only if the generated program passes **all** hidden test cases [2506.10481]. OIBench further introduces **Time Completion Curves** and **Space Completion Curves**, defined over per-test-case ratios relative to the canonical solution:
$$
r_i^{\text{time}} = \frac{T_i^{\text{model}}}{T_i^{\text{canon}}}, \qquad
r_i^{\text{space}} = \frac{M_i^{\text{model}}}{M_i^{\text{canon}}}.
$$
The corresponding completion curves aggregate the fraction of passed test cases within a threshold $x$:
$$
C^{\text{time}}(x) = \frac{1}{N} \sum_{i=1}^N I_i^{\text{time}}(x), \qquad
C^{\text{space}}(x) = \frac{1}{N} \sum_{i=1}^N I_i^{\text{space}}(x).
$$
The paper characterizes these as empirical cumulative distributions of efficiency-constrained success [2506.10481].

The benchmark reports results for **18 models** spanning base models, instruction-tuned models, and explicit reasoning models. On the core OIBench leaderboard, **O4-mini-high** attains **36.35%** overall AC, ahead of **O3-mini-high (26.80%)**, **DeepSeek-R1 (20.50%)**, and **Qwen3-32B (18.95%)** [2506.10481]. The paper further states that **SOTA reasoning models outperform most human participants in both correctness and efficiency**, while still remaining **suboptimal compared to the canonical solutions**.

OIBench therefore uses “O-Bench” to denote a benchmark whose primary pressure point is not tool usage or alignment policy, but unsaturated algorithmic reasoning under strict resource constraints.

## 4. OpaqueToolsBench: learning the behavior of opaque tools

OpaqueToolsBench, explicitly abbreviated **O-Bench**, is a benchmark for studying whether LLM agents can improve performance in environments with **underspecified tools** by interacting with them and refining documentation [2602.15197]. The paper distinguishes two kinds of opacity: **Type 1 – Documentation Opacity**, where the function is describable but poorly documented, and **Type 2 – Intrinsic Opacity**, where the tool’s behavior is complex or hard to describe even for its creators.

The benchmark contains **three environments**. **BFCL-Opaque** adapts function-calling tasks by obfuscating tool identities and degrading documentation. **Chess** requires the agent to choose among move-suggestion tools with identical interfaces but different hidden behavior. **BrowseComp Domains** frames long-trajectory question answering through opaque domain-specific search tools [2602.15197]. These environments are explicitly associated with different abilities: structured or unstructured inputs, process feedback, cross-trajectory learning, test-time generalization, and tool sequencing.

BFCL-Opaque defines three documentation regimes: **Anon. Fn. Names Only**, **Anon. Fn. Names + Real Desc**, and **Anon. Fn. Names + Parameter Names** [2602.15197]. Its metrics are **Execution Accuracy**, **Parameter Accuracy**, and **AST Accuracy**. Chess is evaluated with **Best Tool Accuracy** and **Streaming Elo**. BrowseComp Domains is evaluated with **accuracy** and **number of tool calls**. These are not interchangeable targets: one environment measures structural function-call fidelity, another practical gameplay strength, and another long-horizon search behavior.

The associated method, **ToolObserver**, alternates between **exploration** and **reflection**. In offline mode, the algorithm initializes documentation $\mathcal{D}^{(0)}$, collects trajectories with an agent model, and updates documentation through an editor model over $K$ iterations. In online mode, used for instance-specific unseen tools, refinement occurs per test instance and uses only execution feedback rather than gold answers [2602.15197]. This framework is contrasted with **EasyTool** and **Play2Prompt**, both of which are reported to be less effective when tools are opaque.

The benchmark’s reported results emphasize both effectiveness and efficiency. In BFCL-Opaque with **Anon. Fn. Names Only**, **ToolObserver** raises **GPT-5** from **E = 0.00** under the Base setting to **E = 0.80**, with corresponding **P = 0.78** and **A = 0.89** [2602.15197]. In Chess, ToolObserver improves over Base in both **Best Tool Accuracy** and **Streaming Elo**. In BrowseComp, it generally exceeds or matches the baselines, especially in harder settings. The paper further states that, for **test-time tool exploration**, ToolObserver consumes **3.5–7.5× fewer total tokens** than the best baseline [2602.15197].

In this usage, “O-Bench” designates a benchmark for agentic adaptation under partial observability of tool semantics, rather than for model alignment or algorithmic code synthesis.

## 5. Relation to earlier benchmark methodology

The historical survey “Object Database Benchmarks” provides a useful baseline for interpreting the later uses of “O-Bench,” even though it explicitly states that no benchmark by that name appears in the paper [1701.07739]. In that survey, a database benchmark consists of two main parts: **a workload model**—“a database and a set of read and write operations to apply on this database”—and **performance metrics**, defined as “simple or composite metrics aimed at expressing the performance of a system” [1701.07739].

The survey further emphasizes Gray’s four criteria for a good benchmark: **relevance, portability, simplicity,** and **scalability** [1701.07739]. It compares object-database benchmarks such as **OO1**, **HyperModel**, **OO7**, and **OCB**, and highlights the persistent conflict between realism and implementability. The paper also distinguishes **generic benchmarks** from **specific benchmarks**, noting that more generic and parameterized benchmarks can emulate wider workloads but are harder to configure meaningfully [1701.07739].

This suggests a continuity at the level of benchmark design principles, even though the application domains are entirely different. OR-Bench, OIBench, and OpaqueToolsBench each instantiate the same broad pattern: a deliberately constructed workload or task environment, paired with explicit metrics that operationalize a particular failure mode or capability frontier. A plausible implication is that the reuse of the “O-Bench” label reflects naming convenience, whereas the deeper continuity lies in benchmark methodology rather than in benchmark identity.

## 6. Comparative interpretation

Taken together, the contemporary “O-Bench” usages span three different evaluation objects. OR-Bench measures **policy calibration** in aligned LLMs, specifically the balance between refusing harmful prompts and answering safe but risky-looking ones [2405.20947]. OIBench measures **algorithmic reasoning and implementation efficiency** under olympiad-style constraints [2506.10481]. OpaqueToolsBench measures **tool-behavior acquisition** when documentation is degraded or the tool is intrinsically opaque [2602.15197].

They also differ sharply in what counts as a successful model. In OR-Bench, the central outcome is a low **ORR** together with a high **TRR**. In OIBench, success requires **all-correct** execution under hidden tests, supplemented by favorable **Time/Space Completion Curves**. In OpaqueToolsBench, success is environment-specific: structurally correct function calls in BFCL-Opaque, stronger tool-selection policy in Chess, or more accurate and efficient search trajectories in BrowseComp [2405.20947] [2506.10481] [2602.15197].

A plausible implication is that “O-Bench” has become a convenient label for **frontier stress tests**, but not for a unified benchmarking standard. The supplied literature instead supports a more precise conclusion: references to “O-Bench” are only meaningful when anchored to the full benchmark name and research domain. Without that disambiguation, the term can refer to over-refusal analysis, olympiad-level code reasoning, opaque tool learning, or, historically, to no named benchmark at all [1701.07739].

Source: https://www.emergentmind.com/topics/o-bench