O-Bench: Overloaded Evaluation Benchmarks

• O-Bench is a non-standardized shorthand used for various benchmarks, each targeting different tasks such as LLM safety, algorithmic reasoning, and tool interaction.
• OR-Bench evaluates over-refusal in safety-aligned models, while OIBench and OpaqueToolsBench focus on rigorous algorithmic performance and opaque tool learning respectively.
• Each variant employs specialized metrics—like over-refusal rates, all-correct rates, and execution accuracies—to measure domain-specific model performance.

“O-Bench” is not a single standardized benchmark name in the supplied literature. Instead, it appears as a field-dependent shorthand attached to multiple distinct evaluation suites. In contemporary LLM work, the label is used for OR-Bench, a benchmark for over-refusal in safety-aligned models (Cui et al., 2024), for OIBench, a private olympiad-level informatics benchmark for algorithmic reasoning (Zhu et al., 12 Jun 2025), and for OpaqueToolsBench, a benchmark for learning the behavior of opaque tools through interaction (Hallinan et al., 16 Feb 2026). By contrast, the earlier survey “Object Database Benchmarks” explicitly states that it does not mention any benchmark called “O-Bench” (Darmont, 2017). The term therefore denotes an overloaded naming convention rather than a canonical benchmark family.

1. Terminological status

Within the supplied sources, “O-Bench” functions as an informal or contextual abbreviation rather than a uniquely identified artifact. The OR-Bench material states that the query “O-Bench” refers to OR-Bench, while also noting that there is no separate benchmark called “O-Bench” in that paper (Cui et al., 2024). The OIBench material states that OIBench is “sometimes abbreviated ‘O-Bench’ in discussion” (Zhu et al., 12 Jun 2025). The OpaqueToolsBench paper explicitly titles its benchmark overview “OpaqueToolsBench (O-Bench)” (Hallinan et al., 16 Feb 2026). In contrast, the object-database survey states that it does not mention a benchmark called “O-Bench” and treats the phrase only as a possible generic label for “an object database benchmark” (Darmont, 2017).

Usage of “O-Bench”	Domain	Core evaluation target
OR-Bench	LLM safety alignment	Over-refusal vs toxic rejection
OIBench	Algorithmic code reasoning	Olympiad-level correctness and efficiency
OpaqueToolsBench	Tool-using LLM agents	Learning opaque tool behavior through interaction

This terminological dispersion is consequential for interpretation. Any reference to “O-Bench” requires immediate disambiguation by domain, because the underlying task, metric, and benchmark construction differ substantially across these usages.

2. OR-Bench: over-refusal in safety-aligned LLMs

OR-Bench is a benchmark designed to quantify over-refusal, defined as the case where an LLM refuses benign prompts that appear risky at a surface level but are actually safe and socially acceptable (Cui et al., 2024). Its central object of measurement is the trade-off between safety and helpfulness: aligned models should reject harmful prompts, but should not over-refuse benign prompts that merely look unsafe.

The benchmark contains three components. OR-Bench-80K comprises approximately 80,000 safe, seemingly toxic prompts across 10 common rejection categories. OR-Bench-Hard-1K is a subset of approximately 1,000 hard prompts that remain challenging for strong aligned models. OR-Bench-Toxic contributes approximately 600 genuinely toxic prompts used to measure whether a model properly rejects harmful content (Cui et al., 2024). The ten categories are deception, harassment, harmful, hate, illegal, privacy, self-harm, sexual, unethical, and violence.

Its formal evaluation centers on two rates. If $\mathcal{S}$ is the safe prompt set, $\mathcal{T}$ the toxic prompt set, and $R_M(x)$ indicates whether model $M$ refuses prompt $x$, then the over-refusal rate is

$$\text{ORR}(M) = \frac{1}{|\mathcal{S}|} \sum_{x \in \mathcal{S}} R_M(x),$$

while the toxic rejection rate is

$$\text{TRR}(M) = \frac{1}{|\mathcal{T}|} \sum_{x \in \mathcal{T}} R_M(x).$$

The benchmark evaluates models in the $(\text{ORR}, \text{TRR})$ plane, where the ideal region is described as top-left: low rejection of safe prompts and high rejection of toxic prompts (Cui et al., 2024).

Methodologically, OR-Bench is built around a fully automatic prompt-generation pipeline. Toxic seeds are generated with Mixtral 8×7B, rewritten into “seemingly toxic” but benign prompts, then filtered with a 3-way LLM ensemble consisting of GPT-4-turbo-2024-04-09, Llama-3-70B, and Gemini-1.5-pro. A second pass uses Mistral-7B-Instruct-v0.3 to reduce false positives from over-sensitive moderation (Cui et al., 2024). The paper reports a Spearman rank correlation of

$$\rho_{\text{Spearman}}(\text{ORR}, \text{TRR}) = 0.878,$$

quantifying a strong positive association between safety on toxic prompts and over-refusal on safe but subtle prompts.

As a benchmark interpretation, OR-Bench is notable because it treats refusal behavior itself as the measured phenomenon rather than as a hidden side effect of alignment. This makes it useful for comparing alignment policies that would otherwise appear strong under safety-only evaluation.

3. OIBench: olympiad-level informatics for strong reasoning models

OIBench is a bilingual (Chinese/English), olympiad-level informatics benchmark of 250 algorithmic programming problems, each accompanied by large, rigorous test data and canonical C++ reference solutions (Zhu et al., 12 Jun 2025). It is described as private because the problems were taken from coaches’ private repositories or newly authored, and verified to be unpublished before release. Its stated aims are to evaluate algorithmic reasoning and competitive-programming–style coding, to be contamination-resistant, to expose time/space efficiency differences, and to enable direct human–model comparison.

Each problem includes a full statement, I/O format, data range and constraints, at least one sample, full test data averaging 37.5M bytes per problem, canonical C++ solutions averaging 75.2 lines, and metadata such as difficulty and algorithm tags (Zhu et al., 12 Jun 2025). Difficulty is both coach-labeled and empirically calibrated. In the comparison table reported in the paper, GPT-4O achieves 2.6% on OIBench, lower than on HumanEval (87.2%), CodeContests (34.7%), USACO (8.3%), CodeElo (16.8%), LiveCodeBench (41.9%), and EffiBench (50.8%) (Zhu et al., 12 Jun 2025). OIBench is accordingly labeled with four stars of difficulty in that summary.

Its core metric is AC (All Correct) rate: a problem is solved only if the generated program passes all hidden test cases (Zhu et al., 12 Jun 2025). OIBench further introduces Time Completion Curves and Space Completion Curves, defined over per-test-case ratios relative to the canonical solution:

$$r_i^{\text{time}} = \frac{T_i^{\text{model}}}{T_i^{\text{canon}}}, \qquad r_i^{\text{space}} = \frac{M_i^{\text{model}}}{M_i^{\text{canon}}}.$$

The corresponding completion curves aggregate the fraction of passed test cases within a threshold $\mathcal{T}$0:

$\mathcal{T}$1

The paper characterizes these as empirical cumulative distributions of efficiency-constrained success (Zhu et al., 12 Jun 2025).

The benchmark reports results for 18 models spanning base models, instruction-tuned models, and explicit reasoning models. On the core OIBench leaderboard, O4-mini-high attains 36.35% overall AC, ahead of O3-mini-high (26.80%), DeepSeek-R1 (20.50%), and Qwen3-32B (18.95%) (Zhu et al., 12 Jun 2025). The paper further states that SOTA reasoning models outperform most human participants in both correctness and efficiency, while still remaining suboptimal compared to the canonical solutions.

OIBench therefore uses “O-Bench” to denote a benchmark whose primary pressure point is not tool usage or alignment policy, but unsaturated algorithmic reasoning under strict resource constraints.

4. OpaqueToolsBench: learning the behavior of opaque tools

OpaqueToolsBench, explicitly abbreviated O-Bench, is a benchmark for studying whether LLM agents can improve performance in environments with underspecified tools by interacting with them and refining documentation (Hallinan et al., 16 Feb 2026). The paper distinguishes two kinds of opacity: Type 1 – Documentation Opacity, where the function is describable but poorly documented, and Type 2 – Intrinsic Opacity, where the tool’s behavior is complex or hard to describe even for its creators.

The benchmark contains three environments. BFCL-Opaque adapts function-calling tasks by obfuscating tool identities and degrading documentation. Chess requires the agent to choose among move-suggestion tools with identical interfaces but different hidden behavior. BrowseComp Domains frames long-trajectory question answering through opaque domain-specific search tools (Hallinan et al., 16 Feb 2026). These environments are explicitly associated with different abilities: structured or unstructured inputs, process feedback, cross-trajectory learning, test-time generalization, and tool sequencing.

BFCL-Opaque defines three documentation regimes: Anon. Fn. Names Only, Anon. Fn. Names + Real Desc, and Anon. Fn. Names + Parameter Names (Hallinan et al., 16 Feb 2026). Its metrics are Execution Accuracy, Parameter Accuracy, and AST Accuracy. Chess is evaluated with Best Tool Accuracy and Streaming Elo. BrowseComp Domains is evaluated with accuracy and number of tool calls. These are not interchangeable targets: one environment measures structural function-call fidelity, another practical gameplay strength, and another long-horizon search behavior.

The associated method, ToolObserver, alternates between exploration and reflection. In offline mode, the algorithm initializes documentation $\mathcal{T}$2, collects trajectories with an agent model, and updates documentation through an editor model over $\mathcal{T}$3 iterations. In online mode, used for instance-specific unseen tools, refinement occurs per test instance and uses only execution feedback rather than gold answers (Hallinan et al., 16 Feb 2026). This framework is contrasted with EasyTool and Play2Prompt, both of which are reported to be less effective when tools are opaque.

The benchmark’s reported results emphasize both effectiveness and efficiency. In BFCL-Opaque with Anon. Fn. Names Only, ToolObserver raises GPT-5 from E = 0.00 under the Base setting to E = 0.80, with corresponding P = 0.78 and A = 0.89 (Hallinan et al., 16 Feb 2026). In Chess, ToolObserver improves over Base in both Best Tool Accuracy and Streaming Elo. In BrowseComp, it generally exceeds or matches the baselines, especially in harder settings. The paper further states that, for test-time tool exploration, ToolObserver consumes 3.5–7.5× fewer total tokens than the best baseline (Hallinan et al., 16 Feb 2026).

In this usage, “O-Bench” designates a benchmark for agentic adaptation under partial observability of tool semantics, rather than for model alignment or algorithmic code synthesis.

5. Relation to earlier benchmark methodology

The historical survey “Object Database Benchmarks” provides a useful baseline for interpreting the later uses of “O-Bench,” even though it explicitly states that no benchmark by that name appears in the paper (Darmont, 2017). In that survey, a database benchmark consists of two main parts: a workload model—“a database and a set of read and write operations to apply on this database”—and performance metrics, defined as “simple or composite metrics aimed at expressing the performance of a system” (Darmont, 2017).

The survey further emphasizes Gray’s four criteria for a good benchmark: relevance, portability, simplicity, and scalability (Darmont, 2017). It compares object-database benchmarks such as OO1, HyperModel, OO7, and OCB, and highlights the persistent conflict between realism and implementability. The paper also distinguishes generic benchmarks from specific benchmarks, noting that more generic and parameterized benchmarks can emulate wider workloads but are harder to configure meaningfully (Darmont, 2017).

This suggests a continuity at the level of benchmark design principles, even though the application domains are entirely different. OR-Bench, OIBench, and OpaqueToolsBench each instantiate the same broad pattern: a deliberately constructed workload or task environment, paired with explicit metrics that operationalize a particular failure mode or capability frontier. A plausible implication is that the reuse of the “O-Bench” label reflects naming convenience, whereas the deeper continuity lies in benchmark methodology rather than in benchmark identity.

6. Comparative interpretation

Taken together, the contemporary “O-Bench” usages span three different evaluation objects. OR-Bench measures policy calibration in aligned LLMs, specifically the balance between refusing harmful prompts and answering safe but risky-looking ones (Cui et al., 2024). OIBench measures algorithmic reasoning and implementation efficiency under olympiad-style constraints (Zhu et al., 12 Jun 2025). OpaqueToolsBench measures tool-behavior acquisition when documentation is degraded or the tool is intrinsically opaque (Hallinan et al., 16 Feb 2026).

They also differ sharply in what counts as a successful model. In OR-Bench, the central outcome is a low ORR together with a high TRR. In OIBench, success requires all-correct execution under hidden tests, supplemented by favorable Time/Space Completion Curves. In OpaqueToolsBench, success is environment-specific: structurally correct function calls in BFCL-Opaque, stronger tool-selection policy in Chess, or more accurate and efficient search trajectories in BrowseComp (Cui et al., 2024, Zhu et al., 12 Jun 2025, Hallinan et al., 16 Feb 2026).

A plausible implication is that “O-Bench” has become a convenient label for frontier stress tests, but not for a unified benchmarking standard. The supplied literature instead supports a more precise conclusion: references to “O-Bench” are only meaningful when anchored to the full benchmark name and research domain. Without that disambiguation, the term can refer to over-refusal analysis, olympiad-level code reasoning, opaque tool learning, or, historically, to no named benchmark at all (Darmont, 2017).