---
title: 'OPTION Ontology: Unified Benchmarking Model'
url: https://www.emergentmind.com/topics/option-ontology
type: topic
---

# OPTION Ontology: Unified Benchmarking Model

Searching arXiv for the specified OPTION ontology papers and related versions.
arxiv_search.query({"search_query":"ti:\"OPTION: OPTImization Algorithm Benchmarking ONtology\" OR abs:\"OPTImization Algorithm Benchmarking ONtology\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"})
Looking up the exact arXiv records to ground the article in the cited versions.
arxiv_search.query({"search_query":"id:2211.11332 OR id:2104.11889","max_results":5,"sort_by":"relevance","sort_order":"descending"})
OPTION is an ontology for optimization algorithm benchmarking that provides a semantically rich, machine-readable vocabulary for the core entities of the benchmarking process, including algorithms, problems, evaluation measures, scenarios, executions, and provenance. It was introduced to address the fragmentation of benchmarking data across platforms such as COCO, Nevergrad, and IOHprofiler, where heterogeneous CSV, text-file, JSON, and file-based schemas hinder identification of relevant datasets, interpretation of results, interoperability, and reuse. In its mature formulation, OPTION is designed as an interoperable knowledge-graph layer that supports semantic annotation, automatic data integration, federated querying, logical inference, and meta-analysis over benchmark data [2211.11332].

## 1. Motivation and problem setting

Benchmarking continuous or discrete black-box optimizers produces performance data in a variety of weakly compatible formats. The reported motivation for OPTION is that this fragmentation makes it difficult to identify which algorithms have been run on which problem instances, compare evaluation measures across studies, and reuse exploratory landscape analysis (ELA) features that characterize problem instances. In response, OPTION provides a shared ontology-based vocabulary for entities such as “algorithm,” “problem,” “run,” “measure,” “scenario,” and “provenance,” together with formal semantics in OWL and an RDF+SPARQL infrastructure for integration and querying [2104.11889].

The 2021 and 2022 presentations describe the same core objective with different emphases. The earlier account presents OPTION primarily as a shared conceptual model and query substrate for previously siloed benchmark outputs. The later account expands this into a semantically aligned ontology that supports automatic integration, improved interoperability, powerful querying capabilities, and reuse of landscape features, and demonstrates these goals on benchmark performance data from the BBOB collection of the COCO framework and the YABBOB family of the Nevergrad environment, as well as through integration into IOHprofiler [2211.11332].

A common misunderstanding is to treat OPTION as merely a file-conversion layer. The ontology is instead intended to formalize the benchmarking process itself: it models not only raw results, but also study design, algorithm implementations, executions, performance measures, problem instances, and provenance. This broader scope is explicit in both versions of the work and underlies the ontology’s role in reasoning and meta-analysis rather than only storage conversion [2104.11889].

## 2. Ontological architecture and external alignment

The early presentation splits the vocabulary into four conceptual modules, each in its own URI sub-namespace: Algorithms, Problems, Evaluation, and Provenance, with a small core class `:BenchmarkEntity` in `http://w3id.org/option/core#` used so that all modeled entities can be typed as benchmarking artifacts. The module structure includes classes such as `:Algorithm`, `:AlgorithmRun`, `:Problem`, `:ProblemInstance`, `:Dimension`, `:PerformanceMeasure`, `:BenchmarkScenario`, `:MeasurementType`, `:Study`, and `:Publication` [2104.11889].

The later presentation recasts the ontology in a more explicitly OBO-aligned form. OPTION aligns with upper-level ontologies such as BFO, mid-level ontologies such as the Information Artifact Ontology and OBI, and reuses standard relations from the Relations Ontology, Dublin Core, OntoDT, and SIO. Ontology engineering followed the OBO Foundry principles of single inheritance, no orphan classes, and heavy reuse of externally defined relations. This alignment is intended to ensure that heterogeneous datasets can be annotated in a shared conceptual model, thereby enabling automatic integration, federated querying, and logical inference over benchmark data [2211.11332].

A central modeling commitment is the specification–implementation–execution pattern. In practice, every core concept such as an optimization algorithm is split into three OWL classes: a specification class for the general, platform-independent definition; an implementation class for software-specific details; and an execution class for run-time events and inputs/outputs. This pattern distinguishes conceptual identity from software realization and from observed execution. A plausible implication is that the ontology was designed to preserve comparability across benchmarking platforms even when they expose different levels of operational detail.

## 3. Core entities, properties, and formal constraints

In the later hierarchy, the top-level distinction follows BFO’s continuant/process split. Under `InformationArtifact` appear classes such as `BenchmarkProblem`, `OptimizationAlgorithm`, `PerformanceEvaluationMeasure`, and `DataItem`; under `Process` appear `OptimizationAlgorithmBenchmarkStudyDesign`, `OptimizationAlgorithmBenchmarkStudyDesignExecution`, and `OptimizationAlgorithmBenchmarkExecution`, the latter containing `OptimizationAlgorithmExecution`, `ExperimentRun`, and `FunctionEvaluationRun`. This organization makes the ontology explicitly event-oriented for executions while retaining informational artifacts for problem and measure descriptions [2211.11332].

Representative axioms are given in Description Logic form:

$$
\text{AlgorithmImplementation} \sqsubseteq \text{OptimizationAlgorithm}
$$

$$
\text{OptimizationAlgorithmExecution} \sqsubseteq \text{Process}
\sqcap \exists \text{hasInput}.\text{OptimizationAlgorithmImplementation}
\sqcap \exists \text{hasOperand}.\text{BenchmarkProblem}
$$

$$
\text{OptimizationAlgorithmBenchmarkExecution} \equiv \text{Process}
\sqcap \exists \text{hasPart}.\text{OptimizationAlgorithmExecution}
\sqcap \exists \text{hasOutput}.\text{ExperimentRun}
$$

$$
\text{ExperimentRun} \sqsubseteq \text{Process}
\sqcap \exists \text{hasInput}.\text{BenchmarkProblem}
\sqcap \exists \text{hasOutput}.\text{PerformanceEvaluationMeasure}
$$

$$
\text{Solution} \sqsubseteq \text{InformationArtifact}
\sqcap \exists \text{hasCoordinateValue}.\text{xsd:double}
$$

The ontology also uses Manchester-syntax constraints such as `BenchmarkProblem` being a subclass of `InformationArtifact` with `hasDimension exactly 1 xsd:integer` and `hasNumberOfObjectives exactly 1 xsd:integer`, and `PerformanceEvaluationMeasure` being a subclass of `DataItem`. Data-property constraints include `BenchmarkProblem ⊑ hasDimension exactly 1 xsd:integer` and `OptimizationAlgorithmExecution ⊑ hasExecutionTime max 1 xsd:duration` [2211.11332].

The 2021 account presents a related but simpler class/property schema centered on `:AlgorithmRun`, `:ProblemInstance`, `:PerformanceMeasure`, and `:BenchmarkScenario`. There, `hasAlgorithm` is functional, each `AlgorithmRun` has exactly one `Algorithm`, each `ProblemInstance` belongs to exactly one `Problem` via a functional `instanceOf` relation, and `BenchmarkScenario` has exactly one of `hasBudget` or `hasTarget`. It also defines evaluation-oriented data properties such as `:achievedFopt` and `:functionEvaluations` as subproperties of `:hasValue`, and provenance-oriented properties such as `:hasDOI`, `:hasTitle`, and `:hasYear` for `:Publication` [2104.11889].

The object and data properties emphasized in the 2022 account include `hasOperand`, `hasPart`, `hasInput`, `hasOutput`, and `isAbout`. `ro:has_part` is reused as a transitive property linking study executions with individual algorithm executions. `isAbout` links a `DataItem`, such as an `ELAFeatureValue`, to the corresponding `BenchmarkProblem`. Data properties include `hasDimension`, `hasNumberOfInstances`, `hasNoiseLevel`, and `hasCoordinateValue`. Together these properties support both event-level benchmarking traces and problem-level descriptors such as landscape features [2211.11332].

## 4. Representation, querying, and provenance

OPTION is implemented as an RDF and SPARQL-based infrastructure. The 2021 presentation already describes an Apache Jena/Fuseki stack enabling powerful cross-platform queries and inferences, and notes that such an infrastructure can serve as the basis for web GUIs or tool integrations that hide SPARQL complexity from end users [2104.11889].

The later implementation details this workflow more concretely. An annotation pipeline built with the Apache Jena RDF API ingests raw COCO and Nevergrad files and emits RDF/XML graphs, which are loaded into an Apache Jena TDB2 triplestore. A Fuseki2 SPARQL endpoint then exposes the store through a RESTful query service at `/sparql` and an upload service at `/upload`. Provenance is encoded either through dedicated `Study` and `Publication` structures or through Dublin Core properties, depending on the presentation; the 2022 use cases explicitly capture DOI, paper title, authors, and year via Dublin Core properties [2211.11332].

The sample queries illustrate the ontology’s intended retrieval semantics. One 2022 query retrieves the final noise-free optima (`Fopt`) for the first five instances of functions `f1` and `f7`, within a fixed-budget range of 1000–2000 evaluations, for all algorithms in a study with a given DOI. Another lists all algorithms run on BBOB `f17` in dimension 20 together with the median number of function evaluations required to reach target `1e-8`. The 2021 presentation gives a related query for CMA-ES at budget 1000 over functions `f1` and `f7`, returning `fopt` across instances 1–5 [2211.11332].

These examples show that OPTION does not merely store benchmark results as opaque blobs. It exposes typed relations between algorithms, implementations, problem instances, scenarios, runs, and measures. This suggests that the ontology is optimized for compositional queries in which provenance, benchmarking conditions, and performance values are retrieved together rather than through platform-specific parsers.

## 5. Empirical integrations and annotated resources

The principal empirical demonstrations involve COCO-BBOB, Nevergrad/YABBOB, and ELA datasets. The following summary reflects the reported integrations [2211.11332].

| Resource | Coverage | Annotation characteristics |
|---|---|---|
| COCO-BBOB | 226 algorithms from the 2009–2020 workshops; 24 benchmark functions; dimensions $\{2,3,5,10,20,40\}$ | Data recorded at each improving function evaluation; `*.info` and `*.dat` files merged into RDF individuals; provenance captured via Dublin Core |
| Nevergrad / YABBOB | 32 algorithms across 10 YABBOB variants | Only final solution quality recorded; metadata include number of parallel workers and noise level; missing function-evaluation runs produce no `FunctionEvaluationRun` individuals |
| ELA | 46 ELA features for first five instances of 24 BBOB functions in dimensions $\{5,10,15,20,25,30\}$ | Features computed by `flacco`; each value is a `DataItem` linked by `isAbout`; qualified by sampling technique and sample size; 100 repetitions aggregated via `medianValue` |

For COCO-BBOB, semantic annotations are generated by a parser that merges `*.info` and `*.dat` files into RDF individuals. Each problem instance, such as `:f1_i1_dim2`, is typed as `:f1`, annotated with `hasDimension 2`, and linked through `OPTION:hasOperand` to algorithm-execution individuals. The recorded measurements include evaluation number, raw \(f(x)\), transformed \(f(x)\), and best-so-far at each improving function evaluation. This yields a fine-grained execution trace model rather than only endpoint summaries [2211.11332].

For Nevergrad and the YABBOB family, the ontology accommodates a coarser granularity. Only final solution quality is recorded, along with metadata such as the number of parallel workers and the noise level. The reported result is that OPTION’s flexible design required no schema changes; missing function-evaluation runs simply yield no `FunctionEvaluationRun` individuals. This is an important modeling point because it shows that OPTION was constructed to represent both dense time-series-like traces and sparse final-result datasets within the same ontology [2211.11332].

The integration of ELA features extends the ontology beyond algorithm performance data into problem-characterization metadata. The dataset contains 46 ELA features, computed with the R package `flacco` and publicly archived on Zenodo. Each feature value is modeled as a `DataItem` connected to its `BenchmarkProblem` with `isAbout`, further qualified by the sampling technique, such as `LHS` or `Sobol`, and by sample size, such as `1000D`, with 100 repetitions aggregated via `medianValue`. This allows benchmark performance and problem landscape descriptors to coexist in the same knowledge base [2211.11332].

## 6. Integration into IOHprofiler and analytical consequences

OPTION is integrated into the IOHprofiler environment through IOHanalyzer, the graphical companion of IOHprofiler. To hide SPARQL complexity, a lightweight query interface allows selection of suite, function, algorithm, and budget or target from drop-down menus, while SPARQL queries are constructed dynamically under the hood. The retrieved data flow directly into IOHanalyzer’s performance plots, including ECDF, fixed-target curves, and budget-vs-target heatmaps [2211.11332].

The deployment also includes a curator-assisted web portal through which researchers can upload new COCO or Nevergrad data. After format validation, the data are semantically annotated and pushed into the OPTION knowledge base. The stated purpose is to promote organic growth of the knowledge base. This operational detail situates OPTION not only as a static ontology but as the data model underlying a continuously extensible benchmarking repository [2211.11332].

The ontology’s analytical value follows from three specific design choices reported in the 2022 account: uniform classes for fixed-target versus fixed-budget scenarios, explicit links from performance measures to both problem instances and algorithm implementations, and encoded provenance. These modeling choices enable meta-analyses framed in the paper as questions such as “Which algorithm families are best on ill-conditioned, noisy functions under a budget of 10 000 evals?”, “How does dimension scaling interact with sampling strategy in landscape features?”, and “Can we predict performance (link prediction in a knowledge graph) for new algorithm–problem pairs via KG embeddings?” [2211.11332].

Taken together, the 2021 and 2022 formulations present OPTION as an ontology-driven knowledge-graph infrastructure for optimization benchmarking. Its core contribution is not a new performance metric or benchmark suite, but a formal semantic layer that unifies algorithms, problems, executions, measures, landscape descriptors, and provenance across heterogeneous platforms. A plausible implication is that OPTION occupies the interface between benchmarking practice and machine-actionable research infrastructure: it transforms fragmented benchmark outputs into an interoperable representation suitable for querying, reasoning, visualization, and downstream meta-research [2104.11889].

Source: https://www.emergentmind.com/topics/option-ontology