OPTION Ontology: Unified Benchmarking Model

• OPTION ontology is a semantically rich vocabulary defining algorithms, problems, measures, and provenance for optimization benchmarking.
• It employs a structured, OBO-aligned design with distinct classes for specification, implementation, and execution to ensure cross-platform consistency.
• The RDF+SPARQL-based infrastructure enables semantic querying and integration of diverse benchmark datasets from COCO, Nevergrad, and IOHprofiler.

Searching arXiv for the specified OPTION ontology papers and related versions. arxiv_search.query({"^^^^^^^^3search_query3 OPTImization Algorithm Benchmarking ONtology\"^^^^^^^^3 OR abs:\3^^^^^^^^"OPTImization Algorithm Benchmarking ONtology\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}) Looking up the exact arXiv records to ground the article in the cited versions. arxiv_search.query({"^^^^^^^^3search_query3 OR id:(&&&^^^^3ti:\3^^^^&&&)","max_results":5,"sort_by":"relevance","sort_order":"descending"}) OPTION is an ontology for optimization algorithm benchmarking that provides a semantically rich, machine-readable vocabulary for the core entities of the benchmarking process, including algorithms, problems, evaluation measures, scenarios, executions, and provenance. It was introduced to address the fragmentation of benchmarking data across platforms such as COCO, Nevergrad, and IOHprofiler, where heterogeneous CSV, text-file, JSON, and file-based schemas hinder identification of relevant datasets, interpretation of results, interoperability, and reuse. In its mature formulation, OPTION is designed as an interoperable knowledge-graph layer that supports semantic annotation, automatic data integration, federated querying, logical inference, and meta-analysis over benchmark data (&&&^^^^3search_query3^^^^&&&).

^^^^3ti:\3^^^^. Motivation and problem setting

Benchmarking continuous or discrete black-box optimizers produces performance data in a variety of weakly compatible formats. The reported motivation for OPTION is that this fragmentation makes it difficult to identify which algorithms have been run on which problem instances, compare evaluation measures across studies, and reuse exploratory landscape analysis (ELA) features that characterize problem instances. In response, OPTION provides a shared ontology-based vocabulary for entities such as “algorithm,” “problem,” “run,” “measure,” “scenario,” and “provenance,” together with formal semantics in OWL and an RDF+SPARQL infrastructure for integration and querying (&&&^^^^3ti:\3^^^^&&&).

The ^^^^3 OR abs:\3search_query3 OR abs:\3ti:\3^^^^ and ^^^^3 OR abs:\3search_query3 OR abs:\3 OR abs:\3^^^^ presentations describe the same core objective with different emphases. The earlier account presents OPTION primarily as a shared conceptual model and query substrate for previously siloed benchmark outputs. The later account expands this into a semantically aligned ontology that supports automatic integration, improved interoperability, powerful querying capabilities, and reuse of landscape features, and demonstrates these goals on benchmark performance data from the BBOB collection of the COCO framework and the YABBOB family of the Nevergrad environment, as well as through integration into IOHprofiler (&&&^^^^3search_query3^^^^&&&).

A common misunderstanding is to treat OPTION as merely a file-conversion layer. The ontology is instead intended to formalize the benchmarking process itself: it models not only raw results, but also study design, algorithm implementations, executions, performance measures, problem instances, and provenance. This broader scope is explicit in both versions of the work and underlies the ontology’s role in reasoning and meta-analysis rather than only storage conversion (&&&^^^^3ti:\3^^^^&&&).

^^^^3 OR abs:\3^^^^. Ontological architecture and external alignment

The early presentation splits the vocabulary into four conceptual modules, each in its own URI sub-namespace: Algorithms, Problems, Evaluation, and Provenance, with a small core class :BenchmarkEntity in http://w3id.org/option/core# used so that all modeled entities can be typed as benchmarking artifacts. The module structure includes classes such as :Algorithm, :AlgorithmRun, :Problem, :ProblemInstance, :Dimension, :PerformanceMeasure, :BenchmarkScenario, :MeasurementType, :Study, and :Publication (&&&^^^^3ti:\3^^^^&&&).

The later presentation recasts the ontology in a more explicitly OBO-aligned form. OPTION aligns with upper-level ontologies such as BFO, mid-level ontologies such as the Information Artifact Ontology and OBI, and reuses standard relations from the Relations Ontology, Dublin Core, OntoDT, and SIO. Ontology engineering followed the OBO Foundry principles of single inheritance, no orphan classes, and heavy reuse of externally defined relations. This alignment is intended to ensure that heterogeneous datasets can be annotated in a shared conceptual model, thereby enabling automatic integration, federated querying, and logical inference over benchmark data (&&&^^^^3search_query3^^^^&&&).

A central modeling commitment is the specification–implementation–execution pattern. In practice, every core concept such as an optimization algorithm is split into three OWL classes: a specification class for the general, platform-independent definition; an implementation class for software-specific details; and an execution class for run-time events and inputs/outputs. This pattern distinguishes conceptual identity from software realization and from observed execution. A plausible implication is that the ontology was designed to preserve comparability across benchmarking platforms even when they expose different levels of operational detail.

3. Core entities, properties, and formal constraints

In the later hierarchy, the top-level distinction follows BFO’s continuant/process split. Under InformationArtifact appear classes such as BenchmarkProblem, OptimizationAlgorithm, PerformanceEvaluationMeasure, and DataItem; under Process appear OptimizationAlgorithmBenchmarkStudyDesign, OptimizationAlgorithmBenchmarkStudyDesignExecution, and OptimizationAlgorithmBenchmarkExecution, the latter containing OptimizationAlgorithmExecution, ExperimentRun, and FunctionEvaluationRun. This organization makes the ontology explicitly event-oriented for executions while retaining informational artifacts for problem and measure descriptions (&&&^^^^3search_query3^^^^&&&).

Representative axioms are given in Description Logic form:

PRESERVED_PLACEHOLDER_^^^^3search_query3^^^^

PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^

PRESERVED_PLACEHOLDER_^^^^3 OR abs:\3^^^^

$$\text{ExperimentRun} \sqsubseteq \text{Process} \sqcap \exists \text{hasInput}.\text{BenchmarkProblem} \sqcap \exists \text{hasOutput}.\text{PerformanceEvaluationMeasure}$$

$$\text{Solution} \sqsubseteq \text{InformationArtifact} \sqcap \exists \text{hasCoordinateValue}.\text{xsd:double}$$

The ontology also uses Manchester-syntax constraints such as BenchmarkProblem being a subclass of InformationArtifact with hasDimension exactly ^^^^3ti:\3^^^^ xsd:integer and hasNumberOfObjectives exactly ^^^^3ti:\3^^^^ xsd:integer, and PerformanceEvaluationMeasure being a subclass of DataItem. Data-property constraints include BenchmarkProblem ⊑ hasDimension exactly ^^^^3ti:\3^^^^ xsd:integer and OptimizationAlgorithmExecution ⊑ hasExecutionTime max ^^^^3ti:\3^^^^ xsd:duration (&&&^^^^3search_query3^^^^&&&).

The ^^^^3 OR abs:\3search_query3 OR abs:\3ti:\3^^^^ account presents a related but simpler class/property schema centered on :AlgorithmRun, :ProblemInstance, :PerformanceMeasure, and :BenchmarkScenario. There, hasAlgorithm is functional, each AlgorithmRun has exactly one Algorithm, each ProblemInstance belongs to exactly one Problem via a functional instanceOf relation, and BenchmarkScenario has exactly one of hasBudget or hasTarget. It also defines evaluation-oriented data properties such as :achievedFopt and :functionEvaluations as subproperties of :hasValue, and provenance-oriented properties such as :hasDOI, :hasTitle, and :hasYear for :Publication (&&&^^^^3ti:\3^^^^&&&).

The object and data properties emphasized in the ^^^^3 OR abs:\3search_query3 OR abs:\3 OR abs:\3^^^^ account include hasOperand, hasPart, hasInput, hasOutput, and isAbout. ro:has_part is reused as a transitive property linking study executions with individual algorithm executions. isAbout links a DataItem, such as an ELAFeatureValue, to the corresponding BenchmarkProblem. Data properties include hasDimension, hasNumberOfInstances, hasNoiseLevel, and hasCoordinateValue. Together these properties support both event-level benchmarking traces and problem-level descriptors such as landscape features (&&&^^^^3search_query3^^^^&&&).

4. Representation, querying, and provenance

OPTION is implemented as an RDF and SPARQL-based infrastructure. The ^^^^3 OR abs:\3search_query3 OR abs:\3ti:\3^^^^ presentation already describes an Apache Jena/Fuseki stack enabling powerful cross-platform queries and inferences, and notes that such an infrastructure can serve as the basis for web GUIs or tool integrations that hide SPARQL complexity from end users (&&&^^^^3ti:\3^^^^&&&).

The later implementation details this workflow more concretely. An annotation pipeline built with the Apache Jena RDF API ingests raw COCO and Nevergrad files and emits RDF/XML graphs, which are loaded into an Apache Jena TDB^^^^3 OR abs:\3^^^^ triplestore. A Fuseki^^^^3 OR abs:\3^^^^ SPARQL endpoint then exposes the store through a RESTful query service at /sparql and an upload service at /upload. Provenance is encoded either through dedicated Study and Publication structures or through Dublin Core properties, depending on the presentation; the ^^^^3 OR abs:\3search_query3 OR abs:\3 OR abs:\3^^^^ use cases explicitly capture DOI, paper title, authors, and year via Dublin Core properties (&&&^^^^3search_query3^^^^&&&).

The sample queries illustrate the ontology’s intended retrieval semantics. One ^^^^3 OR abs:\3search_query3 OR abs:\3 OR abs:\3^^^^ query retrieves the final noise-free optima (Fopt) for the first five instances of functions f^^^^3ti:\3^^^^ and f7, within a fixed-budget range of ^^^^3ti:\3search_query3search_query3search_query3^^^^–^^^^3 OR abs:\3search_query3search_query3search_query3^^^^ evaluations, for all algorithms in a study with a given DOI. Another lists all algorithms run on BBOB f^^^^3ti:\3^^^^7 in dimension ^^^^3 OR abs:\3search_query3^^^^ together with the median number of function evaluations required to reach target ^^^^3ti:\3^^^^e-8. The ^^^^3 OR abs:\3search_query3 OR abs:\3ti:\3^^^^ presentation gives a related query for CMA-ES at budget ^^^^3ti:\3search_query3search_query3search_query3^^^^ over functions f^^^^3ti:\3^^^^ and f7, returning fopt across instances ^^^^3ti:\3^^^^–5 (&&&^^^^3search_query3^^^^&&&).

These examples show that OPTION does not merely store benchmark results as opaque blobs. It exposes typed relations between algorithms, implementations, problem instances, scenarios, runs, and measures. This suggests that the ontology is optimized for compositional queries in which provenance, benchmarking conditions, and performance values are retrieved together rather than through platform-specific parsers.

5. Empirical integrations and annotated resources

The principal empirical demonstrations involve COCO-BBOB, Nevergrad/YABBOB, and ELA datasets. The following summary reflects the reported integrations (&&&^^^^3search_query3^^^^&&&).

Resource	Coverage	Annotation characteristics
COCO-BBOB	^^^^3 OR abs:\3 OR abs:\3^^^^6 algorithms from the ^^^^3 OR abs:\3search_query3search_query3^^^^9–^^^^3 OR abs:\3search_query3 OR abs:\3search_query3^^^^ workshops; ^^^^3 OR abs:\3^^^^4 benchmark functions; dimensions $\{2,3,5,10,20,40\}$	Data recorded at each improving function evaluation; *.info and *.dat files merged into RDF individuals; provenance captured via Dublin Core
Nevergrad / YABBOB	3^^^^3 OR abs:\3^^^^ algorithms across ^^^^3ti:\3search_query3^^^^ YABBOB variants	Only final solution quality recorded; metadata include number of parallel workers and noise level; missing function-evaluation runs produce no FunctionEvaluationRun individuals
ELA	46 ELA features for first five instances of ^^^^3 OR abs:\3^^^^4 BBOB functions in dimensions $\{5,10,15,20,25,30\}$	Features computed by flacco; each value is a DataItem linked by isAbout; qualified by sampling technique and sample size; ^^^^3ti:\3search_query3search_query3^^^^ repetitions aggregated via medianValue

For COCO-BBOB, semantic annotations are generated by a parser that merges *.info and *.dat files into RDF individuals. Each problem instance, such as :f^^^^3ti:\3^^^^_i^^^^3ti:\3^^^^_dim^^^^3 OR abs:\3^^^^, is typed as :f^^^^3ti:\3^^^^, annotated with hasDimension ^^^^3 OR abs:\3^^^^, and linked through OPTION:hasOperand to algorithm-execution individuals. The recorded measurements include evaluation number, raw $f(x)$, transformed $f(x)$, and best-so-far at each improving function evaluation. This yields a fine-grained execution trace model rather than only endpoint summaries (&&&^^^^3search_query3^^^^&&&).

For Nevergrad and the YABBOB family, the ontology accommodates a coarser granularity. Only final solution quality is recorded, along with metadata such as the number of parallel workers and the noise level. The reported result is that OPTION’s flexible design required no schema changes; missing function-evaluation runs simply yield no FunctionEvaluationRun individuals. This is an important modeling point because it shows that OPTION was constructed to represent both dense time-series-like traces and sparse final-result datasets within the same ontology (&&&^^^^3search_query3^^^^&&&).

The integration of ELA features extends the ontology beyond algorithm performance data into problem-characterization metadata. The dataset contains 46 ELA features, computed with the R package flacco and publicly archived on Zenodo. Each feature value is modeled as a DataItem connected to its BenchmarkProblem with isAbout, further qualified by the sampling technique, such as LHS or Sobol, and by sample size, such as ^^^^3ti:\3search_query3search_query3search_query3^^^^D, with ^^^^3ti:\3search_query3search_query3^^^^ repetitions aggregated via medianValue. This allows benchmark performance and problem landscape descriptors to coexist in the same knowledge base (&&&^^^^3search_query3^^^^&&&).

6. Integration into IOHprofiler and analytical consequences

OPTION is integrated into the IOHprofiler environment through IOHanalyzer, the graphical companion of IOHprofiler. To hide SPARQL complexity, a lightweight query interface allows selection of suite, function, algorithm, and budget or target from drop-down menus, while SPARQL queries are constructed dynamically under the hood. The retrieved data flow directly into IOHanalyzer’s performance plots, including ECDF, fixed-target curves, and budget-vs-target heatmaps (&&&^^^^3search_query3^^^^&&&).

The deployment also includes a curator-assisted web portal through which researchers can upload new COCO or Nevergrad data. After format validation, the data are semantically annotated and pushed into the OPTION knowledge base. The stated purpose is to promote organic growth of the knowledge base. This operational detail situates OPTION not only as a static ontology but as the data model underlying a continuously extensible benchmarking repository (&&&^^^^3search_query3^^^^&&&).

The ontology’s analytical value follows from three specific design choices reported in the ^^^^3 OR abs:\3search_query3 OR abs:\3 OR abs:\3^^^^ account: uniform classes for fixed-target versus fixed-budget scenarios, explicit links from performance measures to both problem instances and algorithm implementations, and encoded provenance. These modeling choices enable meta-analyses framed in the paper as questions such as “Which algorithm families are best on ill-conditioned, noisy functions under a budget of ^^^^3ti:\3search_query3^^^^ ^^^^3search_query3search_query3search_query3^^^^ evals?”, “How does dimension scaling interact with sampling strategy in landscape features?”, and “Can we predict performance (link prediction in a knowledge graph) for new algorithm–problem pairs via KG embeddings?” (&&&^^^^3search_query3^^^^&&&).

Taken together, the ^^^^3 OR abs:\3search_query3 OR abs:\3ti:\3^^^^ and ^^^^3 OR abs:\3search_query3 OR abs:\3 OR abs:\3^^^^ formulations present OPTION as an ontology-driven knowledge-graph infrastructure for optimization benchmarking. Its core contribution is not a new performance metric or benchmark suite, but a formal semantic layer that unifies algorithms, problems, executions, measures, landscape descriptors, and provenance across heterogeneous platforms. A plausible implication is that OPTION occupies the interface between benchmarking practice and machine-actionable research infrastructure: it transforms fragmented benchmark outputs into an interoperable representation suitable for querying, reasoning, visualization, and downstream meta-research (&&&^^^^3ti:\3^^^^&&&).