Croissant Tasks: Declarative ML Evaluation Metadata
- Croissant Tasks is a declarative, machine-actionable metadata format for ML evaluation that separates task definitions from implementations.
- It standardizes crucial evaluation details—such as prompt templates, hyperparameters, preprocessing steps, and metric definitions—to mitigate reproducibility failures.
- Empirical validation shows higher metric-implementation correctness using Croissant Tasks, underscoring its potential for improving conceptual reproducibility.
Croissant Tasks is a declarative, machine-actionable metadata format for representing machine learning tasks, especially benchmark evaluations, in a form that is both human interpretable and directly usable by software agents. It is proposed to address reproducibility failures arising from underspecified execution details and brittle software environments, and it is explicitly aimed at conceptual reproducibility: verifying claims through independent, agent-generated implementations rather than by rerunning the original code in an identical environment (Benjelloun et al., 28 May 2026).
1. Problem formulation and reproducibility model
The format is motivated by a specific diagnosis of reproducibility failure in machine learning. The paper identifies three recurring causes: underspecification of crucial evaluation details such as prompt templates, hyperparameters, preprocessing steps, data filtering and splits, metric definitions, and evaluation-time settings; dependence on brittle software environments involving library versions, operating systems, hardware, APIs, and model snapshots; and a broader mismatch between the conceptual object a paper claims to evaluate and the environment-specific artifacts through which the claim is instantiated (Benjelloun et al., 28 May 2026).
A central distinction is drawn between technical replication and conceptual reproducibility. Technical replication means executing the original code in an identical environment. Conceptual reproducibility means reconstructing the same scientific claim from a precise specification, even when the implementation is independently regenerated. Croissant Tasks is explicitly designed for the latter. Its purpose is not bitwise identity with a historical run, but a faithful, implementation-independent reconstruction of evaluation logic.
This positioning matters because many benchmark claims are stable at the level of task definition, expected inputs and outputs, metrics, and execution constraints, yet unstable at the level of source code dependencies. The format therefore treats the benchmark as a first-class scientific object rather than as an incidental by-product of a codebase.
2. Specification, ontology, and representational model
Croissant Tasks extends the Croissant ecosystem and the broader schema.org vocabulary. In the formal Turtle schema, the top-level class is croissant:Task, modeled as a subclass of schema:CreativeWork. The specification then introduces croissant:TaskProblem, croissant:TaskSolution, croissant:EvaluationTask, and croissant:EvaluationResult, together with task-level properties such as croissant:input, croissant:output, croissant:implementation, croissant:execution, croissant:evaluation, and croissant:subTask (Benjelloun et al., 28 May 2026).
| Entity | Role |
|---|---|
croissant:Task |
Generic task class |
croissant:TaskProblem |
Abstract problem definition |
croissant:TaskSolution |
Concrete response to a problem |
croissant:EvaluationTask |
Evaluation-specific task |
croissant:EvaluationResult |
Metric-value pair |
The paper’s core modeling choice is the separation of problem from solution. A TaskProblem defines an incomplete task using a mixture of concrete givens and specification placeholders. A TaskSolution supplies a concrete implementation, execution configuration, outputs, and evaluation results. The bridge between them is schema:isBasedOn, which links a solution or evaluation back to its defining problem. This formalizes a familiar social structure of benchmarks: creators publish a task definition, and model developers publish solutions against it.
To support incomplete or requirement-style specifications, the ontology defines InputSpec, OutputSpec, ImplementationSpec, ExecutionSpec, and EvaluationSpec. Output schemas can be represented using croissant:RecordSet and croissant:Field. In the MMLU example, the output is constrained to an answer field with data type xsd:string and value pattern ^[A-D]$, while expected evaluation is encoded through cr:expectedMetric with "Accuracy" (Benjelloun et al., 28 May 2026). Subtasks are recursive and can encode categories, benchmark partitions, or dataset-specific slices.
The result is a declarative description of evaluation structure rather than a workflow program. The specification says what a valid task, solution, and evaluation must contain; it does not prescribe line-by-line procedural execution.
3. Authoring, extraction, and validation workflow
The paper contributes not only a schema but also an automated retrofit pipeline, pdf2ct, for converting existing benchmark papers into Croissant Tasks files. The pipeline takes as input a paper PDF, optional paper and dataset URLs, the Croissant Tasks README/specification, SHACL shapes and ontology files, and a Python validator. It writes exactly one problem.jsonld, zero or more solutions/<slug>.jsonld, a summary.md, and a validation_report.json (Benjelloun et al., 28 May 2026).
The extraction procedure is specified step by step. It begins with environment setup using pyshacl and rdflib, then instructs the agent to read the paper and extract task identity, inputs, outputs, metrics, evaluation protocol, subtasks, and reported baselines. The prompt explicitly requires the agent to distinguish what is explicit in the source from what is inferred. It then derives a stable @id base, constructs a TaskProblem, generates TaskSolution files for reported baselines, validates all artifacts using SHACL, iterates repairs for up to three rounds, and finally writes both narrative and machine-readable validation summaries.
This workflow reflects the paper’s broader claim that benchmark reconstruction can be standardized. Rather than treating paper-to-code translation as an ad hoc manual process, Croissant Tasks treats it as a metadata extraction and validation problem. The SHACL layer is therefore not incidental; it is the mechanism that turns a descriptive ontology into a machine-checkable contract.
4. Empirical validation and reproduction results
The empirical evaluation uses five benchmark papers from the NeurIPS Datasets and Benchmarks track: Absence Bench, CoRe, MedSG-Bench, NOVA, and SAGE-Eval. Expressivity is measured by whether the extraction agent can represent the benchmark in Croissant Tasks and pass validation; reproduction is measured by whether an autonomous agent can implement evaluation metrics correctly either from the Croissant Task file alone or from the PDF alone (Benjelloun et al., 28 May 2026).
Coverage results show that the extracted Croissant Tasks descriptions represented all evaluated benchmarks with an average coverage of 97.4%. CoRe’s lower score is explicitly attributed to extraction missing three hyperparameters, not to a schema deficiency.
| Benchmark | Coverage (%) |
|---|---|
| Absence | 100 |
| CoRe | 87 |
| MedSG | 100 |
| NOVA | 100 |
| SAGE | 100 |
The reproduction experiment compares two conditions: Croissant Tasks Only and PDF Only. The agent in the Croissant Tasks condition receives only the task metadata, with web access monitored so that it cannot fetch the original code. Average metric-implementation correctness is 97.1% for Croissant Tasks Only and 90% for PDF Only.
| Benchmark | Croissant Tasks Only (%) | PDF Only (%) |
|---|---|---|
| Absence | 100 | 100 |
| CoRe | 100 | 100 |
| MedSG | 100 | 100 |
| NOVA | 85.7 | 50 |
| SAGE | 100 | 100 |
The appendix adds benchmark-specific detail. In CoRe, the PDF-to-task extraction initially missed three Appendix E hyperparameters—decoding temperature, max_output_tokens = 2048, and fallback-retry count up to 3—yet the paper states that this was an extraction inconsistency rather than a schema limitation. In NOVA, the largest gap came from agents using proxy metrics instead of the proper domain libraries; PDF-only implemented only 7 of 14 metrics correctly, while the Croissant Tasks setting reached 12 of 14. In SAGE-Eval, both pipelines reached full component correctness after one guidance prompt. Across these cases, the paper attributes Croissant Tasks’ advantage to reduced context overload and a clearer structural blueprint for subtasks, metrics, and output schemas (Benjelloun et al., 28 May 2026).
These results do not establish exact numerical replication as the target. The paper explicitly allows small numeric discrepancies and evaluates whether the generated pipeline is conceptually faithful: correct task structure, correct metric computation, and acceptable numerical proximity given model drift and nondeterminism.
5. Position within the Croissant ecosystem
Croissant Tasks is built on the earlier Croissant dataset standard. Croissant itself is a metadata format for ML-ready datasets, built as an extension of schema.org/Dataset and serialized in JSON-LD; it describes dataset identity, resources, structure, and ML-relevant semantics such as labels and splits (Akhtar et al., 2024). Croissant Tasks shifts the focus from dataset description to evaluation description: from “what is the data?” to “what task consumes the data, under what execution assumptions, and what constitutes a valid result?” (Benjelloun et al., 28 May 2026).
It also sits alongside Croissant-RAI, which extends Croissant with machine-readable responsible-AI documentation covering dataset life cycle, labeling, participatory data, fairness and safety, and compliance-relevant fields such as rai:dataBiases, rai:dataLimitations, rai:dataUseCases, and rai:personalSensitiveInformation (Jain et al., 2024). A plausible implication is that Croissant Tasks supplies the evaluation-layer counterpart to a broader ecosystem in which datasets, governance metadata, and evaluations are all represented in interoperable JSON-LD artifacts.
The surrounding ecosystem already includes tool support and dataset-side adoption. Croissant Baker generates validated Croissant metadata locally from dataset directories through a modular handler registry and was evaluated on more than 140 datasets, including MIMIC-IV at 886 million rows and 374 Parquet files (Attrach et al., 14 May 2026). Several benchmark or dataset releases now describe Croissant metadata as a release artifact: MIRROR states that “Code, data, and Croissant metadata will be released publicly with the benchmark” and specifies an intended file path data/croissant_metadata.json (Wang, 15 Apr 2026); GroupAffect-4 reports a public release with a BIDS-inspired structure, Croissant metadata, a datasheet, per-session quality reports, and open processing scripts (Seikavandi et al., 19 May 2026); HackerSignal states that its code, data, and Croissant metadata are available through Hugging Face and GitHub (Ampel et al., 4 May 2026).
In this context, Croissant Tasks can be read as an attempt to add a benchmark-level semantic layer to an ecosystem that already has dataset-level metadata, responsible-AI extensions, local-first generation tools, and emerging repository-side adoption.
6. Scope, limitations, and implications
The paper presents Croissant Tasks as an early feasibility study rather than a complete solution to reproducibility. Its reported limitations are explicit: only five benchmarks are evaluated; the experiments focus on benchmarks and evaluations rather than general ML workflows; metadata fidelity remains critical; extraction quality can vary with long-context reading; some important details may be absent from papers and require code inspection or human input; and exact replication remains out of scope (Benjelloun et al., 28 May 2026).
The observed failure modes are correspondingly concrete. Agents may omit metrics under PDF context overload, substitute simplified proxy metrics for difficult domain metrics, inherit incompleteness when the extracted task skeleton is missing hyperparameters, underperform when prompts are more generic than appendix-specific prompts, or fail to parse model outputs that do not conform to the expected schema. These are not edge cases in the paper’s presentation; they are part of the argument for why a structured intermediate representation is needed.
At the same time, the paper treats the format as extensible and interoperable. It explicitly suggests that Croissant Tasks could be consumed by evaluation harnesses and platforms such as HELM, lm-evaluation-harness, OpenCompass, lmms-eval, and Codabench, and it positions the format above workflow systems such as Snakemake and Nextflow rather than as a replacement for them (Benjelloun et al., 28 May 2026). This suggests a division of labor in which Croissant Tasks specifies the benchmark contract, while execution engines implement the contract.
The broader implication is methodological. If benchmark claims can be published as typed, validated, machine-actionable task objects, then evaluation reproducibility can be shifted away from brittle artifact preservation and toward stable, implementation-independent scientific specification. The paper does not claim that this transition is complete, but it does claim that the intermediate representation is now expressive enough, validated enough, and agent-usable enough to make such a transition technically plausible (Benjelloun et al., 28 May 2026).