Minimum-Dataset Specification

• Minimum-dataset specification is a purpose-dependent description outlining the smallest set of dataset properties required for usability, interpretability, and exchangeability.
• It emphasizes separating stable dataset identity from task-specific interpretation, using tailored metadata and pre-processing to ensure reproducibility and portability.
• It underpins practical validation and comparability, adapting structural, scientific, and safety-critical constraints to meet diverse domain-specific operational needs.

Searching arXiv for the cited works and closely related material on dataset/task specification standards. A minimum-dataset specification is a purpose-dependent description of the smallest set of dataset properties that must be made explicit for a dataset to remain usable, interpretable, and exchangeable in a given technical setting. Across the literature, that minimum is defined differently: as a portable artifact manifest for deep-learning tasks, as high-level physical metadata for astronomical observations, as instance-level integrity constraints for RDF graphs, as structured documentation for machine-learning datasets, as a certification-oriented requirement artifact for safety-critical ML, as a modality-agnostic sample schema for AI corpora, as a terminology-conformant labeling system for clinical endoscopy, and as interoperable metadata for energy disaggregation datasets (Dakkak et al., 2020, Louys et al., 2011, Schmidt et al., 2013, Giner-Miguelez et al., 2022, Picard et al., 2020, Wang et al., 2024, Cychnerski et al., 2022, Kelly et al., 2014).

1. Purpose-relative definitions

The central feature of a minimum-dataset specification is that “minimum” does not mean universally small; it means minimally sufficient for a defined operational objective. In DLSpec, the objective is portability and reproducibility of a deep-learning task, so the dataset manifest is deliberately lean: it defines the training, validation, or test dataset through versioned identity, remote resources, and verification material such as checksums, while task-specific interpretation is delegated to executable pre-processing in the model manifest (Dakkak et al., 2020). In the IVOA Characterisation Data Model, the objective is scientific discovery and interpretation of heterogeneous astronomical data, so the minimum is high-level metadata over physical parameter space: at least one CharacterisationAxis, mandatory units and coordinate system per axis, and at least one of Location or Bounds in Coverage (Louys et al., 2011).

In RDF Data Descriptions, the objective is not portability of files or discovery in physical coordinates, but explicit guarantees about graph structure. There, a minimum-dataset specification is a set of instance-level constraints over an RDF dataset, including keys, required properties, cardinalities, and domain/range constraints, with formal first-order semantics; an RDF dataset is consistent with an RDD only if all derived constraints hold (Schmidt et al., 2013). In the machine-learning dataset DSL and DSDL, the minimum is again different: one emphasizes structured dataset documentation covering metadata, composition, provenance, and social concerns, while the other emphasizes the smallest well-formed typed description that lets a generic tool know what a sample looks like, where raw objects are located, and how annotations should be interpreted (Giner-Miguelez et al., 2022, Wang et al., 2024).

Safety-critical ML pushes the term in yet another direction. In the DDS–DRS–DVP framework, the dataset specification is treated as a requirement artifact tied to system and operational requirements. The Dataset Requirement Specification refines system requirements into dataset-specific requirements concerning validity, completeness, representativeness, innocuity, traceability, acquisition chain, labeling, partitioning, and statistical sufficiency (Picard et al., 2020, Cappi et al., 2021). This suggests that the minimum-dataset specification is best understood as a lower bound on explicitness set by the intended use of the data rather than by a single cross-domain schema.

2. Minimality and the separation of concerns

A recurrent design principle is that minimality is achieved by separating stable dataset identity from unstable or task-specific interpretation. DLSpec is explicit on this point: the specification “should contain only the essential information to use a task and reproduce its reported outcome,” and therefore keeps the dataset manifest as a resource manifest while placing decoding, normalization, tokenization, augmentation, batching, and other transformations in Python pre-processing functions with signature fun(ctx, data) (Dakkak et al., 2020). The dataset is thus defined primarily as retrievable, versioned bytes plus enough layout information for the runtime to pass file paths or raw records to pre-processing.

The astronomical Characterisation model reaches minimality differently. It minimizes compulsory fields but still insists on scientifically meaningful physical abstractions: axes, coverage, resolution, sampling precision, and accuracy. The formal minimum for compliance is low, but the paper distinguishes that from a “minimal but scientifically usable” characterization, which typically includes spatial and spectral bounds, typical resolution, sampling, and typical errors (Louys et al., 2011). Here the separation is between physically invariant metadata and everything else, such as provenance or calibration history, which belong to other VO models.

RDDs separate vocabulary and entailment from integrity. The paper argues that RDFS and OWL are entailment languages rather than explicit constraint languages, and positions RDDs as a pay-as-you-go, human-readable, machine-checkable layer that states what must hold in an RDF dataset. The minimum therefore lies in explicit obligations such as TOTAL, MIN(n), MAX(n), KEY, DOMAIN, RANGE, and optional OWA or CWA closure over classes or properties (Schmidt et al., 2013). The constrained fragment may be closed-world even though RDF as a whole remains open-world.

The ML dataset DSL broadens the minimum beyond structure by making provenance and social concerns first-class parts of the description. Its three top-level components—Metadata, Composition, and Provenance and Social Concerns—encode not only schema and statistics but also gathering processes, labeling processes, demographics, known social issues, recommended uses, and discouraged uses (Giner-Miguelez et al., 2022). By contrast, DSDL keeps the technical core smaller: $$dsdl-version</code>, type definitions in <code>defs</code> or imported libraries, and a <code>data</code> section with <code>sample-type</code> and samples or sample paths (<a href="/papers/2405.18315" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Wang et al., 2024</a>). The contrast is methodological rather than contradictory. One paper treats the minimum as the least responsible documentation for data-centric ML; the other treats it as the least typed description necessary for generic tooling.</p> <h2 class='paper-heading' id='recurring-structural-elements'>3. Recurring structural elements</h2> <p>Despite domain variation, several recurring elements appear across minimum-dataset specifications: identity, <a href="https://www.emergentmind.com/topics/scope" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">scope</a> or role, structure, access, validation, and contextual semantics. The particular realization depends on whether the dataset is a file collection, a graph, an observation in physical parameter space, a safety artifact, or a typed AI corpus.</p> <div class='overflow-x-auto max-w-full my-4'><table class='table border-collapse w-full' style='table-layout: fixed'><thead><tr> <th>Setting</th> <th>Minimal elements</th> <th>Primary objective</th> </tr> </thead><tbody><tr> <td>DLSpec</td> <td>ID, version, role, URLs, checksum, access layout</td> <td>Reproducible task exchange</td> </tr> <tr> <td>Characterisation DM</td> <td>Axis, units, coord system, <code>Location</code> or <code>Bounds</code></td> <td>Discovery and scientific interpretation</td> </tr> <tr> <td>RDD</td> <td>Keys, cardinalities, domain/range, OWA/CWA sections</td> <td>Instance-level RDF guarantees</td> </tr> <tr> <td>ML dataset DSL</td> <td>Metadata, data instances, attributes, provenance, social concerns</td> <td>Search, comparison, reproducibility, bias visibility</td> </tr> <tr> <td>DSDL</td> <td><code>$$dsdl-version, defs/imports, sample-type, sample-path, samples Generic typed dataset processing DDS–DRS–DVP ODD features, coverage constraints, traceability, labeling and split rules Certification-oriented dataset assurance NILM metadata Dataset, building, meter device, meter, appliance, wiring tree Cross-dataset interoperability ERS/MST-based design MST finding codes plus added healthy, blood, and quality labels Clinically aligned ML labeling

In DLSpec, the minimum dataset specification inferred from the design principles comprises a manifest ID and version, the dataset role, remote resources such as FTP, HTTP, file server, S3, or Zenodo URLs, optional checksums for verification, and enough local access information for the runtime to derive the list of file paths passed to pre-processing (Dakkak et al., 2020). In DSDL, the corresponding minimum is language-level: $dsdl-version must be explicit, a sample schema must exist in defs or be imported, and data must declare a sample-type and either inline samples or an external sample-path; when Label is used, a class_domain must be available (Wang et al., 2024).

The safety-critical literature introduces additional minimum elements that do not appear in purely exchange-oriented schemas. The DRS and DDS require identification of ODD-related features, coverage and representativeness constraints over those features and over class distributions, acquisition-chain specification, partitioning rules, label validity, integrity and change control, and enough data to support performance estimates at a chosen confidence level (Picard et al., 2020, Cappi et al., 2021). This suggests that the structural core of a minimum specification expands as soon as the dataset becomes part of a safety argument rather than only a reusable benchmark artifact.

4. Validation, reproducibility, and comparability

Minimum-dataset specifications matter because they define what can be validated. DLSpec validates artifact identity through remote resources and checksums, validates execution context through separate hardware, software, dataset, and model manifests, and records outcomes in a reference log that binds accuracy and performance to specific manifest IDs and settings (Dakkak et al., 2020). Its decoupled design supports comparison by fixing some manifests and varying others: dataset plus model with different hardware for system comparison, or dataset plus hardware with different software for runtime comparison.

RDDs make validation logically explicit. Their semantics is defined by translation into first-order constraints over the dataset’s triple relation, so conformance is not a narrative property but a model-theoretic one: missing required properties, duplicated key tuples, or absent typing triples required by DOMAIN and RANGE are specification violations (Schmidt et al., 2013). Because the same constraints can be mapped to SPARQL ASK queries, the minimum specification also becomes an executable validation artifact.

The DDS–DRS–DVP framework turns dataset specification into a verification workflow. The Dataset Verification Plan assigns a verification procedure to each DRS requirement and may be automatic or manual. The railway example operationalizes sun-elevation coverage by computing sun elevation from GPS coordinates and timestamps and checking that the empirical histogram does not deviate from the DRS histogram by more than 10% in any category (Picard et al., 2020). The related DDS recommendations add non-overlap between train and test sets, secrecy of the test set during model development, traceability and integrity through reproducible transformations and cryptographic hashes, and a test-set size rationale based on a high-probability bound on the true error (Cappi et al., 2021).

Typed AI dataset specifications also make validation uniform. DSDL interpreters use the declared type system to validate values against Int, BBox, Label[dom=...], and other type constructors, while $optional determines which fields may be absent without warning (Wang et al., 2024). The ML dataset DSL adds structural and referential validation through its metamodel and plans OCL-based checking for ConsistencyRules, although OCL parsing and validation are not yet implemented in the current tooling (Giner-Miguelez et al., 2022). In both cases, a minimum specification is not only descriptive; it defines the contract against which tooling can reject malformed data.

5. Domain-specific realizations

Domain-specific minimum specifications differ most sharply in what they treat as irreducible context. In astronomy, the irreducible context is the physical parameter space of the observation. Characterisation requires datasets to be described in terms of spatial, spectral, temporal, observable, and other axes; coverage may be given through Location, Bounds, Support, and Sensitivity, while Resolution, SamplingPrecision, and Accuracy refine how data occupy those axes (Louys et al., 2011). The minimum compliant instance is deliberately small, but the scientifically useful instance is richer, because discovery and basic analysis depend on typical resolution, pixel or bin sampling, and typical errors.

In clinical endoscopy, the irreducible context is controlled medical terminology. The ERS dataset uses MST 3.0 as the authoritative code system for gastrointestinal findings and implements 104 MST 3.0 labels, organized as upper-endoscopy gXX and colonoscopy cXX codes, then extends them with 19 non-MST labels for healthy tissue, blood presence, and image quality attributes relevant to ML (Cychnerski et al., 2022). The paper distinguishes precise expert annotations from imprecise labels and masks propagated to neighboring video frames, preserves zero-count MST codes to demonstrate compliance with the full terminology, and recommends patient-level partitioning so that the same patient does not appear in both train and test subsets (Cychnerski et al., 2022). Here the minimum specification is terminological and label-centric rather than schema-centric.

In energy disaggregation, the irreducible context is the electrical topology of the dataset. The proposed metadata schema centers on dataset, building, meter device, electricity meter, appliance, appliance type, priors, and appliance models, with explicit representation of the mains wiring tree through site_meter and submeter_of, and at least one data_location per sensor (Kelly et al., 2014). A NILM dataset is not practically usable unless whole-home measurements, submeters, and their meter–appliance mappings can be interpreted consistently across datasets. The minimum specification therefore includes structural metadata that generic dataset schemas typically omit.

These examples show that a minimum-dataset specification is domain-complete only when it preserves the smallest context without which the data cease to have operational meaning: physical axes in astronomy, code systems in medicine, wiring and appliance semantics in NILM, or ODD and acquisition conditions in safety-critical ML (Louys et al., 2011, Cychnerski et al., 2022, Kelly et al., 2014, Picard et al., 2020).

6. Tensions, limitations, and extensions

A persistent tension in the literature is between declarative compactness and executable or contextual completeness. DLSpec resolves this by keeping the dataset manifest minimal and moving interpretation into model-side pre-processing code, but this also means that semantics such as normalization, tokenization, augmentation, or split logic are not declaratively visible in the dataset manifest itself (Dakkak et al., 2020). RDD takes the opposite route for RDF, making structural guarantees explicit and formal, yet the paper notes that it still lacks more general functional dependencies, disjunctive dependencies, value enumeration restrictions, and arbitrary user-defined constraints (Schmidt et al., 2013).

Another tension is between formal compliance and practical usefulness. Characterisation explicitly notes that the formal minimum for compliance can be much lower than the metadata needed for robust scientific interoperability; one can be compliant with only Location or Bounds, but scientific use is weakened without resolution, sampling, and accuracy information (Louys et al., 2011). The ML dataset DSL similarly shows that structured documentation should extend beyond schema into provenance and social concerns, but its current tooling does not yet validate OCL consistency rules and its evaluation was preliminary, with only three datasets modeled (Giner-Miguelez et al., 2022). DSDL, conversely, keeps its core intentionally small and generic, which improves portability but leaves richer metadata and task-specific semantics to templates, global-info, and external libraries (Wang et al., 2024).

Safety-critical specifications reveal a different limitation: many requirements remain qualitative or application-specific. The DDS and DRS emphasize representativeness, innocuity, and statistical sufficiency, but several thresholds must still be derived per operational context, and identifying all relevant ODD features remains an expert-driven challenge (Picard et al., 2020, Cappi et al., 2021). In clinical imaging, ERS shows that even a mature medical terminology such as MST 3.0 is insufficient by itself for ML-ready datasets because it lacks explicit normal classes and image-quality labels, requiring carefully bounded extensions for healthy tissue, blood presence, and artifacts (Cychnerski et al., 2022).

Taken together, these works suggest that a minimum-dataset specification is best viewed as a layered construct. The bottom layer fixes identity, structure, and access; the next layer fixes validation and comparability; and the final layer introduces the domain semantics—physical, logical, social, clinical, or safety-related—without which the minimum would be formally valid but operationally incomplete (Dakkak et al., 2020, Louys et al., 2011, Schmidt et al., 2013, Giner-Miguelez et al., 2022, Picard et al., 2020, Wang et al., 2024, Cychnerski et al., 2022, Kelly et al., 2014).