National Science Data Fabric (NSDF)

• NSDF is a national-scale cyberinfrastructure that integrates data transformation, hosting, indexed access, and workflow coordination over federated storage systems.
• NSDF demonstrated its utility by converting a dark-matter calibration dataset using MIDAS parsing, NumPy processing, multi-resolution IDX output, and an interactive web dashboard.
• NSDF systems emphasize distributed storage, metadata-driven discovery, and reproducible workflows to enable seamless cross-institutional scientific collaboration.

The National Science Data Fabric (NSDF) denotes a national-scale scientific cyberinfrastructure pattern in which data management, transformation, hosting, access, and workflow integration are provided as shared services over heterogeneous scientific datasets and execution environments. In the most explicit published NSDF use case, NSDF services were used to convert the University of Minnesota R76 Cryogenic Dark Matter Search calibration dataset from a collaboration-specific MIDAS representation into an open, multi-resolution IDX structure, expose it through a web-based dashboard and a Python-compatible CLI, and stage selected subsets into HPC workflows. Closely related systems place these capabilities within a broader family of federated scientific data fabrics that emphasize distributed storage, common access layers, metadata-driven discovery, and reproducible cross-institutional use (Roberts et al., 17 Jul 2025, Stansberry et al., 2020, Andrijauskas et al., 14 May 2026).

1. Architectural conception

A recurring misconception is that an NSDF is simply a repository. The direct NSDF dark-matter example points instead to an infrastructure layer that performs data transformation, hosting, indexed access, and workflow coupling. In closely aligned systems, DataFed is described as “a lightweight, distributed SDMS that spans a federation of storage systems within a loosely-coupled network of scientific facilities,” with managed entities represented as “two distinct yet synchronized parts: a raw data object and a data record,” and with a control protocol “distinct from the GridFTP protocol.” The ESCAPE Scientific Data Lake separates a control layer formed by Rucio, FTS, and Indigo IAM from heterogeneous storage endpoints and compute environments. OSDF organizes access through origins, caches, and redirectors, with Pelican providing centralized registration, cache selection, and monitoring. This suggests that NSDF is best understood less as a monolithic archive than as a control-and-coordination layer over distributed storage, transport, caching, and access services (Stansberry et al., 2020, Grange et al., 2022, Andrijauskas et al., 14 May 2026, Roberts et al., 17 Jul 2025).

National-scale analogues reinforce this interpretation. The Japanese mdx platform is framed as a nation-wide academic cloud tightly coupled to SINET and supercomputers, while EOSC CZ is described as a federated National Repository Platform and National Data Infrastructure rather than a single repository instance. In both cases, the national layer is an overlay spanning multiple institutional systems, identity domains, and service tiers (Suzumura et al., 2022, Antol et al., 2024).

2. Direct NSDF realization in the R76 dark-matter dataset

The clearest concrete NSDF workflow is the R76 CDMS calibration dataset. R76 is an above-ground calibration dataset collected between March and December 2022 using a CDMS-style germanium detector at the University of Minnesota. The dataset includes phonon traces from six detector channels and was produced while studying a calibration strategy involving sodium iodide scintillators and radioactive sources $^{22}\mathrm{Na}$, PuBe, and $^{241}\mathrm{Am}$ under the shielding configurations Basic, Lead Bricks, and Lead + Borax. The raw acquisition format is MIDAS, where each acquisition is an event containing multiple banks and the raw CDMS payload resides in the Data bank. For this NSDF release, only the phonon traces were analyzed (Roberts et al., 17 Jul 2025).

The transformation pipeline is explicit. MIDAS files are parsed with the CDMS-specific IOLibrary; extracted traces are written into an intermediate NumPy Zipped format preserving event structure, detector metadata, and channel assignments; and OpenVisusPy then produces three outputs: an IDX file containing waveform data in a multi-resolution, hierarchically indexed structure, a TXT mapping file linking samples to detectors, events, and channels, and a metadata file containing trigger type, readout configuration, and timestamps. These artifacts are uploaded to NSDF storage services, which the paper describes as supporting collections ranging from terabytes to petabytes. IDX is characterized as open, multi-resolution, hierarchically indexed, cache-oblivious, and suited to slicing, subsetting, and progressive streaming (Roberts et al., 17 Jul 2025).

The user-facing layer is equally central. A browser-accessible dashboard built with Panel and supporting JavaScript frameworks is integrated with NSDF-managed storage and compute services. It supports MID file selection, event navigation, detector or readout filtering, channel toggling, metadata inspection, and dynamic waveform plotting. Programmatic access is provided by a Typer-based CLI that supports file-level retrieval and fine-grained event/channel selection, returning standard NumPy arrays. A representative downstream workflow uses the dashboard to identify relevant MID files, the CLI to move selected data into an HPC environment, and Pegasus to orchestrate GAN training on a workload requiring nearly a terabyte of memory (Roberts et al., 17 Jul 2025).

3. Federation, storage, and transport

NSDF-class systems are characterized by storage heterogeneity and transport specialization. DataFed attaches Data Repository Servers to facility-local storage through dedicated Globus endpoints and explicitly allows organizations to use any storage system “so long as it is supported by Globus,” while retaining control over allocations and data policies. The ESCAPE Scientific Data Lake similarly federates heterogeneous endpoints using Rucio for rule-based data management, FTS for inter-site transfer, and Indigo IAM for authentication and authorization, with a common namespace abstracting physical location. OSDF adds a cache-centric access path in which applications usually contact the nearest cache, selected via GeoIP, and the cache resolves origin location through redirectors; one OSDF deployment report describes 37 caches across 14 institutions and 15 origins across 6 institutions, with 294.7 PB of transfers and 5,861,721,390 transfer operations over the reported year (Stansberry et al., 2020, Grange et al., 2022, Andrijauskas et al., 14 May 2026).

National deployments show that data-fabric behavior is inseparable from network design. mdx is jointly operated by 9 national universities and 2 national research institutes and combines 368 CPU nodes, 40 GPU nodes, 1.0 PB NVMe storage, 16.3 PB Lustre HDD storage, and 10.3 PB S3-compatible object storage. It is deeply integrated with SINET through two 100 Gbps links, planned to become two 400 Gbps links, and supports SINET Layer-2 VPN extension into arbitrary VMs, VXLAN overlays, RDMA, RoCE, and MPI across VMs. ARIM-mdx adopts a related hybrid model for materials science, combining approximately 3 PB of DDN EXAScaler Storage (Lustre), Nextcloud 29.0, Jupyter-based interactive analysis on mdx, and IoT transfer devices that present themselves as USB flash drives to non-networked experimental instruments. In ARIM-mdx measurements, median latency from ARIM-mdx Jupyter to storage was 0.87 ms, and a 10 GB Rclone transfer test reported 598.8 MB/s from ARIM-mdx Jupyter to storage. This suggests that NSDF-scale fabrics are network-native and storage-heterogeneous rather than purely repository-centric (Suzumura et al., 2022, Hanai et al., 2024).

OSDF further illustrates a distinct architectural variant: origin autonomy plus geographically distributed caches. In the Big Bear Solar Observatory use case, OSDF integration provided standard and reliable data access, and OSDF caches provided local data worldwide. The key pattern is that project data remain at origins while distributed caches and redirectors provide locality, fan-out distribution, and operational observability (Montiel et al., 14 May 2026).

4. Metadata, namespace, and semantic integration

The direct NSDF dark-matter implementation uses a relatively lean metadata scheme. Its released representation consists of an IDX file for waveform values, a TXT mapping file connecting those values to detectors, events, and channels, and a metadata file encoding trigger type, readout configuration, and timestamps. The paper situates this work in the context of NSDF services such as NSDF-Catalog and FAIR digital object efforts, but the concretely described indexing layer is the hierarchical IDX representation together with the mapping and metadata sidecars (Roberts et al., 17 Jul 2025).

Adjacent systems implement richer metadata and namespace layers. DataFed provides a “logical view of data rather than a physical storage path to a named file,” with top-level metadata including title, description, keywords, references/provenance, identifier, owner, and alias, together with full-text search and queryable domain-specific structured metadata. The ESCAPE Scientific Data Lake uses Rucio’s common namespace and supports custom metadata via plugin or JSON file, while also identifying limitations: automatic rule execution over custom metadata and inequality predicates such as $01h42m30s < RA < 01h45m30s$ were still under development. Data Lab stores approximately 50 TB of catalogs comprising around 150 billion rows in PostgreSQL with Q3C spatial indexing and exposes them through SQL, ADQL, TAP, SIA, UWS, and VOSpace interfaces. MDF generates JSON metadata records via automated file extractors, extending DataCite and NIST Materials Resource Registry conventions, and supports full-text, range, fuzzy, wildcard, and faceted search (Stansberry et al., 2020, Grange et al., 2022, Olsen et al., 2019, Blaiszik et al., 2019).

The most semantically elaborate NSDF analogue in the cited literature is the Quantum Data Hub. QDH uses GEMD++, a directed typed graph extending GEMD with entities such as Instrument Run, Person, Organization, Project, Dataset, Report, Tool, Service, and Infrastructure. Its polystore spans PostgreSQL, Neo4j, Apache Solr, QDrant, and Swift, and it formalizes both FAIR and UNIT principles: usability, navigability, interpretability, and timeliness. QDH thereby models not only samples and processes but also contributors, organizations, analytical services, and infrastructure, suggesting one path by which domain-specific hubs can operate as semantically rich NSDF nodes rather than flat file stores (Gupta et al., 2024).

5. Workflows, reproducibility, and executable services

Reproducibility is a central theme across NSDF-like systems. DataFed explicitly presents federated data management as a route “Towards Reproducible Research” by assigning unique identifiers, capturing metadata and provenance, and enabling reliable staging of the correct data into the desired environment. Its APIs support instrument-produced data and metadata ingest, workflow output ingest, and provenance capture based on input-output relationships, and it introduces a Data Gateway device for edge devices and scientific instruments. The direct NSDF R76 workflow performs a similar function for dark-matter research by replacing dependence on the original monolithic CDMS software stack with an open representation, browser visualization, NumPy-returning CLI access, and compatibility with Pegasus, Snakemake, Python scientific libraries, TensorFlow, and GAN workflows (Stansberry et al., 2020, Roberts et al., 17 Jul 2025).

Data Lab expresses the same logic through a compute-near-data design: authenticated Jupyter notebooks, myDB tables, asynchronous queries, server-side crossmatches, and image cutout services allow archival workflows without first downloading petabyte-scale holdings. MDF and DLHub extend this pattern by connecting data publication and discovery to remote execution of machine-learning models and arbitrary Python functions. DLHub packages servables as Docker or Singularity containers, registers them with metadata extending Kipoi and DataCite, and executes them through Parsl on resources that include a 200-processor cluster at Argonne National Laboratory and Amazon Web Services. At a more abstract level, Open Data Fabric defines a protocol model centered on immutable event ledgers, append-only metadata chains, deterministic transformations, Event Time and System Time, and the watermark tuple $(Ts, Te)$. That protocol is not an NSDF implementation, but it clarifies how provenance-rich, replayable data products can be formalized within a decentralized data ecosystem (Olsen et al., 2019, Blaiszik et al., 2019, Mikhtoniuk et al., 2021).

These systems collectively indicate that an NSDF is not exhausted by storage federation. Its operational value lies in connecting data publication, metadata normalization, selective staging, interactive exploration, remote execution, and provenance capture into a coherent workflow substrate.

6. Identity, governance, and open questions

Identity and access in NSDF-like infrastructures are predominantly federated. DataFed requires users to hold a Globus account linked with one or more scientific institutes. The Scientific Data Lake uses Indigo IAM with institutional identities, including EduGAIN, and was migrating from X509 certificates toward OIDC, although some storage backends still required X509 for downloads. mdx uses GakuNin for single sign-on across universities and research organizations. QDH uses token-based CILogon authentication and formalizes authorization as $A_i \longrightarrow f(O_i, S_i, \beta_i)$ with $Z \rightarrow Z_g + Z_d$. OSDF operationalizes protected access through SciTokens or certificates and actively monitors authenticated access, protected-file enforcement, SSL validity, redirector health, cache and origin load, and transfer rates; one operations report lists 352 checks/monitoring across the deployment (Stansberry et al., 2020, Grange et al., 2022, Suzumura et al., 2022, Gupta et al., 2024, Andrijauskas et al., 14 May 2026).

Governance is equally important. EOSC CZ describes a national FAIR-data ecosystem centered on a National Repository Platform, National Metadata Directory, Central Discovery Portal, a Perun-based Single Authentication and Authorization Infrastructure, Data Stewardship Wizard, and twelve EOSC CZ working groups, with phased rollout from 2024 through 2026. The materials-science policy paper “Now is the time to build a national data ecosystem for materials science and chemistry research data” argues for a “distributed but federated network,” minimum metadata standards, common discovery and access protocols such as Optimade, and allocating approximately $~2\%$ of research investment to shared repositories and interoperability standards. This suggests that NSDF maturity depends on national coordination, training, stewardship roles, and durable funding as much as on transport and storage mechanics (Antol et al., 2024, Campo et al., 2022).

The literature also delineates unresolved issues. In the direct NSDF dark-matter publication, authentication, authorization, long-term governance, and formal provenance models are not described. In adjacent systems, recurring open problems include formal metadata interoperability across communities, mature schema governance, domain-specific metadata indexing and inequality predicates, DOI-based publication workflows, replication and caching policies, audit trails, quantified scalability, and detailed performance evaluation. NSDF therefore remains both a deployed practice and an evolving architectural program: one that already supports concrete cross-disciplinary workflows, but whose national-scale semantics, governance, and measurement disciplines are still under active formation (Roberts et al., 17 Jul 2025, Stansberry et al., 2020, Grange et al., 2022, Gupta et al., 2024).