Automated Data Readiness for Scientific AI

Abstract: Leadership computing facilities steward large-scale scientific datasets that routinely require substantial transformation before serving as AI training data. However, no existing framework fully unifies automated transformation, readiness assessment, provenance tracking, and agent-native deployment. We present REDI, an open-source framework that addresses this gap through a unified five-stage pipeline (ingest, preprocess, transform, structure, and output) with per-stage instrumentation for reproducibility and deployment as an agent-callable skill; companion tool SetGo automates FAIR compliance and catalog publication. Evaluated across climate, proteomics, materials science, and nuclear fusion, REDI transforms all datasets from raw to AI-ready, with outputs validated against domain-expert references, and preliminary results show near-ideal parallel scaling to 100 nodes on Frontier for the climate case. Provenance-instrumented profiling reveals file I/O as the dominant pipeline cost, with format selection a first-order optimization lever. These results establish REDI as a cross-domain platform providing automated data readiness for scientific AI, transforming data preparation bottlenecks into reproducible, reusable community assets.

• The paper introduces REDI, a lifecycle-aware framework that automates the transformation of raw, heterogeneous data into AI-ready formats.
• It details a five-stage IPTSO pipeline with integrated provenance tracking, domain-specific transformations, and rigorous validation against ground truths.
• Empirical evaluations across climate, proteomics, materials science, and nuclear fusion demonstrate scalability and highlight I/O optimization challenges.

Automated Data Readiness for Scientific AI: REDI Framework

Introduction

The Readiness Engine for Data Integration (REDI) is an open-source, lifecycle-aware framework designed to automate data readiness for scientific AI. The framework orchestrates a five-stage pipeline—Ingest, Preprocess, Transform, Structure, and Output (IPTSO)—that integrates readiness assessment, transformation, provenance tracking, and agent-native deployment. REDI addresses the fundamental bottleneck in scientific AI: transitioning large, complex, and heterogeneous raw scientific data into AI-ready assets suitable for robust, reproducible training workflows, especially at leadership computing scale. The REDI ecosystem is further augmented by SetGo, an automated tool for FAIR compliance, metadata enrichment, and catalog publication.

Data Readiness Challenges and Motivations

AI-driven scientific discovery depends critically on data that is not only accessible but also structured, validated, and semantically rich. Existing approaches to data management, even those aligned with FAIR principles, remain inadequate for AI uses due to domain inconsistencies, missing metadata, or lack of operational standardization. The consequence is a proliferation of bespoke, irreproducible, ad hoc transformation pipelines, consuming the majority labor share in AI development cycles and introducing subtle errors or inconsistencies.

REDI directly responds to increasing demands from the scientific community and policy makers for unified, auditable, cross-domain data readiness strategies, as outlined in recent reports and standards (2607.02771). It also addresses novel failure modes, such as those introduced by LLM-driven agentic workflows, where absence of structured readiness guidance often leads to untraceable leakages or unreproducible artifacts. The REDI design space is fundamentally shaped by the following requirements:

1. Formalization of data readiness levels and taxonomies.
2. Transparent integration with HPC parallel storage and file formats.
3. Cross-domain, automated, and provenance-linked transformations.
4. Seamless integration with agentic AI programming environments.

Framework Architecture and Modes

REDI operationalizes its five-stage IPTSO pipeline via a modular, reusable architecture. Its core abstractions include a persistent shared PipelineContext, a uniform PipelineStep API with Flowcept-based provenance injection, and an extensible orchestrator capable of hosting both domain-agnostic and domain-specific steps. REDI's operation modes offer different entry points for readiness workflows:

• redi run: Fully automated, end-to-end readiness for known domains, supporting dynamic preprocessing decisions based on content, such as PII anonymization or spatial regridding.
• redi discover: For novel datasets, this mode analyzes input files, infers metadata, and generates a documented, domain-informed preprocessing plan.
• redi inspect: Observer mode for external pipelines, providing audits, intermediate data inspection, and data quality suggestions.
• redi assess: Quantitative, domain-aware readiness scoring with delta reports instrumented by provenance.
• redi validate: Statistical and semantic validation against reference ground-truth datasets via feature-wise correlation and error metrics.

REDI supports three parallelization frameworks—concurrent.futures for multi-thread/process, MPI for tightly coupled workflows, and GNU Parallel for embarrassingly parallel scenarios—along with generation of platform-native batch submission scripts for leadership HPC environments.

Empirical Evaluation and Use Cases

REDI was evaluated across four demanding scientific domains: climate science (ClimaX), proteomics (OpenFold), materials science (HydraGNN), and nuclear fusion (XGC1). Each domain presents distinct data modalities, scale challenges, and reference preprocessing pipelines.

Cross-Domain Readiness Validation

Across all cases, REDI was able to transform raw scientific data into fully AI-ready formats compatible with domain-specific foundation models:

• Spatial grids (NetCDF) for climate were regridded, normalized, and serialized.
• Protein structure and sequence data (FASTA, mmCIF) for OpenFold were feature-engineered and encoded with reference-faithful gap handling.
• Atomistic DFT outputs were graph-structured and normalized for HydraGNN.
• Fusion plasma particle-mesh outputs were projected and partitioned for direct model ingestion.

REDI outputs were validated both structurally and statistically against pre-existing domain ground truths, achieving Pearson correlations of 1.000 and essentially zero MAE across science modalities, indicating no measurable information loss through the automated pipeline.

Pipeline Bottlenecks

A detailed stage-by-stage timing analysis revealed file I/O, and format selection, as the dominant bottlenecks for readiness at scale. For instance, NPZ serialization dominated pipeline cost in climate, LMDB ingestion in materials science, and ADIOS2 BP file reads in fusion (Figure 1).

(Figure 1)

Figure 1: REDI pipeline stage timings based on dataset subsamples, each run on a single compute node. Normalization is omitted for XGC1 because it is applied during the training process.

Optimization opportunities include parallel I/O strategies, chunked array formats (e.g., Zarr for climate/fusion, HDF5 for irregular science data), and I/O pattern alignment with HPC storage infrastructure. Empirical tests demonstrated that Zarr format achieved approximately 3× higher throughput compared to NPZ under parallel access patterns.

Parallel Scalability

Preliminary scalability testing on the OLCF Frontier system for the ClimaX use case exhibited near-ideal efficiency ($\eta > 0.95$) up to 100 nodes, confirming REDI's suitability for leadership-scale workflows where strong scaling of I/O-bound tasks is vital (Figure 2).

(Figure 2)

Figure 2: Preliminary REDI parallel scalability results for the Climate (ClimaX/AWI-ESM) case. Each data point is annotated with its parallel efficiency ($\eta = S/p$).

Provenance, Metadata, and Catalog Integration

By embedding Flowcept-based event tracing into every pipeline step, REDI yields complete, queryable provenance records that precisely report the state of each dataset before and after each transformation. This provenance is lifecycle-integrated and catalog-ready. SetGo complements REDI by programmatically assessing metadata readiness across six dimensions (including FAIR compliance and governance), enabling automated publication to catalog repositories (Hugging Face, CKAN, OpenMetadata).

Implications and Future Directions

REDI establishes a generalizable, reproducible foundation for scientific data readiness, critical for scaling foundation model methods, surrogate modeling, and uncertainty quantification in the computational sciences. It formalizes data readiness as a measurable, automated, closed-loop process across scientific domains, displacing fragmented, manual pipelines.

Pragmatically, REDI lowers labor and reproducibility barriers for science teams, enabling agent and LLM-driven data workflows without sacrificing rigor, transparency, or auditability. The decoupling of domain-specific transformations and orchestration, and the deep instrumentation of provenance, are distinguishing architectural choices with implications for future self-adaptive, agentic AI systems.

Limitations exist: redi discover remains heuristic and does not cover deeply specialized domain logic; stage boundary semantics (e.g., where to classify subsampling) are sometimes ambiguous; and extension to emerging data types or sensitive data regimes remains ongoing work.

Future development will focus on further automation in discovery and execution, fine-grained I/O optimization, cross-domain modality abstraction (particularly for mesh-based data common to multiple sciences), and seamless integration with emerging DOE and global data readiness missions. As leadership AI adoption accelerates, frameworks such as REDI, which produce community-scale, reusable data transformation assets, will be indispensable infrastructure.

Conclusion

REDI provides a unified, validated approach to cross-domain scientific data readiness for AI, delivering robust, provenance-rich pipelines adaptable to leadership HPC environments. Empirical results underscore its reproducibility and scaling properties, while revealing persistent optimization levers in I/O and storage infrastructure. By bridging the preparation gap from raw to AI-ready data, REDI emerges as a key enabling technology for next-generation computational science and multi-domain foundation model training.

Reference: "Automated Data Readiness for Scientific AI" (2607.02771)