SciHorizon-DataEVA: An Agentic System for AI-Readiness Evaluation of Heterogeneous Scientific Data

Abstract: AI-for-Science (AI4Science) is increasingly transforming scientific discovery by embedding machine learning models into prediction, simulation, and hypothesis generation workflows across domains. However, the effectiveness of these models is fundamentally constrained by the AI-readiness of scientific data, for which no scalable and systematic evaluation mechanism currently exists. In this work, we propose SciHorizon-DataEVA, a novel agentic system to scalable AI-readiness evaluation of heterogeneous scientific data. At the evaluation-criteria level, we introduce the Sci-TQA2 principles, which organize AI-readiness into four complementary dimensions: Governance Trustworthiness, Data Quality, AI Compatibility, and Scientific Adaptability. Each dimension is decomposed into measurable atomic elements that enable fine-grained and executable assessment. To operationalize these principles at scale, we develop Sci-TQA2-Eval, a hierarchical multi-agent evaluation approach orchestrated through a directed, cyclic workflow. Our Sci-TQA2-Eval dynamically constructs dataset-aware evaluation specifications by combining lightweight dataset profiling, applicability-aware metric activation, and knowledge-augmented planning grounded in domain constraints and dataset-paper signals. These specifications are executed through an adaptive, tool-centric evaluation mechanism with built-in verification and self-correction, enabling scalable and reliable assessment across heterogeneous scientific data. Extensive experiments on scientific datasets spanning multiple domains demonstrate the effectiveness and generality of SciHorizon-DataEVA for principled AI-readiness evaluation.

• The paper presents an agent-driven framework that evaluates AI-readiness of scientific datasets using a hierarchical, multi-agent methodology.
• It details the Sci-TQA² principles, quantifying governance, data quality, AI compatibility, and scientific adaptability with tailored, context-aware metrics.
• Experimental results reveal that traditional quality metrics alone are insufficient for AI-readiness, emphasizing the need for domain-specific evaluation.

SciHorizon-DataEVA: Agentic Evaluation of AI-Readiness in Scientific Data

Motivation and Problem Formulation

The increasing penetration of AI into scientific workflows—encompassing prediction, simulation, and hypothesis generation—necessitates a rigorous framework for evaluating the “AI-readiness” of scientific datasets. Heterogeneity in content, structure, governance, and documentation introduces significant barriers to the generalization, robustness, and scientific validity of AI models deployed in scientific discovery. Traditional data quality and FAIR (Findable, Accessible, Interoperable, Reusable) assessment frameworks insufficiently address these AI-specific requirements, particularly along axes such as algorithmic compatibility and causal/interpretive depth. Existing automated tools are largely static, format-constrained, and ill-equipped to reason about dataset suitability for diverse, modern AI methodologies.

The Sci-TQA² Principles

SciHorizon-DataEVA formalizes AI-readiness through the Sci-TQA² principles, comprising four orthogonal dimensions, each capturing a critical facet of dataset assessment:

• Governance Trustworthiness: Evaluates legal and ethical transparency, including provenance, proper licensing, adherence to FAIR, and scientific ethics (consent, dual-use risk).
• Data Quality: Examines domain-appropriate completeness, accuracy, uniqueness, and intra-dataset consistency beyond basic structural checks or label noise statistics.
• AI Compatibility: Focuses on the representational, statistical, and scaling properties required for effective modern ML (e.g., model adaptability, feature sufficiency, sample size, class balance). The system emphasizes identification of subtleties such as feature inadequacy for specific model families—e.g., 3D molecular representations for equivariant graph neural networks.
• Scientific Adaptability: Determines the extent to which datasets support mechanism-aware reasoning and generalization, including task generalizability, coverage of confounding factors for causality, and parameter regime diversity.

Each principle is hierarchically decomposed into atomic evaluative elements with context-dependent quantifiability.

The Sci-TQA²-Eval Architecture

The core of SciHorizon-DataEVA is Sci-TQA²-Eval, a distributed multi-agent evaluation framework orchestrated around a directed cyclic graph workflow.

Figure 1: Schematic of the Sci-TQA²-Eval architecture, highlighting the interplay of dataset profiling, metric selection, knowledge augmentation, tool-centric execution, verification, and report generation.

Two interlocking agentic modules underpin the pipeline:

1. Dataset-Aware Evaluation Specification Generation:
   • Uses a lightweight profile inspector for schema and metadata extraction without full data loading.
   • An applicability-aware metric selector auto-prunes inapplicable checks for each dataset-modality instance, embracing scalability and efficiency.
   • A knowledge-augmented planner connects dataset profiles to a domain-curated scientific knowledge base, generating element-level evaluation strategies tailored to specific evidence requirements and assessment recipes.
2. Adaptive Tool-Centric Execution Ecosystem:
   • Employs a structured library of atomic, meta-tagged tools, guided by persistent execution memory to encourage robust, reusable patterns.
   • For each activated metric, the agent first attempts tool retrieval or orchestrates code synthesis with runtime and semantic output verification; failed attempts iterate via reflection and planner feedback.
   • Outputs and diagnostics are normalized, semantically aggregated, and synthesized into interpretable, multi-dimensional AI-readiness reports.

Notably, this system shifts scientific data evaluation from static script-driven audits to closed-loop, knowledge-grounded, continually extensible agentic workflows.

Experimental Evaluation

Experiments span datasets from Astronomy, Biomedicine, Earth Science, Materials Chemistry, Physics/Engineering, and Socio-economics, collated from leading open repositories to stress-test cross-domain generalizability.

Key empirical findings:

• The agent system reliably produces multi-dimensional evaluation reports across all tested modalities, evidenced by high System Correctness and Tool Creation Success Rates.
• Governance Trustworthiness and Data Quality consistently score higher than AI Compatibility and Scientific Adaptability, indicating widespread gaps in modality-specific representational alignment and mechanism-awareness across open-science datasets.
• The Applicability-Aware Selector prunes between 10-40% of metrics per dataset, yielding significant efficiency gains by avoiding irrelevant or infeasible evaluations.

Strong claims in the results include the assertion that “datasets that satisfy traditional quality criteria may still be unsuitable for modern AI models,” with multiple empirical cases of datasets failing feature sufficiency or exhibiting critical regime gaps despite being technically and legally robust.

Implications and Forward-Looking Directions

Practically, SciHorizon-DataEVA provides an operational blueprint for institutions managing large, diverse data assets to rapidly identify AI deployment bottlenecks, prioritize remediation actions, and systematically document downstream suitability. The explicit evaluation along AI compatibility and scientific adaptability axes marks a critical paradigm shift: it compels data stewards and AI practitioners to recognize that typical FAIR or technical quality compliance does not imply effective AI-readiness.

Theoretically, the hierarchical atomic-element design supports formal benchmarking, meta-analyses of dataset gaps, and provides a modular substrate for further automating dataset curation, transfer-learning advisement, or interpretability auditing. Furthermore, the agentic, tool-centric architecture is poised to integrate with advanced LLM-based retrieval, code generation, and scientific reasoning systems, synergizing with the broader movement towards fully autonomous scientific workflows.

Opportunities for future work include:

• Expansion and formalization of the domain knowledge base to better support context-specific metric selection and explainability.
• Semi-supervised or self-supervised curriculum development for tool synthesis agents, leveraging cross-domain tool transfer.
• Integration with upstream experimental design and automated data curation agents to close the loop on scientific discovery pipelines.

Conclusion

SciHorizon-DataEVA introduces a comprehensive, agent-driven framework for scalable, interpretable, and scientifically meaningful AI-readiness evaluation of heterogeneous datasets, formalizing both theoretical principles and operational mechanisms. The architecture’s multi-agent design, rigorous criteria, and empirical coverage establish a robust foundation for next-generation, trustworthy AI-for-Science pipelines, addressing long-standing limitations in data stewardship and scientific reproducibility (2604.26645).