EO-Agents: A Three-Agent LLM Pipeline for Earth Observation Hypothesis Generation

Abstract: LLMs have recently been explored for scientific hypothesis generation, but most prior work relies on unstructured literature and free-form textual claims. We present a pipeline for Earth observation that grounds hypothesis generation directly in the NASA Earth Observation Knowledge Graph. A heterogeneous graph neural network trained on historical co-usage relations ranks candidate dataset pairings, and a three-agent LLM pipeline filters, generates, and evaluates structured research hypotheses. Applied to 1,475 NASA datasets, the system produces 160 hypotheses spanning multiple Earth-science domains, including ecohydrology, glaciology, aerosol--cloud interactions, vegetation phenology, and stratospheric chemistry. Model-predicted novel dataset pairings are rated nearly as plausible as held-out real co-usages from the literature, indicating that the pipeline surfaces scientifically coherent yet unexplored combinations. A 2*2*2 factorial experiment across GPT-5.2 and Claude Sonnet 4.6 shows that hypothesis rankings remain stable, while absolute scores depend strongly on judge identity, highlighting limitations of single-judge LLM evaluation.

• The paper introduces a three-agent LLM pipeline that grounds Earth observation hypotheses in dataset pairs from a NASA EO knowledge graph.
• It employs a heterogeneous GNN for dataset pair ranking and a factorial experiment varying agent roles to ensure robust hypothesis evaluation.
• The system evidences nearly as plausible novel hypotheses as real data pairings, emphasizing reproducibility and reliable execution in EO research.

EO-Agents: A Three-Agent LLM Pipeline for Earth Observation Hypothesis Generation

Introduction and Motivation

The EO-Agents pipeline addresses the challenge of scientific hypothesis generation in the Earth observation (EO) domain by grounding ideation directly in structured data resources. Unlike traditional LLM-based scientific ideation which predominantly retrieves and generates hypotheses from unstructured textual corpora, EO-Agents leverages the NASA Earth Observation Knowledge Graph (EO-KG) to connect hypotheses explicitly to pairs of measurement products. This approach is motivated by the combinatorial nature of EO research, where the value emerges from novel dataset pairings for joint analysis.

Figure 1: Three-panel architecture: (a) NASA EO-KG, (b) GNN-based candidate dataset ranking, (c) Three-agent LLM cascade for filtering, generating, and judging hypotheses.

Pipeline Architecture and Methodology

The system comprises four principal stages:

1. Dataset Pair Ranking via Heterogeneous GNN: The NASA EO-KG—containing 150,351 nodes and 436,203 edges—serves as the basis for candidate selection. A two-layer heterogeneous GraphSAGE GNN, integrating type-specific message passing across various edge and node types, performs supervised link prediction to identify promising, yet unobserved, dataset pairs. Notably, the model is trained with temporally split co-usage supervision, preventing label leakage across the train-test boundary.
2. Multi-Agent LLM Cascade:
   • Agent 1 (Filter): Evaluates candidate pairs for plausibility and novelty, ensuring the survivor pairs are both scientifically reasonable and non-trivial.
   • Agent 2 (Generator): Receives metadata for the selected pairs and synthesizes structured, executable research hypotheses comprising research question, testable claim, analysis design, expected findings, scientific importance, and domain.
   • Agent 3 (Judge): Ranks hypotheses on importance, tractability, and novelty. Evaluation is performed twice: blind (hypothesis text only) and contextual (with dataset metadata).
3. Full Factorial Experiment: A $2\times2\times2$ factorial experiment is conducted, varying the LLM backbone (GPT-5.2 vs. Claude Sonnet 4.6) in each agent role, and the judge is additionally subjected to the blind/contextual contrast. This yields 160 distinct hypotheses, each evaluated in four validation conditions for robust analysis of variance attribution.
4. Release of Resources: The pipeline outputs, trained models, hypotheses, and all judgment artifacts are publicly released, enabling reproducibility and further scientific study.

Dataset and Graph-Based Retrieval

The NASA EO-KG is a large, heterogeneous knowledge graph that explicitly encodes relationships among publications, datasets, instruments, and metadata constructs crucial to the EO community.

Figure 3: Node-type and edge-type structure of the NASA EO-KG, highlighting the dominance of publications and the prevalence of citation and research-area links.

The link prediction task in this structured setting is to identify dataset pairs likely to yield productive scientific analyses. High performance of the GNN is achieved by utilizing publication-derived co-usage edges, SPECTER2 embeddings for dataset nodes, and a degree-biased negative sampling strategy to enhance discrimination among candidate pairs.

Numerical Results and Comparative Evaluation

Empirical findings strongly endorse the heterogeneous GNN’s effectiveness:

• Performance on Link Prediction: Hits@10 on the full, cold-start, and cross-DAAC pools substantially outpaces baselines including popularity, structural graph heuristics, and content-based dense retrieval (SPECTER2, BGE-base). Notably, the GNN’s dot-product and MLP heads form a Pareto frontier across cold-start and cross-archive evaluation subsets.
• Output Quality: Predicted novel dataset pairs are rated by LLM judges as nearly as plausible as real co-usages from recent literature, with only a marginal gap in mean plausibility scores. This demonstrates that the graph-driven ranking does not merely recover trivial or spurious combinations but surfaces genuinely coherent, executable hypotheses.

  Figure 4: Per-judge plausibility and novelty grid for the GNN-predicted, novel dataset pairs, indicating concentration in high-plausibility cells and subtle inter-model calibration shifts.

Variance Decomposition and Evaluation Analysis

A major contribution of this work is the atomic decomposition of judgment variance across agent identity and evaluation modality via factorial design.

Figure 2: ANOVA-based factor decomposition of validator scores: judge identity dominates importance, context condition dominates tractability, and novelty variance is predominantly unexplained by agent factors.

Key findings:

• Judge Identity Dominates Absolute Scores: Approximately 25% of variance in importance scoring is attributable solely to the judge LLM, with filter and generator identity being negligible (<2%). The absolute scale of judgments, therefore, is not robust to judge choice, though rankings are.
• Contextual Metadata Lowers Tractability Ratings: The provision of dataset metadata (contextual condition) significantly lowers tractability scores, underscoring the over-optimism in text-only evaluation.
• Novelty Scores Are Weakly Discriminative: No variance source explains more than ∼10% on the novelty axis, aligning with previously observed instability in LLM-based novelty assessment.

  Figure 6: Inter-rater scatter plots for GPT and Claude judges, showing highest agreement on tractability, lowest on importance, and reduction of bias via contextualized evaluation.

Notably, hypothesis ranking is highly stable—variation across judge conditions rarely shifts a hypothesis more than two points. For practical selection scenarios, the ordering of top hypotheses is robust to backbone and prompt variations, implying absolute scores should be used with caution but ordinal rankings remain actionable.

Implications and Future Directions

Theoretical and Methodological

• Structured Retrieval Superiority in Data-Rich Domains: Grounding hypothesis ideation in a typed knowledge graph, rather than purely textual retrieval, directly aligns generation outputs with actionable, testable scientific questions in EO contexts.
• Variance Decomposition as Best Practice: The factorial analysis establishes a methodological paradigm for multi-agent LLM pipelines—variance attribution is critical, especially when evaluation is itself model-mediated and not human-judged.

Practical Deployment

• The pipeline’s hypotheses are structurally executable, directly referencing specific NASA datasets and relevant analytic constructs. This alignment with operational needs in EO facilitates downstream adoption and empirical validation.
• The instability in absolute scoring signals the necessity for ensemble or multi-judge protocols in LLM-as-scientist frameworks, paralleling findings in LLM-as-judge research in other domains (Ye et al., 2024, Wataoka et al., 2024, Yang et al., 21 May 2025).

Open Problems and Future Work

• Integration of Human Judgment: A key limitation is the lack of human expert evaluation for the hypotheses; augmenting LLM validation with domain-expert assessment could calibrate and potentially rescale the system’s outputs.
• Edge-Weight and Richer Semantic Modeling: Future GNN models may incorporate edge multiplicity or develop richer representations of dataset interdependencies.
• Prompt Engineering and Factorial Extension: Expanding the factorial to include prompt phrasings, temperature, and position/order effects would further elucidate the reliability limits of LLM-mediated scientific pipelines.

Conclusion

EO-Agents delivers a reproducible computational framework for the generation and evaluation of structured, dataset-grounded research hypotheses in observational sciences, leveraging heterogeneous GNN ranking and multi-agent LLM protocols. Its results demonstrate that knowledge-graph-centric retrieval empowers LLMs to surface executable, high-quality hypotheses at scale, while factorial analysis exposes the critical dependencies and variance sources inherent in automated scientific discovery systems. The release of code, data, and judgments furnishes a valuable substrate for both EO research and methodology in agentic, LLM-centered science.

• [EO-Agents: A Three-Agent LLM Pipeline for Earth Observation Hypothesis Generation, (2607.01584)]
• MT-Bench: Judging LLM-as-a-Judge (Zheng et al., 2023)
• Justice or prejudice? quantifying biases in LLM-as-a-judge (Ye et al., 2024)
• Self-Preference Bias in LLM-as-a-Judge (Wataoka et al., 2024)
• Any LLM Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector (Yang et al., 21 May 2025)

---