Compass: Navigating Global Marine Lead Data Integration through Expert-Guided LLM Agent

Abstract: Marine lead (Pb) and its isotopes are critical tracers for ocean circulation and anthropogenic pollution, yet in-situ observations remain costly and sparse. While vast historical records exist, they lie buried within the unstructured content of academic papers, creating "data silos" inaccessible to comprehensive analysis. Manual extraction is unscalable, while general-purpose LLMs lack the necessary domain-specific knowledge, leading to hallucinations and scientifically invalid outputs. To address this, we introduce an expert-guided adaptation approach that enables LLMs to perform rigorous scientific data extraction without fine-tuning. We operationalize this approach through Compass, an LLM agent framework enhanced by a Knowledge Tree co-designed with marine scientists, which decomposes complex tasks into verifiable steps, guiding the agent's reasoning to ensure scientific validity. Deploying Compass across a corpus of over 230,000 relevant open-access papers, we successfully extract 3,751 previously unincorporated Pb records. This effort establishes the largest integrated marine Pb database to date. Beyond standard metrics, Compass demonstrates superior reliability through multi-layered validation, achieving 92% accuracy as confirmed through expert manual verification. The newly integrated data expand coverage in previously under-sampled regions such as the East China Sea and the Southern Ocean, providing an enriched data foundation for future scientific discoveries. We release an interactive visualization platform to facilitate open scientific access. Our work demonstrates that expert-guided agents can effectively bridge the gap between general-purpose LLMs and high-stakes scientific domains, enabling scalable data discovery in geosciences.

• The paper introduces Compass, an expert-designed Knowledge Tree and LLM-agent pipeline that classifies papers and tables, extracts marine lead records, normalizes headers and units, and validates outputs with physical constraints.
• Compass achieves a benchmark extraction F1 of 0.465, rising to 0.619 with validation-driven rollback, while outperforming larger general-purpose and domain-fine-tuned models on precision-focused database construction.
• The system processes more than 230,000 papers to add 3,751 records, creating a 35,563-record marine lead database with 92% expert-validated accuracy and improved coverage of under-sampled ocean regions.

Motivation and problem statement

Marine lead (Pb) and its isotopes are among the most informative tracers of ocean circulation and anthropogenic pollution, but in-situ measurements are expensive: they require dedicated oceanographic cruises and strict trace-metal-clean sampling protocols. Decades of published observations remain locked inside the unstructured text, tables, and figures of academic papers—a "data silo" problem that manual curation programs such as GEOTRACES cannot scale against. The authors formalize integration as constructing a unified dataset $D^*$ from heterogeneous sources via a pipeline of collection ($\mathcal{C}$), extraction ($\mathcal{E}$), and aggregation ($\mathcal{A}$) functions. Two barriers dominate: general-purpose LLMs lack the domain knowledge and logical constraints needed for scientifically valid extraction, and the records themselves are structurally heterogeneous—dissolved versus particulate phases, multiple isotope ratio conventions, inconsistent units, depth references, and coordinate formats.

The Knowledge Tree and expert-guided adaptation

Compass's central design choice is to avoid fine-tuning entirely. The paper argues that fine-tuning improves domain vocabulary but degrades instruction-following ability; RAG injects unstructured fragments without enforcing logical constraints; and ontologies capture declarative rather than procedural knowledge. Instead, Compass encodes expertise as a Knowledge Tree $T = (N, H, K)$ co-designed with marine scientists, where each node carries four knowledge dimensions: background knowledge ($BK$), logical constraints ($LC$), operational guidelines ($OG$), and validation criteria ($VC$). Each node is compiled into a domain-aware prompt governing one agent component.

The marine Pb tree (~20 nodes) was constructed in 6–7 hours by two marine scientists working with AI researchers—a modest one-time cost that the paper positions as a practical alternative to domain-specific model training. Agent execution follows a three-phase pipeline with five components: paper classification (collection), table classification using MinerU PDF parsing plus contextual caption analysis (extraction), and data association, header normalization, and unit conversion via LLM-generated functions (aggregation). A validation mechanism performs automated physical constraint checks—value ranges, unit cross-validation, geographic boundary filtering—with rollback and re-processing on failure.

Benchmark results

Evaluation uses a 10-category benchmark of 337 tables (63 positive) yielding 1,397 data points, comparing Compass (Qwen2.5-8B/32B backbones) against GPT-4o, Gemini-2.5-pro, K2, OceanGPT, Llama-3.1-8B, Qwen3-8B, and Qwen2.5-32B under identical prompts.

Model	Paper Cls. F1	Table Cls. F1	Extraction F1
GPT-4o	0.845	0.688	0.373
Gemini-2.5-pro	0.913	0.681	0.404
OceanGPT (7B)	0.748	0.385	0.009
K2 (7B)	0.037	0.000	0.000
Qwen2.5-32B (vanilla)	0.887	0.684	0.353
Compass (32B)	0.956	0.865	0.465
Compass + rollback	0.956	0.865	0.619

Three findings stand out. First, Compass exceeds 90% accuracy on both classification tasks and surpasses both frontier proprietary models on end-to-end extraction, with Gemini's higher recall (0.511 vs. 0.429) offset by substantially lower precision (0.334 vs. 0.508)—a tradeoff the authors argue is unacceptable for database construction, where data integrity is paramount. Second, the fine-tuned domain models fail badly: K2 is crippled by its 2,048-token context limit, and OceanGPT achieves near-zero extraction F1 despite ocean-science training, supporting the claim that fine-tuning degrades the multi-step instruction-following this task requires. Third, ablations attribute gains to both tree structure (F1 drops to 0.402 without it) and knowledge nodes (0.381 without them); activating validation-driven rollback raises extraction F1 from 0.465 to 0.619 (precision 0.750), demonstrating that the $VC$ dimension contributes materially to reliability. Rollback numbers are reported separately because baselines were run single-pass for fairness—an honest caveat about comparability.

Deployment at scale

Deployed with the Qwen2.5-32B backbone over more than 230,000 open-access papers (~52 GPU-hours on dual RTX 3090s, claimed as over four orders of magnitude faster than manual curation), Compass identified 110 relevant papers and extracted 3,751 previously unincorporated records. Combined with GEOTRACES (19,108 records) and public datasets (12,704), the integrated database totals 35,563 marine Pb records across eight measurement types—an 86% increase over GEOTRACES alone, which the authors state is the largest integrated marine Pb database to date. Coverage expands notably in the East China Sea, Arabian Sea, and Southern Ocean, regions with sparse historical sampling.

Quality assurance combines automated checks (100% pass rates on value-range and geographic constraints; 95% on unit conversions; rollback triggered in ~0.06% of cases) with expert manual validation of a stratified 20% sample of papers (22 papers, 946 data points): 92% accuracy (95% CI ±1.7%), with errors attributed mainly to semantic confusion (~56%, e.g., rainwater misclassified as seawater), data association mistakes (~34%), and upstream PDF parsing failures (~10%). Average per-paper recall on target-bearing benchmark papers is 0.745, meaning roughly a quarter of extractable records within identified papers are missed—the framework prioritizes precision over completeness, consistent with its design philosophy. The gap between the 0.619 benchmark F1 and 92% deployment accuracy is explained by benchmark penalization of unextracted data and expert tolerance of minor coordinate variants; this discrepancy is acknowledged rather than hidden, though it also means the two metrics are not directly comparable.

An interactive platform (jingwei.acemap.cn/lead) provides spatial visualization, querying, and full source provenance, with over 1,590 visits reported since launch.

Limitations and open questions

The paper is explicit about scope boundaries. Extraction covers tables and text only; figures are excluded on the grounds that most marine Pb records appear tabularly, but this leaves figure-embedded data unrecovered. Quality is bounded by upstream PDF parsing tools, whose layout errors propagate through the pipeline (accounting for ~10% of observed errors). The Knowledge Tree requires expert-guided initialization, so scaling to new domains depends on recruiting domain scientists, although the authors argue accumulated structural patterns make subsequent adaptation cheaper than rebuilding. Open questions include whether the expert-guided paradigm transfers to domains where physical validation criteria are less well defined, and how much of the residual ~8% error rate—dominated by semantic confusion—can be eliminated without deeper multimodal document understanding.

Conclusion

Compass demonstrates that structured expert knowledge, injected as an executable Knowledge Tree into an LLM agent workflow, can achieve scientifically validated data extraction at literature scale without fine-tuning. Its concrete deliverable—a 35,563-record marine Pb database with 92% expert-verified accuracy on newly extracted records and expanded coverage in under-sampled oceans—is a substantive contribution to marine geochemistry, and its evaluation provides evidence that procedural knowledge encoding outperforms both prompting-only baselines and domain-fine-tuned models for high-stakes scientific integration.