- The paper introduces AgentCAT, a LangGraph-based multi-agent system that uses schema evolution, evidence-grounded extraction, expert review, and a provenance-linked knowledge graph to reconstruct synthesis–structure–performance relationships.
- The system processes about 800 energy-catalysis papers, with 82.0% of 4,398 extracted sections passing review, 2.7% receiving major-error flags, and strong expert ratings for accuracy and readability.
- AgentCAT reaches 86.67% correctness on a 60-trial natural-language query benchmark, but query latency rises to 308.32 seconds for hard questions and cross-domain generalization remains unresolved.
AgentCAT addresses the data bottleneck in chemical engineering, where catalytic reaction data reported in peer-reviewed literature remains the primary source of experimental facts. The authors argue that existing public chemistry datasets (ORD, Orderly, DigCat) represent reactions as isolated tuples, which is inadequate for chemical engineering because experimental facts are only interpretable when linked along the catalytic process — the Synthesis–Structure–Performance (SSP) lineage connecting synthesis protocols to physicochemical structures, mechanistic pathways, and macroscopic outcomes.
The paper formalizes the extraction task as reconstructing an SSP-oriented, process-coupled "partial picture" comprising mechanistic links, measurement evidence, and experimental control logic. Using H-ZSM-5-catalyzed methanol-to-olefins (MTO) as a running example, the authors show that a metric such as ethene selectivity is scientifically incomplete unless anchored to catalyst identity and synthesis-derived descriptors (e.g., Si/Al ratio), reactor mode, temporal regime, and normalization choices such as adjusting bed mass to equalize total Brønsted acid site (BAS) counts across samples.
Failure modes of general-purpose LLMs
A case study on a representative MTO paper identifies three recurring failure modes of general LLMs applied directly to this task:
- Lost-in-the-middle in causal linking: dependent information distributed across distant sections is not reconnected; e.g., Grok-3 records a BAS-unification loading adjustment but fails to link it to the ~24% conversion result it was designed to validate.
- Hallucination in fine-grained parameter alignment: protocol- and time-indexed measurements lose their context; e.g., NH3-TPD acid amounts are attributed to methanol adsorption sites, and distinct pulse protocols (11 vs. 14+4 pulses) are merged.
- Lack of domain logic constraints: physically invalid attributions occur, such as assigning a zeolite's BAS density (131 μmol/g) as the site density of transient aromatic hydrocarbon-pool species, or omitting steric confinement constraints central to mechanistic interpretation.
These observations motivate a schema-governed, closed-loop design rather than free-form extraction.
System design
AgentCAT is a multi-agent orchestration framework built on LangGraph, governed by plan-then-execute execution and closed-loop self-correction, instantiated in three stages:
Adaptive information extraction. A Human-in-the-Loop Planning mechanism seeds a preliminary schema (DomainKeyElements), which a Schema-Evolution Agent iteratively refines over stratified reference papers while maintaining backward compatibility. Extraction follows a two-phase "Candidate-then-Resolve" procedure: Phase 1 collects verbatim text fragments as evidence without interpretation; Phase 2 resolves candidates into structured fields with their original context, forcing traceable grounding. A review agent issues PASS / MINOR_FIX / MAJOR_ERROR verdicts, with MAJOR_ERROR triggering re-extraction with injected error-reason hints.
Knowledge graph construction. Structured JSON is transformed into a reaction-network graph in Neo4j, linking catalysts/active sites, synthesis-derived descriptors, mechanistic claims with supporting evidence, and macroscopic outcomes, with provenance links to source PDFs. A Label Manager supports conservative dynamic schema expansion, and a Molecular Normalizer unifies chemical nomenclature (e.g., "propylene"/"propene"/C3H6).
General querying and exploration. A natural-language querying agent decomposes complex queries into plans executed against the graph, integrated with a visualization interface that unifies mechanisms across papers on a single canvas for cross-document comparison.
Evaluation
The system is evaluated on ~800 expert-curated peer-reviewed energy catalysis publications, using Gemini-2.5-Pro for extraction and doubao-seed-1.8 for graph construction and querying.
Schema evolution exhibits bootstrapping-then-convergence: starting from an empty schema over 10 PDFs, new items per round drop sharply after the initial round, indicating rapid stabilization.
Extraction quality is assessed by three chemical engineering experts on twenty randomly selected papers along Completeness, Accuracy, and Readability (1–10 scales). Accuracy and Readability are consistently strong across nearly all cases; Completeness shows modest variation but remains robust. At scale, over 4,398 extracted sections from 733 papers, the review agent assigns 82.0% PASS, 15.3% MINOR_FIX, and only 2.7% MAJOR_ERROR. Appendix analysis shows MAJOR_ERROR cases concentrate in Catalytic Performance sections (58) versus Catalyst Identity (20) and Synthesis Procedure (1). The non-trivial MINOR_FIX fraction indicates single-run-pass acceptance remains an optimization target — the authors state this plainly rather than claiming full reliability.
Querying performance on a 12-query benchmark (three difficulty levels, five runs each, 60 trials) yields an overall correctness rate of 86.67%, with error rates of 10% (Easy/Medium) and 20% (Hard). Latency grows with difficulty: 96.36s (Easy), 149.17s (Medium), 308.32s (Hard); the modest Easy–Medium gap suggests planning rather than Cypher execution dominates latency, and even easy queries are far slower than conventional database queries.
Limitations and open questions
The paper concedes three limitations: interactive querying is time-consuming due to multi-step planning and query repair; fine-grained misalignment and cross-section linking errors may persist when evidence is implicit or distributed across modalities; and evaluation is restricted to an energy-catalysis corpus, so generalization to other chemical engineering subdomains requires further schema refinement and validation. Methodologically, the expert evaluation covers only 20 papers, JSON line count is acknowledged as a coarse proxy for information density, and no quantitative baseline comparison against direct Gemini/Grok extraction is provided beyond qualitative failure examples. Open questions include whether review-driven gating can raise single-run PASS rates substantially, and whether the SSP-oriented abstraction transfers to subdomains whose causal structure differs from zeolite catalysis.
Conclusion
AgentCAT contributes a formal, AI-friendly abstraction of catalytic reaction data extraction, a schema-governed multi-agent pipeline with progressive schema evolution and evidence-grounded two-phase extraction, a dependency-aware reaction-network knowledge graph with provenance traceability, and natural-language querying with visualization. Its evaluation on ~800 publications reports strong expert ratings, an 82% PASS rate with 2.7% major-error rate at scale, and 86.67% query correctness, while explicitly identifying latency, residual alignment errors, and domain generality as unresolved issues.