Deep reinforcement learning uncovers processes for separating azeotropic mixtures without prior knowledge (2310.06415v1)
Abstract: Process synthesis in chemical engineering is a complex planning problem due to vast search spaces, continuous parameters and the need for generalization. Deep reinforcement learning agents, trained without prior knowledge, have shown to outperform humans in various complex planning problems in recent years. Existing work on reinforcement learning for flowsheet synthesis shows promising concepts, but focuses on narrow problems in a single chemical system, limiting its practicality. We present a general deep reinforcement learning approach for flowsheet synthesis. We demonstrate the adaptability of a single agent to the general task of separating binary azeotropic mixtures. Without prior knowledge, it learns to craft near-optimal flowsheets for multiple chemical systems, considering different feed compositions and conceptual approaches. On average, the agent can separate more than 99% of the involved materials into pure components, while autonomously learning fundamental process engineering paradigms. This highlights the agent's planning flexibility, an encouraging step toward true generality.
- R.W.H. Sargent “Advances in modelling and analysis of chemical process systems” In Comput. Chem. Eng. 7, 1983, pp. 219–237
- V. Venkatasubramanian “The Promise of Artificial Intelligence in Chemical Engineering: Is It Here, Finally?” In AIChE J. 65, 2018, pp. 225502
- J.H. Lee, J. Shin and M.J. Realff “Machine learning: Overview of the recent progresses and implications for the process systems engineering field” In Comput. Chem. Eng. 114, 2018, pp. 111–121
- “Machine Learning in Chemical Engineering: Strengths, Weaknesses, Opportunities, and Threats” In Engineering 7.9, 2021, pp. 1201–1211
- “Machine Learning in Chemical Engineering: A Perspective” In Chem. Ing. Tech. 93, 2021, pp. 2029–2039
- “Process systems engineering - The generation next?” In Comput. Chem. Eng. 147, 2021, pp. 107252
- “Analyzing Learned Molecular Representations for Property Prediction” In J. Chem. Inf. Model. 59.8, 2019, pp. 3370–3388
- “A graph-convolutional neural network model for the prediction of chemical reactivity” In Chem. Sci. 10, 2019, pp. 370–377
- “Large-scale chemical language representations capture molecular structure and properties” In Nat. Mach. Intell. 4, 2022, pp. 1256–1264
- “Mapping the space of chemical reactions using attention-based neural networks” In Nat. Mach. Intell. 3, 2021, pp. 144–152
- Z. Zhou, X. Li and R.N. Zare “Optimizing Chemical Reactions with Deep Reinforcement Learning” In ACS Cent. Sci. 3.12, 2017, pp. 1337–1344
- “Overview of Surrogate Modeling in Chemical Process Engineering” In Chem. Ing. Tech. 91.3, 2019, pp. 1–13
- J.J. Siirola “Strategic process synthesis: advances in the hierarchical approach” In Comput. Chem. Eng. 20.2, 1996, pp. 1637–1643
- A.W. Westerberg “A retrospective on design and process synthesis” In Comput. Chem. Eng. 28.4, 2004, pp. 447–458
- “From conceptual design to process design optimization: a review on flowsheet synthesis” In Oil Gas Sci. Technol. 74.80, 2019
- R.L. Kirkwood, M.H. Locke and J.M. Douglas “A prototype expert system for synthesizing chemical process flowsheets” In Comput. Chem. Eng. 12.4, 1988, pp. 329–343
- “A knowledge based system for the selection of thermodynamic models” In Comput. Chem. Eng. 13.4-5, 1989, pp. 397–404
- “A review on superstructure optimization approaches in process system engineering” In Comput. Chem. Eng. 136.106808, 2020
- “Mastering the game of Go without human knowledge” In Nature 550, 2017, pp. 354–359
- “A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play” In Science 362, 2018, pp. 1140–1144
- “Discovering faster matrix multiplication algorithms with reinforcement learning” In Nature 610.7930 Nature Publishing Group, 2022, pp. 47–53
- “Faster sorting algorithms discovered using deep reinforcement learning” In Nature 618.7964 Nature Publishing Group UK London, 2023, pp. 257–263
- L.I. Midgley “Deep Reinforcement Learning for Process Synthesis”, 2020 URL: https://doi.org/10.48550/arXiv.2009.13265
- “Searching for optimal process routes: A reinforcement learning approach” In Comput. Chem. Eng. 141, 2020, pp. 107027
- Q. Göttl, D.G. Grimm and J. Burger “Automated synthesis of steady-state continuous processes using reinforcement learning” In Front. Chem. Sci. Eng. 16, 2022, pp. 288–302
- Q. Göttl, D.G. Grimm and J. Burger “Automated Process Synthesis Using Reinforcement Learning” In Comput. Aided Chem. Eng. 50, 2021, pp. 209–214
- “Automated Flowsheet Synthesis Using Hierarchical Reinforcement Learning: Proof of Concept” In Chem. Ing. Tech. 93.12, 2021, pp. 2010–2018
- Q. Göttl, D.G. Grimm and J. Burger “Using Reinforcement Learning in a Game-like Setup for Automated Process Synthesis without Prior Process Knowledge” In Comput. Aided Chem. Eng. 49, 2022, pp. 1555–1560
- “Designing the process designer: Hierarchical reinforcement learning for optimisation-based process design” In Chem. Eng. Process. 180, 2022, pp. 108885
- J.R. Seidenberg, A.A. Khan and A.A. Lapkin “Boosting autonomous process design and intensification with formalized domain knowledge” In Comput. Chem. Eng. 169, 2023, pp. 108097
- “Flowsheet generation through hierarchical reinforcement learning and graph neural networks” In AIChE J. 69.1, 2023, pp. 17938
- “Proximal Policy Optimization Algorithms”, 2017 URL: https://doi.org/10.48550/arXiv.1707.06347
- “Transfer learning for process design with reinforcement learning” In Comput. Aided Chem. Eng. 52, 2023, pp. 2005–2010
- “Policy improvement by planning with gumbel” In ICLR, 2022
- “MLP-mixer: An all-MLP architecture for vision” In NeurIPS 34, 2021, pp. 24261–24272
- “Reinforcement learning: An introduction” MIT press, 2018
- “Attention is all you need” In NeurIPS, 2017, pp. 5998–6008
- “Unique Design Considerations for Maximum-Boiling Azeotropic Systems via Extractive Distillation: Acetone/Chloroform Separation” In Ind. Eng. Chem. Res. 57.38, 2018, pp. 12884–12894
- “Techno-economic comparison of energy usage between azeotropic distillation and hybrid system for water-ethanol separation” In Renew. Energ. 51, 2013, pp. 310–316
- W.L. Luyben “Control of the Heterogeneous Azeotropic n-Butanol/Water Distillation System” In Energ. Fuel. 22, 2008, pp. 4249–4258
- “Design and Control of a Hybrid Extraction/Distillation System for the Separation of Pyridine and Water” In Ind. Eng. Chem. 54, 2015, pp. 7715–7727
- O. Ryll, S. Blagov and H. Hasse “Convex envelope method for the determination of fluid phase diagrams” In Fluid Ph. Equilibria 324, 2012, pp. 108–116
- O. Ryll, S. Blagov and H. Hasse “Inf/Inf Analysis of homogeneous distillation processes” In Chem. Eng. Sci. 84, 2012, pp. 315–332
- “Convex Envelope Method for determining liquid multi-phase equilibria in systems with arbitrary number of components” In Comput. Chem. Eng. 177, 2023, pp. 108321
- “Learning values across many orders of magnitude” In Advances in Neural Information Processing Systems 29, 2016
- “Observe and Look Further: Achieving Consistent Performance on Atari” In arXiv:1805.11593, 2018
- “Policy-Based Self-Competition for Planning Problems” In International Conference on Learning Representations, 2023
- C.M. Bishop “Pattern Recognition and Machine Learning” New York: Springer, 2006
- O. Ryll “Thermodynamische Analyse gekoppelter Reaktions-Destillations-Prozesse: konzeptioneller Entwurf, Modellierung, Simulation und experimentelle Validierung (in German)”, 2009
- “Multiple Steady States in Distillation: Inf/Inf Predictions, Extensions, and Implications for Design, Synthesis, and Simulation” In Ind. Eng. Chem. Res. 35, 1996, pp. 4264–4280
- “Chemical Engineering Design: Principles, Practice and Economics of Plant and Process Design” Butterworth-Heinemann, 2022
- “NIST Chemistry WebBook” Gaithersburg MD, 20899: National Institute of StandardsTechnology, 2023
- DDBST GmbH “Dortmund Data Bank”, 2023 URL: www.ddbst.com
- Aspen Technology Inc. “Aspen Plus”, 2015
- “Basic Theoretical Physics” Berlin, Heidelberg: Springer, 2007
- J.I. Yellott Jr. “The relationship between Luce’s choice axiom, Thurstone’s theory of comparative judgment, and the double exponential distribution” In Journal of Mathematical Psychology 15.2 Elsevier, 1977, pp. 109–144
- T. Vieira “Gumbel-max trick and weighted reservoir sampling”, http://timvieira.github.io/blog/post/2014/08/01/gumbel-max-trick-and-weighted-reservoir-sampling/, 2014
- Z. Karnin, T. Koren and O. Somekh “Almost Optimal Exploration in Multi-Armed Bandits” In ICML, 2013
- J.M. Prausnitz, R.N. Lichtenthaler and E.G.d. Azevedo “Molecular Thermodynamics of Fluid-Phase Equilibria” Prentice Hall PTR, 1999
- F.B. Petlyuk “Distillation Theory and Its Application to Optimal Design of Separation Units” Cambridge University Press, 2004
- L.T. Biegler, I.E. Grossmann and A.W. Westerberg “Systematic Methods of Chemical Process Design” Prentice Hall PTR, 1997
- “SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python” In Nature Methods 17, 2020, pp. 261–272