Papers
Topics
Authors
Recent
Detailed Answer
Quick Answer
Concise responses based on abstracts only
Detailed Answer
Well-researched responses based on abstracts and relevant paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses
Gemini 2.5 Flash
Gemini 2.5 Flash 81 tok/s
Gemini 2.5 Pro 57 tok/s Pro
GPT-5 Medium 31 tok/s Pro
GPT-5 High 23 tok/s Pro
GPT-4o 104 tok/s Pro
GPT OSS 120B 460 tok/s Pro
Kimi K2 216 tok/s Pro
2000 character limit reached

DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RL (2410.04631v2)

Published 6 Oct 2024 in cs.AI and cs.LG

Abstract: Linear temporal logic (LTL) has recently been adopted as a powerful formalism for specifying complex, temporally extended tasks in multi-task reinforcement learning (RL). However, learning policies that efficiently satisfy arbitrary specifications not observed during training remains a challenging problem. Existing approaches suffer from several shortcomings: they are often only applicable to finite-horizon fragments of LTL, are restricted to suboptimal solutions, and do not adequately handle safety constraints. In this work, we propose a novel learning approach to address these concerns. Our method leverages the structure of B\"uchi automata, which explicitly represent the semantics of LTL specifications, to learn policies conditioned on sequences of truth assignments that lead to satisfying the desired formulae. Experiments in a variety of discrete and continuous domains demonstrate that our approach is able to zero-shot satisfy a wide range of finite- and infinite-horizon specifications, and outperforms existing methods in terms of both satisfaction probability and efficiency. Code available at: https://deep-ltl.github.io/

Definition Search Book Streamline Icon: https://streamlinehq.com
References (41)
  1. Modular Multitask Reinforcement Learning with Policy Sketches. In Proceedings of the 34th International Conference on Machine Learning, pp.  166–175. PMLR, July 2017.
  2. The Logical Options Framework. In Proceedings of the 38th International Conference on Machine Learning, pp.  307–317, July 2021.
  3. Using temporal logics to express search control knowledge for planning. Artificial Intelligence, 116(1):123–191, 2000. ISSN 0004-3702. doi: 10.1016/S0004-3702(99)00071-5.
  4. Directed Exploration in Reinforcement Learning from Linear Temporal Logic. In arXiv. arXiv, 2024. doi: 10.48550/arXiv.2408.09495.
  5. Principles of Model Checking. MIT Press, 2008. ISBN 978-0-262-02649-9.
  6. J. R. Büchi. Symposium on Decision Problems: On a Decision Method in Restricted Second Order Arithmetic. In Studies in Logic and the Foundations of Mathematics, volume 44 of Logic, Methodology and Philosophy of Science, pp.  1–11. Elsevier, January 1966. doi: 10.1016/S0049-237X(09)70564-6.
  7. Modular Deep Reinforcement Learning for Continuous Motion Planning With Temporal Logic. IEEE Robotics and Automation Letters, 6(4):7973–7980, October 2021. ISSN 2377-3766. doi: 10.1109/LRA.2021.3101544.
  8. LTL and beyond: Formal languages for reward function specification in reinforcement learning. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, pp.  6065–6073. ijcai.org, 2019. doi: 10.24963/IJCAI.2019/840.
  9. On the Properties of Neural Machine Translation: Encoder–Decoder Approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp.  103–111. Association for Computational Linguistics, 2014. doi: 10.3115/v1/W14-4012.
  10. A Survey of Robotic Language Grounding: Tradeoffs between Symbols and Embeddings, June 2024.
  11. Reinforcement learning for ltlf/ldlf goals. In arXiv, 2018.
  12. Using natural language for reward shaping in reinforcement learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI’19, pp.  2385–2391. AAAI Press, 2019. ISBN 978-0-9992411-4-1.
  13. Certified reinforcement learning with logic guidance. Artificial Intelligence, 322:103949, 2023. ISSN 0004-3702. doi: 10.1016/j.artint.2023.103949.
  14. Logically-Constrained Reinforcement Learning. arXiv, 2018. doi: 10.48550/arXiv.1801.08099.
  15. OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Research, May 2023.
  16. Compositional Reinforcement Learning from Logical Specifications. In Advances in Neural Information Processing Systems, volume 34, pp.  10026–10039, 2021.
  17. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, 2015. doi: 10.48550/arXiv.1412.6980.
  18. Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulas. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.  5604–5610, October 2020. doi: 10.1109/IROS45743.2020.9341325.
  19. Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning. In arXiv. arXiv, 2021. doi: 10.48550/arXiv.2006.08767.
  20. In a Nutshell, the Human Asked for This: Latent Goals for Following Temporal Specifications. In International Conference on Learning Representations, 2022.
  21. Grounding Complex Natural Language Commands for Temporal Tasks in Unseen Environments. In Proceedings of The 7th Conference on Robot Learning, pp.  1084–1110. PMLR, 2023.
  22. Skill Transfer for Temporal Task Specification. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp.  2535–2541, 2024. doi: 10.1109/ICRA57147.2024.10611432.
  23. Goal-Conditioned Reinforcement Learning: Problems and Solutions. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, pp.  5502–5511, July 2022. ISBN 978-1-956792-00-3. doi: 10.24963/ijcai.2022/770.
  24. A survey of reinforcement learning informed by natural language. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, pp.  6309–6317. ijcai.org, 2019. doi: 10.24963/IJCAI.2019/880.
  25. A hierarchy of temporal properties. In Proceedings of the Ninth Annual ACM Symposium on Principles of Distributed Computing, PODC ’90, pp.  377–410, New York, NY, USA, 1990. Association for Computing Machinery. ISBN 978-0-89791-404-8. doi: 10.1145/93385.93442.
  26. Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey. Journal of Machine Learning Research, 21(181):1–50, 2020. ISSN 1533-7928.
  27. Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning. In Proceedings of the 34th International Conference on Machine Learning, pp.  2661–2670. PMLR, July 2017.
  28. Data-Efficient Learning of Natural Language to Linear Temporal Logic Translators for Robot Task Specification. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp.  11554–11561, 2023. doi: 10.1109/ICRA48891.2023.10161125.
  29. Amir Pnueli. The temporal logic of programs. In 18th Annual Symposium on Foundations of Computer Science (SFCS 1977), pp.  46–57, October 1977. doi: 10.1109/SFCS.1977.32.
  30. Instructing Goal-Conditioned Reinforcement Learning Agents with Temporal Logic Objectives. In Thirty-Seventh Conference on Neural Information Processing Systems, November 2023.
  31. A learning based approach to control synthesis of Markov decision processes for linear temporal logic specifications. In 53rd IEEE Conference on Decision and Control, pp.  1091–1096, December 2014. doi: 10.1109/CDC.2014.7039527.
  32. Proximal Policy Optimization Algorithms. arXiv, (preprint), August 2017. doi: 10.48550/arXiv.1707.06347.
  33. LTL-Constrained Policy Optimization with Cycle Experience Replay. In arXiv, 2024. doi: 10.48550/arXiv.2404.11578.
  34. John J. Shynk. Probability, Random Variables, and Random Processes: Theory and Signal Processing Applications. John Wiley & Sons, 2012. ISBN 978-1-118-39395-6.
  35. Limit-Deterministic Büchi Automata for Linear Temporal Logic. In Computer Aided Verification, Lecture Notes in Computer Science, pp.  312–332, 2016. ISBN 978-3-319-41540-6. doi: 10.1007/978-3-319-41540-6˙17.
  36. Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning. Journal of Artificial Intelligence Research, 73:173–208, January 2022. ISSN 1076-9757. doi: 10.1613/jair.1.12440.
  37. LTL2Action: Generalizing LTL Instructions for Multi-Task RL. In Proceedings of the 38th International Conference on Machine Learning, pp.  10497–10508. PMLR, July 2021.
  38. Policy Optimization with Linear Temporal Logic Constraints. Advances in Neural Information Processing Systems, 35:17690–17702, December 2022.
  39. Eventual Discounting Temporal Logic Counterfactual Experience Replay. In Proceedings of the 40th International Conference on Machine Learning, pp.  35137–35150. PMLR, July 2023.
  40. Generalization of temporal logic tasks via future dependent options. Machine Learning, August 2024. ISSN 1573-0565. doi: 10.1007/s10994-024-06614-y.
  41. Deep Sets. In Advances in Neural Information Processing Systems, volume 30, 2017.
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Ai Generate Text Spark Streamline Icon: https://streamlinehq.com

Paper Prompts

Sign up for free to create and run prompts on this paper using GPT-5.

List Streamline Icon: https://streamlinehq.com Explore 10 Community Prompts
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Don't miss out on important new AI/ML research

See which papers are being discussed right now on X, Reddit, and more:

Explore Trending Papers

“Emergent Mind helps me see which AI papers have caught fire online.”

Philip

Philip

Creator, AI Explained on YouTube