Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Behavior Generation with Latent Actions (2403.03181v2)

Published 5 Mar 2024 in cs.LG, cs.AI, and cs.RO

Abstract: Generative modeling of complex behaviors from labeled datasets has been a longstanding problem in decision making. Unlike language or image generation, decision making requires modeling actions - continuous-valued vectors that are multimodal in their distribution, potentially drawn from uncurated sources, where generation errors can compound in sequential prediction. A recent class of models called Behavior Transformers (BeT) addresses this by discretizing actions using k-means clustering to capture different modes. However, k-means struggles to scale for high-dimensional action spaces or long sequences, and lacks gradient information, and thus BeT suffers in modeling long-range actions. In this work, we present Vector-Quantized Behavior Transformer (VQ-BeT), a versatile model for behavior generation that handles multimodal action prediction, conditional generation, and partial observations. VQ-BeT augments BeT by tokenizing continuous actions with a hierarchical vector quantization module. Across seven environments including simulated manipulation, autonomous driving, and robotics, VQ-BeT improves on state-of-the-art models such as BeT and Diffusion Policies. Importantly, we demonstrate VQ-BeT's improved ability to capture behavior modes while accelerating inference speed 5x over Diffusion Policies. Videos and code can be found https://sjlee.cc/vq-bet

Definition Search Book Streamline Icon: https://streamlinehq.com
References (57)
  1. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12945, 2024.
  3. On the “bang-bang” control problem. Quarterly of Applied Mathematics, 14(1):11–18, 1956.
  4. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
  5. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  6. Bushaw, D. W. Differential equations with a discontinuous forcing term. PhD thesis, Princeton University, 1952.
  7. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  11621–11631, 2020.
  8. Playfusion: Skill acquisition via diffusion from language-annotated play. In Conference on Robot Learning, pp.  2012–2029. PMLR, 2023.
  9. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
  10. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:2210.10047, 2022.
  11. Continuous control with action quantization from demonstrations. arXiv preprint arXiv:2110.10149, 2021.
  12. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341, 2020.
  13. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pp.  49–58. PMLR, 2016.
  14. Implicit behavioral cloning. In Conference on Robot Learning, pp.  158–168. PMLR, 2022.
  15. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956, 2019.
  16. Deep reinforcement learning in parameterized action space. arXiv preprint arXiv:1511.04143, 2015.
  17. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
  18. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  19. Safe local motion planning with self-supervised freespace forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  12732–12741, 2021.
  20. St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In European Conference on Computer Vision, pp.  533–549. Springer, 2022.
  21. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  17853–17862, 2023.
  22. Discrete factorial representations as an abstraction for goal conditioned reinforcement learning. arXiv preprint arXiv:2211.00247, 2022.
  23. Vad: Vectorized scene representation for efficient autonomous driving. arXiv preprint arXiv:2303.12077, 2023.
  24. Learning objective functions for manipulation. In 2013 IEEE International Conference on Robotics and Automation, pp.  1331–1336. IEEE, 2013.
  25. The design of stretch: A compact, lightweight mobile manipulator for indoor human environments. In 2022 International Conference on Robotics and Automation (ICRA), pp.  3150–3157. IEEE, 2022.
  26. Differentiable raycasting for self-supervised occupancy forecasting. In European Conference on Computer Vision, pp.  353–369. Springer, 2022.
  27. Automating reinforcement learning with example-based resets. IEEE Robotics and Automation Letters, 7(3):6606–6613, 2022.
  28. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp.  2980–2988, 2017.
  29. Action-quantized offline reinforcement learning for robotic skill learning. In Conference on Robot Learning, pp.  1348–1361. PMLR, 2023.
  30. Learning latent plans from play. In Conference on robot learning, pp.  1113–1132. PMLR, 2020.
  31. Roboturk: A crowdsourcing platform for robotic skill learning through imitation. In Conference on Robot Learning, pp.  879–893. PMLR, 2018.
  32. Gpt-driver: Learning to drive with gpt. arXiv preprint arXiv:2310.01415, 2023a.
  33. A language agent for autonomous driving. arXiv preprint arXiv:2311.10813, 2023b.
  34. Choreographer: Learning and adapting skills in imagination. arXiv preprint arXiv:2211.13350, 2022.
  35. Discrete sequential prediction of continuous actions for deep rl. arXiv preprint arXiv:1705.05035, 2017.
  36. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301.10677, 2023.
  37. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp.  188–204. PMLR, 2021.
  38. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
  39. Toward the fundamental limits of imitation learning. Advances in Neural Information Processing Systems, 33:2914–2924, 2020.
  40. Goal-conditioned imitation learning using score-based diffusion policies. arXiv preprint arXiv:2304.02532, 2023.
  41. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10684–10695, 2022.
  42. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp.  627–635. JMLR Workshop and Conference Proceedings, 2011.
  43. Behavior transformers: Cloning k𝑘kitalic_k modes with one stone. Advances in neural information processing systems, 35:22955–22968, 2022.
  44. On bringing robots home. arXiv preprint arXiv:2311.16098, 2023.
  45. Parrot: Data-driven behavioral priors for reinforcement learning. arXiv preprint arXiv:2011.10024, 2020.
  46. Learning options in reinforcement learning. In Abstraction, Reformulation, and Approximation: 5th International Symposium, SARA 2002 Kananaskis, Alberta, Canada August 2–4, 2002 Proceedings 5, pp.  212–223. Springer, 2002.
  47. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181–211, 1999.
  48. Action branching architectures for deep reinforcement learning. In Proceedings of the aaai conference on artificial intelligence, volume 32, 2018.
  49. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
  50. A review of vector quantization techniques. IEEE Potentials, 25(4):39–47, 2006.
  51. Perceive, attend, and drive: Learning spatial attention for safe self-driving. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp.  4875–4881. IEEE, 2021.
  52. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806, 2021.
  53. Maximum entropy deep inverse reinforcement learning. arXiv preprint arXiv:1507.04888, 2015.
  54. Soundstream: An end-to-end neural audio codec. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:495–507, 2021.
  55. End-to-end interpretable neural motion planner. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  8660–8669, 2019.
  56. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023.
  57. Masked audio generation using a single non-autoregressive transformer. arXiv preprint arXiv:2401.04577, 2024.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Seungjae Lee (45 papers)
  2. Yibin Wang (26 papers)
  3. Haritheja Etukuru (3 papers)
  4. H. Jin Kim (58 papers)
  5. Nur Muhammad Mahi Shafiullah (9 papers)
  6. Lerrel Pinto (81 papers)
Citations (37)
View on Semantic Scholar

Summary

Enhancing Behavior Generation through Hierarchical Vector Quantization

Introduction to Vector-Quantized Behavior Transformers

Within the landscape of behavior modeling in artificial intelligence, generating complex, multimodal actions sequences reflective of real-world decision-making stands as a formidable challenge. Where traditional methods of behavior cloning or generative modeling may stumble in capturing the intricacies and variability inherent to dynamic environments, the novel approach of Vector-Quantized Behavior Transformers (VQ-BeT) emerges as a promising solution. VQ-BeT leverages the power of hierarchical vector quantization to tokenize continuous action spaces, subsequently enabling a transformer-based architecture to model and generate nuanced action sequences. This method has demonstrated superior performance across a range of environments including simulated manipulation, autonomous driving, and real-world robotics, setting new benchmarks in the field.

Technical Overview and Methodological Contributions

The core innovation of VQ-BeT lies in its use of a hierarchical vector quantization module to discretize continuous actions, a technique inspired by advancements in generative modeling of audio and visual media. This hierarchical approach allows for the efficient capturing of multimodal action distributions, addressing the limitations of previous k-means clustering methods used in Behavior Transformers (BeT).

VQ-BeT's architecture can be divided into two primary stages:

  1. Action Discretization Phase: Continuous actions are encoded into a latent space using a hierarchical vector quantization process, which efficiently compresses the action information into discrete tokens while preserving the action sequences' variability and richness.
  2. Behavior Generation Phase: The discretized actions serve as input to a transformer-based model, which, leveraging the temporal dependencies and multimodal nature of actions, generates action sequences conditioned on observed or partial environment states.

Across seven simulated environments, including tasks from simulated manipulation to autonomous driving, VQ-BeT has demonstrated not only improved accuracy in behavior prediction but also an enhanced ability to capture multiple modes of behavior, showcasing its robustness and versatility.

Implications and Future Prospects

The adoption of VQ-BeT for behavior generation carries several practical and theoretical implications:

  • Improved Modeling of Complex Behaviors: By accurately capturing the multimodal nature of actions in diverse environments, VQ-BeT paves the way for more sophisticated models of decision-making that better reflect the variability seen in real-world behaviors.
  • Enhanced Performance in Robotics and Autonomous Systems: The ability to generate nuanced, context-aware action sequences makes VQ-BeT particularly well-suited for applications in robotics and autonomous vehicles, where adaptability and decision-making under uncertainty are crucial.
  • Future Developments in AI and Generative Modeling: The success of VQ-BeT suggests that further exploration of hierarchical vector quantization and transformer-based architectures could yield significant advances in other areas of AI, particularly in generative modeling tasks beyond behavior prediction.

In conclusion, VQ-BeT represents a significant step forward in the generative modeling of complex behaviors, offering a versatile and effective tool for capturing the dynamic, multimodal nature of real-world decision-making. As this research progresses, the potential applications and enhancements of VQ-BeT hint at an exciting future for artificial intelligence, robotics, and beyond.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube