Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Traffic Signal Control Using Lightweight Transformers: An Offline-to-Online RL Approach (2312.07795v1)

Published 12 Dec 2023 in cs.LG and cs.AI

Abstract: Efficient traffic signal control is critical for reducing traffic congestion and improving overall transportation efficiency. The dynamic nature of traffic flow has prompted researchers to explore Reinforcement Learning (RL) for traffic signal control (TSC). Compared with traditional methods, RL-based solutions have shown preferable performance. However, the application of RL-based traffic signal controllers in the real world is limited by the low sample efficiency and high computational requirements of these solutions. In this work, we propose DTLight, a simple yet powerful lightweight Decision Transformer-based TSC method that can learn policy from easily accessible offline datasets. DTLight novelly leverages knowledge distillation to learn a lightweight controller from a well-trained larger teacher model to reduce implementation computation. Additionally, it integrates adapter modules to mitigate the expenses associated with fine-tuning, which makes DTLight practical for online adaptation with minimal computation and only a few fine-tuning steps during real deployment. Moreover, DTLight is further enhanced to be more applicable to real-world TSC problems. Extensive experiments on synthetic and real-world scenarios show that DTLight pre-trained purely on offline datasets can outperform state-of-the-art online RL-based methods in most scenarios. Experiment results also show that online fine-tuning further improves the performance of DTLight by up to 42.6% over the best online RL baseline methods. In this work, we also introduce Datasets specifically designed for TSC with offline RL (referred to as DTRL). Our datasets and code are publicly available.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Learning an interpretable traffic signal control policy. arXiv preprint arXiv:1912.11023.
  2. Reinforcement learning benchmarks for traffic signal control. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
  3. Beckmann, K. J. 2006. Integrierte Mikro-Simulation von Raum-und Verkehrsentwicklung: Theorie, Konzepte, Modelle, Praxis; Tagungsband zum 7. Aachener Kolloquium” Mobilität und Stadt”. ISB.
  4. Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 3414–3421.
  5. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 15084–15097.
  6. Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Transactions on Intelligent Transportation Systems, 21(3): 1086–1095.
  7. Traffic Signal Control Using Offline Reinforcement Learning. In 2021 China Automation Congress (CAC), 8090–8095. IEEE.
  8. Model-based graph reinforcement learning for inductive traffic signal control. arXiv preprint arXiv:2208.00659.
  9. Uncertainty-aware model-based offline reinforcement learning for automated driving. IEEE Robotics and Automation Letters, 8(2): 1167–1174.
  10. Safelight: A reinforcement learning method toward collision-free traffic signal control. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 14801–14810.
  11. Deep learning. MIT press.
  12. Benchmarking offline reinforcement learning on real-robot hardware. arXiv preprint arXiv:2307.15690.
  13. Towards a Unified View of Parameter-Efficient Transfer Learning. In International Conference on Learning Representations.
  14. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  15. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning, 2790–2799. PMLR.
  16. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  17. ModelLight: Model-Based Meta-Reinforcement Learning for Traffic Signal Control. arXiv preprint arXiv:2111.08067.
  18. Dynamic lane traffic signal control with group attention and multi-timescale reinforcement learning. IJCAI.
  19. Multi-Agent Reinforcement Learning for Traffic Signal Control through Universal Communication Method. Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence.
  20. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34: 1022–1035.
  21. Offline Reinforcement Learning for Road Traffic Control. arXiv preprint arXiv:2201.02381.
  22. A Survey on Transformers in Reinforcement Learning. arXiv preprint arXiv:2301.03044.
  23. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190.
  24. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35: 1950–1965.
  25. InTAS–The Ingolstadt Traffic Scenario for SUMO. arXiv preprint arXiv:2011.11995.
  26. Microscopic traffic simulation using sumo. In 2018 21st international conference on intelligent transportation systems (ITSC), 2575–2582. IEEE.
  27. Feudal multi-agent deep reinforcement learning for traffic signal control. In Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 816–824.
  28. Unipelt: A unified framework for parameter-efficient language model tuning. arXiv preprint arXiv:2110.07577.
  29. Melo, L. C. 2022. Transformers are meta-reinforcement learners. In International Conference on Machine Learning, 15340–15359. PMLR.
  30. Transformers are sample efficient world models. International Conference on Learning Representations.
  31. Reinforcement learning in urban network traffic signal control: A systematic literature review. Expert Systems with Applications, 116830.
  32. AdapterHub: A Framework for Adapting Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 46–54.
  33. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems.
  34. Traffic signal control using a cooperative EWMA-based multi-agent reinforcement learning. Applied Intelligence, 53(4): 4483–4498.
  35. Improving language understanding by generative pre-training.
  36. Language models are unsupervised multitask learners. OpenAI blog, 1(8): 9.
  37. Traffic engineering. Pearson/Prentice Hall.
  38. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
  39. Reinforcement learning: An introduction. MIT press.
  40. The sensory neuron as a transformer: Permutation-invariant neural networks for reinforcement learning. Advances in Neural Information Processing Systems, 34: 22574–22587.
  41. Varaiya, P. 2013. The max-pressure controller for arbitrary networks of signalized intersections. Advances in dynamic network modeling in complex transportation systems, 27–66.
  42. Attention is all you need. Advances in neural information processing systems, 30.
  43. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 38–45. Online: Association for Computational Linguistics.
  44. Efficient pressure: Improving efficiency for signalized intersections. arXiv preprint arXiv:2112.02336.
  45. Integrating public transit signal priority into max-pressure signal control: Methodology and simulation study on a downtown network. Transportation Research Part C: Emerging Technologies, 138: 103614.
  46. Yang, S. 2023. Hierarchical graph multi-agent reinforcement learning for traffic signal control. Information Sciences, 634: 55–72.
  47. Metalight: Value-based meta-reinforcement learning for traffic signal control. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 1153–1160.
  48. Expression might be enough: representing pressure and demand for reinforcement learning based traffic signal control. In International Conference on Machine Learning, 26645–26654. PMLR.
  49. Online decision transformer. In International Conference on Machine Learning, 27042–27059. PMLR.
  50. MTLight: Efficient Multi-Task Reinforcement Learning for Traffic Signal Control. In ICLR 2022 Workshop on Gamification and Multiagent Solutions.
  51. MetaVIM: Meta Variationally Intrinsic Motivated Reinforcement Learning for Decentralized Traffic Signal Control. IEEE Transactions on Knowledge and Data Engineering.
  52. Multi-agent broad reinforcement learning for intelligent traffic light control. Information Sciences, 619: 509–525.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Xingshuai Huang (4 papers)
  2. Di Wu (477 papers)
  3. Benoit Boulet (27 papers)
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now