C$^2$T: Captioning-Structure and LLM-Aligned Common-Sense Reward Learning for Traffic--Vehicle Coordination

Abstract: State-of-the-art (SOTA) urban traffic control increasingly employs Multi-Agent Reinforcement Learning (MARL) to coordinate Traffic Light Controllers (TLCs) and Connected Autonomous Vehicles (CAVs). However, the performance of these systems is fundamentally capped by their hand-crafted, myopic rewards (e.g., intersection pressure), which fail to capture high-level, human-centric goals like safety, flow stability, and comfort. To overcome this limitation, we introduce C2T, a novel framework that learns a common-sense coordination model from traffic-vehicle dynamics. C2T distills "common-sense" knowledge from a LLM into a learned intrinsic reward function. This new reward is then used to guide the coordination policy of a cooperative multi-intersection TLC MARL system on CityFlow-based multi-intersection benchmarks. Our framework significantly outperforms strong MARL baselines in traffic efficiency, safety, and an energy-related proxy. We further highlight C2T's flexibility in principle, allowing distinct "efficiency-focused" versus "safety-focused" policies by modifying the LLM prompt.

• The paper presents a novel dual-stage framework that integrates structured natural language captioning with LLM-based common-sense reward learning.
• It leverages interpretable scene representations to mitigate reward hacking and align reinforcement learning policies with human-level traffic laws.
• Empirical results in city-scale simulations show improved traffic throughput, reduced travel times, and enhanced policy robustness over state-of-the-art methods.

C$^2$T: Captioning-Structure and LLM-Aligned Common-Sense Reward Learning for Traffic--Vehicle Coordination

Problem Statement and Motivation

Coordinating traffic signals and vehicles for efficient, safe, and scalable urban mobility is a longstanding challenge that has been modeled using DRL, imitation learning, and, more recently, language-augmented systems. Previous approaches have been impeded by difficulties in representing multi-modal and interpretable coordination rules, reliance on opaque reward systems prone to reward hacking, and limited capacity for reasoning over high-level common-sense requirements. The paper addresses these limitations by proposing C$^2$T, a structured reward formulation and alignment framework that leverages natural language captioning, traffic scene representations, and LLM-based common-sense reward modeling to facilitate more robust coordination between intelligent traffic signals and connected autonomous vehicles.

Technical Contributions

C$^2$T introduces a two-stage framework: 1) Captioning-structure reward modeling and 2) LLM-aligned common-sense reward learning. The method first uses a structured natural language (NL) captioning model to translate traffic states and agent actions into fine-grained scene descriptions aligned with coordination rules (e.g., right-of-way, priority, stoppage). These captions form an interpretable intermediate representation that can be used for downstream RL signal construction.

Second, C$^2$T deploys an LLM-based reward judge, trained with instruction-tuned LLMs, to assess the plausibility, safety, and common-sense adherence of the joint traffic--vehicle scene as represented by the captions. The LLM acts as a sparsity-robust, hack-resilient reward function that aligns RL policy training with human-level commonsense and traffic law, moving beyond brittle, hand-crafted or environment simulator-based reward shaping prevalent in prior work such as GAIL (Ho et al., 2016), AIRL (Diamond, 2018), and DRL with human reward (Christiano et al., 2017).

The paper operationalizes these concepts in a unified multi-agent coordination setting. It creates a custom captioning scheme and integrates a judging LLM with policy optimization (such as PPO (Schulman et al., 2017)), providing step-wise dense reward supervision and scene-level critique at various granularities.

Empirical Evaluation and Results

The method is evaluated across multiple city-scale simulated traffic scenarios (e.g., CityFlow [zhang2019cityflow], RL-based signal control benchmarks [wei2018intellilight, wei2019presslight, wei2019colight, chen2020mplight]), with extensive ablation and comparison to state-of-the-art multi-agent RL, language-guided RL, and rule-based baselines. The evaluation demonstrates that C$^2$T achieves consistent improvements in traffic throughput, average travel time, and policy robustness. Notably, the LLM alignment significantly reduces reward exploitation behaviors and enhances interpretability, as evidenced by inspection of generated captions and LLM feedback. The framework generalizes well to scenarios involving both classical (signal-based) and vehicle-based (connected/autonomous) agents, outperforming conventional reward learning and language-augmented RL baselines [lai2023llmlight, guo2023cotv].

Implications and Future Work

C$^2$T evidences the utility of natural language as a structural scaffolding for RL-based multi-agent coordination, enabling transparent interface between low-level perception/action and high-level traffic laws. This formulation allows integration of external domain expertise, facilitates post-hoc safety verification, and supports incorporation of advances in LLM alignment and instruction tuning [ouyang2022instructgpt, xu2025llmjudge].

Theoretically, C$^2$T demonstrates that LLM-aligned reward modeling can mitigate reward hacking by enforcing interpretability constraints and robust generalization, which could extend to other safety-critical domains. Future developments could include online learning with interactive human feedback, scaling to real-world fleet operations, and multilingual or jurisdiction-adaptive captioning.

Conclusion

C$^2$T establishes a comprehensive pipeline for traffic--vehicle coordination by unifying captioning-based scene representation with LLM-aligned common-sense reward modeling. This approach addresses interpretability, reward alignment, and generalization challenges, yielding superior empirical performance and suggesting a promising pathway for integrating LLMs and RL in complex, real-world safety-critical systems.