Discrete Traffic State Encoding

• DTSE is a discrete representation of traffic states, encoding vehicle occupancy, speed, and signal phases into finite spatial and velocity cells.
• It employs systematic quantization methods in both deep reinforcement learning and kinetic traffic models, balancing granularity and computational tractability.
• DTSE enhances traffic control performance by reducing delays and queue lengths and accurately reproducing traffic dynamics, underpinning adaptive signal control strategies.

A Discrete Traffic State Encoding (DTSE) is a formalism by which the state of a vehicular traffic system is represented as a collection of discrete units—either spatial, velocity, or phase variables—to enable efficient and information-dense state description for control, simulation, and analytic purposes. DTSEs have been developed and utilized in both deep reinforcement learning for traffic signal control and in kinetic theory models of vehicular traffic, providing a flexible and precise bridge between microscopic reality and macroscopic modeling. The following sections systematically detail DTSE’s foundations, mathematical definitions, construction procedures, and impact in recent research.

1. Formal Foundations of Discrete Traffic State Encoding

The core principle underlying DTSE is the partition of position and velocity space into finite, discrete sets, such that the state of traffic at any moment is fully specified by occupancy and vehicle properties in each cell and velocity class. In the reinforcement learning for traffic control context, (Genders et al., 2016) defines the instantaneous state as

$$s_t = (B_t, V_t, p_t) \in \{0,1\}^{M \times n} \times [0,1]^{M \times n} \times \{0,1\}^{|A|}$$

where:

• $B_t[i,j]$ indicates presence (1) or absence (0) of a vehicle in spatial cell $i$ of lane $j$,
• $V_t[i,j]$ is the normalized speed $v_{i,j} / v_{max}$ for occupied cells (else zero),
• $p_t$ is a one-hot phase vector representing the current traffic signal phase.

Analogously, (Fermo et al., 2012) describes DTSE in a kinetic theory framework as placing each vehicle into one of finitely many $(x_i, v_k)$ pairs, where $x_i$ marks spatial cell, and $v_k$ labels velocity class. The state space is then

$$S = \{s_{i,k} = (x_i, v_k) \mid i=1, \dots, N_x; \ k=1, \dots, N_v\}$$

and the system is described by the distribution function $f_{i,k}(t)$.

This formalization enables systemic discretization of both space and velocity, such that all dynamical, control, or analytical processes are executed on a finite-dimensional space.

2. DTSE Construction Methodologies

DTSE construction proceeds via sequential quantization and encoding steps. In the reinforcement learning setting (Genders et al., 2016), the process entails:

1. Parameter Selection: Determining the observation horizon $\ell$ and cell length $c$; $M = \lceil \ell/c \rceil$ cells per lane.
2. Lane Discretization: Partition of roadway segment into $M$ contiguous cells for each of $n$ incoming lanes.
3. Occupancy Encoding $B_t$: Each vehicle detected in $0 \leq x_v \leq \ell$ on lane $j$ is mapped to $B_t[i,j]=1$ for $i = \lceil x_v/c \rceil$.
4. Speed Encoding $V_t$: For each occupied cell, retrieve speed $v_{i,j}$ and set $V_t[i,j] = v_{i,j}/v_{max}$; else set to zero.
5. Phase Encoding $p_t$: Generate a one-hot vector specifying the current phase among the set of available actions.

In the kinetic theory approach (Fermo et al., 2012), the road is divided into cells $I_i$ of length $\ell$, and velocities are quantized into $N_v$ classes. The discrete state is thus $(x_i, v_k)$, with the population measured by $f_{i,k}(t)$.

Both frameworks rely on consistent and lossless mappings from raw simulation or measurement data to the respective discrete state representations, ensuring preservation of spatial and velocity granularity.

3. Mathematical Dynamics and Numerical Implementation

Reinforcement Learning DTSE: The DTSE $(B_t, V_t, p_t)$ forms the input to a deep CNN agent. The network processes $B_t$ and $V_t$ as separate channels, using convolutional towers with the following architecture (Genders et al., 2016):

• Conv1: 16 filters, size $4 \times 4$, stride 2, ReLU activation.
• Conv2: 32 filters, size $2 \times 2$, stride 1, ReLU activation.
• Towers are flattened, concatenated with $p_t$, then processed by two dense layers (sizes 128 and 64, ReLU).
• Final linear layer produces $|A|$ Q-values for action selection.

Kinetic Theory DTSE: Dynamic evolution is governed by a set of ODEs for $f_{i,k}(t)$: $$\frac{d f_{i,k}}{d t} + v_k \left[ \Phi_{i,i+1} f_{i,k} - \Phi_{i-1,i} f_{i-1,k} \right] = Q_{i,k}[f]$$ where $\Phi$ is a flux-limiter (enforcing local density constraints), and $Q_{i,k}[f]$ encodes acceleration, braking, and tandem interactions via probabilistic kernels $A_{h\ell}^k$. Numerical integration employs explicit Euler schemes under CFL-type conditions to ensure stability.

4. Parameter Choices and Representation Rationale

Reinforcement Learning Example Parameters (Genders et al., 2016):

• $\ell = 75\,\mathrm{m}$ (approximate span of 15 car lengths), $c = 5\,\mathrm{m}$ (average vehicle length), $M = 15$,
• $n = 4$ lanes,
• $|A| = 4$ phases,
• Input tensor: $15 \times 4 \times 2 = 120$-dimensional for $(B,V)$, plus 4-dimensional $p$.

This configuration captures individual vehicle granularity while maintaining computational tractability. Observation and quantization parameters reflect a balance between spatial/velocity resolution and input dimensionality.

Kinetic Theory Example Parameters (Fermo et al., 2012):

• $N_x = 10$ cells, $N_v = 6$ velocity classes, $\ell = 0.1$, $v_k = \frac{k-1}{5}$, $\eta_0=1$, $\beta=0.5$, time step $\Delta t = 0.01$.
• Key parameters $\alpha \in [0,1]$ (road condition), $\beta$ (anticipation).

These parameters allow macroscopic traffic patterns—such as free/congested flow phase transition at empirically realistic critical densities—to emerge from a finite, interpretable set of microstates.

5. Expressiveness and Analytical Properties

DTSE provides a rich, yet tractable, embedding of the traffic state. In the deep RL context, the DTSE outperforms low-dimensional state representations, yielding:

• 82% reduction in cumulative delay,
• 66% reduction in average queue length,
• 20% reduction in average travel time,
• Comparable throughput to baselines

(Genders et al., 2016). The scheme’s expressive power arises from explicit modeling of individual vehicle positions and velocities per cell, as opposed to summary statistics (e.g., queue length per lane).

In kinetic traffic models, discrete state encoding enables exact reproduction of granular effects—such as queue formation behind bottlenecks and stop-and-go traffic at signals—through local and pairwise state updates (Fermo et al., 2012). The emergence of fundamental diagrams (flow vs. density, speed vs. density) and phase transitions is obtained directly from the dynamics over the discrete state space.

6. Applications in Traffic Control and Modeling

DTSE has proven especially impactful in:

• Deep RL for Adaptive Traffic Signal Control: Feeding DTSE into convolutional agents controlling intersection signals in micro-simulated environments (SUMO) enables context-sensitive, adaptive policies with robust performance improvements over traditional methods (Genders et al., 2016).
• Analytically Tractable Kinetic Models: DTSE forms the phase space for ODE systems in kinetic traffic theories, supporting rigorous paper of large-scale, realistic networks while encapsulating microscopic features (Fermo et al., 2012).
• Reproducibility and Benchmarking: Because the encoding is formally defined and constructed directly from raw data or simulation, DTSE-based studies are reproducible and facilitate robust cross-comparison.

These applications collectively demonstrate that information-dense, finite-dimensional representations such as DTSE are foundational to modern approaches for modeling, controlling, and analyzing vehicular traffic systems.