Deep Hierarchical Cycle Planner (DHCP)

• The paper introduces a hierarchical DRL framework that allocates fixed signal cycle times predictably while adapting continuous green time splits to demand.
• It decomposes control across two levels, using a high-level agent for directional cycle allocation and low-level agents for splitting time between straight and left-turn movements with DDPG.
• Empirical evaluations on real-world and synthetic networks show that DHCP reduces travel times and congestion, outperforming traditional and other DRL-based traffic control methods.

Searching arXiv for the DHCP paper and closely related context. Deep Hierarchical Cycle Planner (DHCP) is a hierarchical deep reinforcement learning framework for traffic signal control with predictable cycle planning. It was proposed to address a deployment-oriented limitation of prevailing DRL-based traffic signal control methods: the tension between adaptivity to traffic demand and the practical requirement that phase sequences remain predictable, fair, and compatible with traffic engineering practice. In DHCP, a high-level agent allocates the total signal cycle duration between the North-South (NS) and East-West (EW) directions, and low-level agents then divide each directional allocation between straight and left-turn movements. Both levels use Deep Deterministic Policy Gradient (DDPG), enabling continuous time allocation rather than discrete phase selection (Gu et al., 3 Sep 2025).

1. Definition and problem setting

DHCP was introduced in the context of DRL-based traffic signal control (TSC), where two primary control paradigms were identified: “choose phase” and “switch” strategies. In the choose phase paradigm, the agent selects the next active phase adaptively, but this may result in unexpected phase sequences for drivers, disrupting anticipation and potentially compromising safety at intersections. In the switch paradigm, the agent decides whether to switch to the next predefined phase or extend the current phase; this preserves a more predictable order, but can lead to unfair and inefficient phase allocations because some movements may be extended disproportionately while others are neglected (Gu et al., 3 Sep 2025).

Against that background, DHCP formulates TSC as hierarchical, cycle-based green time allocation. Its central design choice is to fix both the overall cycle length and the phase order while learning adaptive splits within that fixed structure. The framework therefore targets a specific operational niche: it is not a free-form phase selector, but a planner for allocating a predetermined cycle in a demand-responsive manner.

A concise way to characterize DHCP is that it combines three properties in a single design: predictable cycle structure, hierarchical division of control responsibility, and continuous allocation of green times. This distinguishes it from flat DRL controllers that directly choose phases or switching decisions.

2. Hierarchical architecture and control decomposition

DHCP uses a two-level hierarchical DRL architecture. The high-level agent operates on the aggregate directional competition between NS and EW flows, while the low-level agents resolve the within-direction competition between straight and left-turn movements. This decomposition mirrors the structure of four-phase intersection control under a fixed phase order.

The high-level agent receives as state the vehicle wave and queue length for all incoming lanes,

$$s_h = \{ wave_m, queue_m ~|~ m \in \mathcal{L}^{in} \}.$$

Its action is a continuous score in $[-1,1]$, transformed to a proportion $\rho_{NS} \in [0,1]$, interpreted as the percentage of the total cycle assigned to the NS direction. Given total cycle duration $D_{total}$ and minimum duration $D_{min}$ per phase, the directional durations are

$$D_{NS} = 2 D_{min} + \rho_{NS} \times (D_{total} - 4 D_{min}),$$

$$D_{EW} = 2 D_{min} + (1 - \rho_{NS}) \times (D_{total} - 4 D_{min}).$$

The low-level layer consists of one agent for each major direction, NS and EW, with shared parameters across directions. For NS, the state is

$$s_{l, NS} = \{ wave_m, queue_m ~|~ m \in \mathcal{L}^{in}_{NS}\} \cup \{D_{NS}\},$$

and analogously for EW,

$$s_{l, EW} = \{ wave_m, queue_m ~|~ m \in \mathcal{L}^{in}_{EW}\} \cup \{D_{EW}\}.$$

Each low-level agent outputs a continuous score in $[-1,1]$, mapped to a split ratio between straight and left-turn movements. For NS,

$[-1,1]$0

$[-1,1]$1

with analogous equations for EW (Gu et al., 3 Sep 2025).

This architecture ensures that all four movements are served within every cycle while retaining flexibility in how much time each receives. A plausible implication is that the decomposition embeds a fairness prior directly into the action space rather than treating fairness as an auxiliary regularizer.

3. Reinforcement learning formulation

Both levels of DHCP use DDPG. The actor-critic objective is given as

$[-1,1]$2

The actor update is

$[-1,1]$3

and the critic update is

$[-1,1]$4

The reward design follows the hierarchical decomposition. The high-level reward is the negative sum of queue lengths on all incoming lanes,

$[-1,1]$5

For the low-level agents, the rewards are direction-specific negative queue sums: $[-1,1]$6

$[-1,1]$7

Training follows a Centralized Learning and Decentralized Execution (CLDE) paradigm for scalability, while low-level parameter sharing is used across directions. At each episode, the agents jointly determine phase splits for the next cycle; one example given is a 60 s total cycle with 5 s minimum per phase and fixed sequence NSS, NSL, EWS, EWL (Gu et al., 3 Sep 2025).

The use of continuous actions is central to the method. Rather than selecting among a small set of hand-defined phase durations, DHCP learns fine-grained allocations. This is the technical reason DDPG is used at both levels: the action spaces are continuous proportions, not categorical phase labels.

4. Predictable cycle planning and relation to conventional paradigms

DHCP’s defining operational concept is predictable cycle planning. In contrast to choose-phase methods, it does not allow arbitrary next-phase selection. In contrast to switch methods, it does not allow indefinite extension of the current phase. Instead, it fixes both cycle length and phase order, then learns how to distribute available green time across the phases within that cycle (Gu et al., 3 Sep 2025).

The distinction can be summarized as follows:

Paradigm	Control choice	Main drawback or property
Choose phase	Select any next phase	Unpredictable phase sequences and cycle lengths
Switch	Decide whether to switch or extend	Variable cycle lengths; unfair or unresponsive allocations
DHCP	Allocate a fixed cycle hierarchically	Predictable, adaptive, and continuous allocation

Within this design, DHCP is intended to preserve driver predictability and safety expectations while still responding to demand. The paper explicitly frames this as bridging the gap between DRL’s adaptability and the practical requirements of traffic engineering, including cycle predictability, fair phase serving, and safety (Gu et al., 3 Sep 2025).

A common misconception is to treat DHCP as merely another hierarchical DRL controller. That description is incomplete. Hierarchy is necessary to the method, but its distinguishing property is the combination of hierarchy with a fixed cycle structure and fixed phase order. Another potential source of confusion is acronym overlap: “DHCP” is also used in the literature for the Directed Hamiltonian Cycle Problem, a combinatorial optimization problem unrelated to traffic signal control (Jäger et al., 2014).

5. Empirical evaluation

DHCP was evaluated on real-world networks, specifically Jinan and Hangzhou, and on synthetic networks, using multiple sets of real and synthetic traffic flows. Baselines included Fixed-Time, SOLT, MaxPressure, DQN, Dueling-DQN, A2C, and CoLight. The reported metrics were average summed episodic reward, reflecting congestion reduction, and average travel time, the primary traffic efficiency metric (Gu et al., 3 Sep 2025).

The principal quantitative result reported in the data is that DHCP achieved the shortest average travel time across all scenarios listed. The values in seconds were:

Model	JinanFlow1	JinanFlow2	JinanFlow3	Hangzhou(Flat)	Hangzhou(Peak)	Synthetic
Fixed-Time	713.72	695.10	660.93	699.74	972.16	847.05
MaxPressure	297.08	288.02	300.96	332.76	446.94	253.18
CoLight	296.99	290.09	282.72	331.34	448.74	240.91
DHCP	292.15	287.27	280.27	327.30	394.37	221.74

The training curves were described as showing that DHCP converges to superior performance after an initial joint learning period, indicating effective mutual adaptation between the hierarchical levels. The framework was also described as scalable because of low-level parameter sharing and CLDE-based multi-intersection deployment support (Gu et al., 3 Sep 2025).

These findings position DHCP as a method whose benefits are not restricted to a single dataset type. The evaluations cover both real and synthetic road networks and both real and synthetic traffic flows, which suggests robustness to network and demand variation, although the exact generalization mechanism beyond the stated architecture is not separately isolated in the provided data.

6. Position within hierarchical planning research

DHCP belongs to a broader class of deep hierarchical planners that decompose long-horizon control into higher-level strategic decisions and lower-level execution decisions. In DHCP, this decomposition occurs over traffic signal cycle allocation. In other domains, related hierarchical decompositions have been used for scenario-specific neural planning in safety-critical driving, long-horizon latent-space planning in robotics, and exploration-navigation decomposition in mobile robotics.

For example, a physics-aware safety-assured hierarchical planner separates a system-level behavior planner from multiple scenario-specific motion planners, with formal verification based on reachable-set overapproximation and partition/union techniques (Liu et al., 2022). In long-horizon robotic planning, HDFlow uses a high-level diffusion planner for latent subgoal generation and a low-level rectified flow planner for dense trajectory generation, explicitly contrasting its design with earlier hierarchical planners such as SHD, HDMI, and DHCP that use diffusion at both high and low levels (Gireesh et al., 6 May 2026). In unknown-environment mobile robotics, HDPlanner decomposes long-term objectives into beacon selection and local waypoint planning through hierarchical attention networks (Liang et al., 2024).

These comparisons do not make DHCP a direct precursor of all such methods, since the domains, state representations, and learning objectives differ substantially. What they show is that DHCP participates in a recurring design pattern in contemporary learning-based planning: hierarchical decomposition is used to constrain decision spaces, separate timescales, and improve tractability. In DHCP, the decomposition is especially tightly aligned with a pre-existing engineering artifact, namely the fixed signal cycle.

7. Significance, limitations, and interpretive context

The significance attributed to DHCP in the source material is that it advances deployable intelligent urban traffic management by combining cycle-based planning with multi-level policy learning. The framework is described as improving practical deployment readiness through predictable plans, fine responsiveness through hierarchical continuous allocation, and superior performance relative to both traditional and DRL baselines under real and synthetic traffic patterns (Gu et al., 3 Sep 2025).

Its practical orientation is inseparable from its constraints. By fixing total cycle time and phase order, DHCP narrows the policy class relative to more flexible DRL approaches. The paper presents this not as a deficiency but as a deliberate design trade-off: predictability and fairness are enforced structurally, while adaptivity is preserved through continuous intra-cycle allocation. This suggests that DHCP is best understood not as a universal TSC solution, but as a framework optimized for settings where operational regularity and engineering compatibility are first-order requirements.

A further interpretive point concerns the term “cycle” in the method’s name. In DHCP, “cycle” refers to traffic signal cycle planning, not to graph cycles or motion-planning cycles. The same acronym appears in unrelated literatures, including the Directed Hamiltonian Cycle Problem (Jäger et al., 2014). For precision, DHCP in traffic signal control denotes the DDPG-based hierarchical cycle-duration allocator introduced for predictable TSC (Gu et al., 3 Sep 2025).