Diffusion-AC: Actor–Critic in Air Traffic

• Diffusion-AC is a diffusion-probabilistic actor–critic framework that uses reverse denoising to generate multimodal actions for safe 3D air traffic conflict resolution.
• It leverages a factorized 27-class discrete action space, dual-Q critics, and a Density-Progressive Safety Curriculum to maintain safety and improve training stability.
• Empirical results show a 94.1% success rate and significant reductions in collision rates, reinforcing its practical safety and efficiency in high-density scenarios.

Diffusion-AC is a diffusion-probabilistic Actor–Critic framework for three-dimensional air traffic Conflict Detection and Resolution (CD&R). It was introduced to address the unimodal bias, or mode collapse, of conventional Deep Reinforcement Learning policies in safety-critical traffic-management settings by modeling the policy as a value-guided reverse denoising process over a factorized discrete action space. The framework is paired with a Density-Progressive Safety Curriculum (DPSC), which increases traffic density and safety-stringency during training. In extensive simulations, Diffusion-AC achieves a $94.1\%$ success rate and reduces Near Mid-Air Collisions (NMACs) by $59\%$ in high-density traffic scenarios (li et al., 2 Sep 2025).

1. Scope and task formulation

Diffusion-AC casts en-route CD&R as a Markov Decision Process in 3D airspace. The state $s_t$ consists of ownship kinematics, the states of the nearest three intruders, and a goal-relative vector. The action $a_t$ is a triple $(\Delta \psi,\Delta v,\Delta FL)$ with $\Delta \psi\in\{-5^\circ,0,+5^\circ\}$, $\Delta v\in\{-50,0,+50\}\,\mathrm{kt}$, and $\Delta FL\in\{-1,0,+1\}$, yielding a 27-class joint action set. Safety constraints include flight-dynamics bounds and separation minima of $10\,\mathrm{km}$ for Loss-of-Separation and $0.2\,\mathrm{km}$ for NMAC (li et al., 2 Sep 2025).

The reward is decomposed into efficiency and safety terms,

$59\%$0

The per-step efficiency component is

$59\%$1

while the safety term is

$59\%$2

Here $59\%$3 if $59\%$4 and $59\%$5, and $59\%$6 if $59\%$7 and $59\%$8. The kinematic update uses

$59\%$9

together with

$s_t$0

This formulation is notable because it encodes multiple valid resolution modes directly into the action representation. Conventional policies with Gaussian or categorical heads are described as converging toward a single mode; Diffusion-AC instead treats action selection as structured generation over discrete maneuver combinations.

2. Diffusion-probabilistic policy

The policy $s_t$1 is represented in logit space $s_t$2. A forward noising process corrupts clean logits $s_t$3 according to a linear schedule $s_t$4, with $s_t$5 and $s_t$6: $s_t$7 A denoiser $s_t$8 predicts clean logits, and the reverse kernel is parameterized as

$s_t$9

(li et al., 2 Sep 2025).

Value guidance is introduced through a soft teacher distribution over feasible actions,

$a_t$0

where $a_t$1, $a_t$2 is a temperature, and $a_t$3 masks out envelope-violating classes. The clean logits used for training are

$a_t$4

The diffusion objective is

$a_t$5

optionally augmented by

$a_t$6

so that the total actor loss is

$a_t$7

A recurrent misconception about diffusion policies is that they are inherently tied to continuous action generation. Diffusion-AC is an explicit counterexample: the stochastic process is continuous in logit space, but the decision domain is the 27-class discrete joint action set. The final action is obtained only after denoising, masking, softmax evaluation, and discrete selection.

3. Actor–critic implementation and decision process

The actor receives a state embedding $a_t$8, noisy logits $a_t$9, and a sinusoidal timestep embedding. Its backbone is a U-Net-style denoiser with residual blocks arranged as $(\Delta \psi,\Delta v,\Delta FL)$0 channels and 8-head self-attention at the bottleneck. The critics $(\Delta \psi,\Delta v,\Delta FL)$1 and $(\Delta \psi,\Delta v,\Delta FL)$2 are separate 3-layer MLPs with hidden dimension 128, each taking $(\Delta \psi,\Delta v,\Delta FL)$3 as input and outputting $(\Delta \psi,\Delta v,\Delta FL)$4 for $(\Delta \psi,\Delta v,\Delta FL)$5 (li et al., 2 Sep 2025).

At inference time, the procedure initializes $(\Delta \psi,\Delta v,\Delta FL)$6, iterates the reverse diffusion process from $(\Delta \psi,\Delta v,\Delta FL)$7 to $(\Delta \psi,\Delta v,\Delta FL)$8, applies a masked softmax over $(\Delta \psi,\Delta v,\Delta FL)$9, and selects $\Delta \psi\in\{-5^\circ,0,+5^\circ\}$0. The mask enforces flight envelope and procedural rules upstream. This design is central to the framework’s safety posture: feasibility is not left entirely to learned behavior, but is partially enforced by structural constraints.

Training uses a replay buffer of size $\Delta \psi\in\{-5^\circ,0,+5^\circ\}$1, batch size 128, and one gradient update per 4 environment steps. The optimizer is Adam with $\Delta \psi\in\{-5^\circ,0,+5^\circ\}$2 and $\Delta \psi\in\{-5^\circ,0,+5^\circ\}$3; the discount is $\Delta \psi\in\{-5^\circ,0,+5^\circ\}$4, and target networks are updated with $\Delta \psi\in\{-5^\circ,0,+5^\circ\}$5. The diffusion model uses $\Delta \psi\in\{-5^\circ,0,+5^\circ\}$6 steps with a linear $\Delta \psi\in\{-5^\circ,0,+5^\circ\}$7 schedule from $\Delta \psi\in\{-5^\circ,0,+5^\circ\}$8 to $\Delta \psi\in\{-5^\circ,0,+5^\circ\}$9; teacher temperature is chosen from $\Delta v\in\{-50,0,+50\}\,\mathrm{kt}$0, smoothing uses $\Delta v\in\{-50,0,+50\}\,\mathrm{kt}$1, and $\Delta v\in\{-50,0,+50\}\,\mathrm{kt}$2.

4. Density-Progressive Safety Curriculum

DPSC is a 12-stage curriculum designed to stabilize learning as the environment shifts from sparse to dense traffic. At stage $\Delta v\in\{-50,0,+50\}\,\mathrm{kt}$3, the environment uses $\Delta v\in\{-50,0,+50\}\,\mathrm{kt}$4 routes and $\Delta v\in\{-50,0,+50\}\,\mathrm{kt}$5 intruders, while the Loss-of-Separation threshold is scheduled as

$\Delta v\in\{-50,0,+50\}\,\mathrm{kt}$6

rising from $\Delta v\in\{-50,0,+50\}\,\mathrm{kt}$7 to $\Delta v\in\{-50,0,+50\}\,\mathrm{kt}$8. The NMAC threshold remains fixed at $\Delta v\in\{-50,0,+50\}\,\mathrm{kt}$9. The policy head and the feasibility mask are kept fixed across stages (li et al., 2 Sep 2025).

Promotion is based on a rolling success statistic. After each episode, $\Delta FL\in\{-1,0,+1\}$0 if the aircraft reaches the goal without LoS or NMAC; otherwise $\Delta FL\in\{-1,0,+1\}$1. Let $\Delta FL\in\{-1,0,+1\}$2 denote the average of $\Delta FL\in\{-1,0,+1\}$3 over the last 100 episodes at stage $\Delta FL\in\{-1,0,+1\}$4. Advancement occurs when $\Delta FL\in\{-1,0,+1\}$5. The training loop alternates between environment interaction, replay-buffer storage, periodic Diffusion-AC updates, and target-network updates.

The curriculum has a specific methodological role. Rather than presenting the full high-density safety-constrained problem from the outset, it increases both traffic density and safety strictness in tandem. This organization is directly tied to the framework’s reported stability and efficiency in training.

5. Empirical performance and ablations

The evaluation environment uses a $\Delta FL\in\{-1,0,+1\}$6 unit airspace, approximately $\Delta FL\in\{-1,0,+1\}$7, with 3 flight levels spaced by $\Delta FL\in\{-1,0,+1\}$8, 12 fixed crossing airways, a time step of approximately $\Delta FL\in\{-1,0,+1\}$9, and a maximum of 1000 steps per episode. Initial separation is at least 100 units, or $10\,\mathrm{km}$0. Reported metrics are Success Rate, LoS Rate, NMAC Rate, Timeout Rate, and Average Steps to goal (li et al., 2 Sep 2025).

Under high-density inference over 1,000 episodes with 48 intruders, Diffusion-AC is reported as follows:

Method	Success Rate	NMAC Rate
Diffusion-AC	94.1%	1.2%
PPO	93.3%	6.4%
Rainbow	90.3%	5.4%
Safe-DQN-X	91.2%	5.8%
TD3	$10\,\mathrm{km}$1	—

In the same evaluation, Diffusion-AC records $10\,\mathrm{km}$2 LoS, $10\,\mathrm{km}$3 Timeout, and 302 average steps. PPO records $10\,\mathrm{km}$4 LoS, $10\,\mathrm{km}$5 Timeout, and 310 average steps; Rainbow records $10\,\mathrm{km}$6 LoS, $10\,\mathrm{km}$7 Timeout, and 368 average steps; Safe-DQN-X records $10\,\mathrm{km}$8 LoS, $10\,\mathrm{km}$9 Timeout, and 383 average steps; TD3 reports Timeouts of approximately $0.2\,\mathrm{km}$0 and average steps greater than 900.

Ablation studies isolate the contribution of each major component. Removing the diffusion policy reduces Success to $0.2\,\mathrm{km}$1 and increases NMAC to $0.2\,\mathrm{km}$2. Removing dual-Q critics reduces Success to $0.2\,\mathrm{km}$3 and increases NMAC to $0.2\,\mathrm{km}$4. Removing DPSC reduces Success to $0.2\,\mathrm{km}$5 and increases Timeout to $0.2\,\mathrm{km}$6. Removing value guidance is described as catastrophic, with Success dropping to $0.2\,\mathrm{km}$7 and NMAC rising to $0.2\,\mathrm{km}$8. These results support the paper’s claim that multimodal diffusion policy learning, conservative dual-Q evaluation, value guidance, and progressive curriculum are all essential to robust performance.

6. Interpretation, limitations, and future directions

Diffusion-AC is distinguished from conventional DRL by its refusal to collapse the action policy into a single optimal maneuver. In the paper’s formulation, the policy is a reverse denoising process guided by a value function, producing a multimodal action distribution over feasible heading, speed, and flight-level adjustments (li et al., 2 Sep 2025).

This design clarifies two points that are often blurred in discussions of diffusion-based control. First, multimodality here is operational rather than merely descriptive: the policy can switch among multiple effective alternatives when a single mode would lead to a decision deadlock. Second, the diffusion model does not replace safety logic wholesale; it operates together with feasibility masking, explicit separation thresholds, and critic guidance.

The reported limitations are also specific. Single-step decision latency is approximately $0.2\,\mathrm{km}$9, which is higher than that of lightweight baselines. The authors identify several directions for extension: 4D trajectory-based operations with uncertain intruder intent and communication delays, decentralized multi-aircraft coordination, and the integration of reachability analysis or barrier certificates for formal safety guarantees. These limitations place Diffusion-AC within a broader line of work on expressive but computationally heavier policies for safety-critical decision making.