---
title: 'Diffusion-AC: Actor–Critic in Air Traffic'
url: https://www.emergentmind.com/topics/diffusion-ac
type: topic
---

# Diffusion-AC: Actor–Critic in Air Traffic

Diffusion-AC is a diffusion-probabilistic Actor–Critic framework for three-dimensional air traffic Conflict Detection and Resolution (CD\&R). It was introduced to address the unimodal bias, or mode collapse, of conventional Deep Reinforcement Learning policies in safety-critical traffic-management settings by modeling the policy as a value-guided reverse denoising process over a factorized discrete action space. The framework is paired with a Density-Progressive Safety Curriculum (DPSC), which increases traffic density and safety-stringency during training. In extensive simulations, Diffusion-AC achieves a \(94.1\%\) success rate and reduces Near Mid-Air Collisions (NMACs) by \(59\%\) in high-density traffic scenarios [2509.03550].

## 1. Scope and task formulation

Diffusion-AC casts en-route CD\&R as a Markov Decision Process in 3D airspace. The state \(s_t\) consists of ownship kinematics, the states of the nearest three intruders, and a goal-relative vector. The action \(a_t\) is a triple \((\Delta \psi,\Delta v,\Delta FL)\) with \(\Delta \psi\in\{-5^\circ,0,+5^\circ\}\), \(\Delta v\in\{-50,0,+50\}\,\mathrm{kt}\), and \(\Delta FL\in\{-1,0,+1\}\), yielding a 27-class joint action set. Safety constraints include flight-dynamics bounds and separation minima of \(10\,\mathrm{km}\) for Loss-of-Separation and \(0.2\,\mathrm{km}\) for NMAC [2509.03550].

The reward is decomposed into efficiency and safety terms,
\[
R(s,a)=R_{\mathrm{eff}}(s,a)+R_{\mathrm{safe}}(s,a).
\]
The per-step efficiency component is
\[
r_{\mathrm{eff}}=1_{\mathrm{goal}}-\|p-p_{\mathrm{goal}}\|-0.5\cdot 1_{\mathrm{boundary}}-0.01,
\]
while the safety term is
\[
r_{\mathrm{safe}}=-\sum_{\text{intruders }k}\left[P_{\mathrm{NMAC}}(d,\Delta FL)+P_{\mathrm{LOS}}(d,\Delta FL)\right].
\]
Here \(P_{\mathrm{NMAC}}(d,\Delta FL)=1\) if \(\Delta FL=0\) and \(d<0.2\,\mathrm{km}\), and \(P_{\mathrm{LOS}}(d,\Delta FL)=0.5\) if \(\Delta FL=0\) and \(0.2\le d<d_{\mathrm{LOS}}\). The kinematic update uses
\[
x_{t+1}=x_t+v_t\cos\psi_t\,\Delta t,\qquad
y_{t+1}=y_t+v_t\sin\psi_t\,\Delta t,
\]
together with
\[
\psi_{t+1}=\psi_t+\Delta\psi,\qquad
v_{t+1}=\mathrm{clip}(v_t+\Delta v,v_{\min},v_{\max}),\qquad
FL_{t+1}=FL_t+\Delta FL.
\]

This formulation is notable because it encodes multiple valid resolution modes directly into the action representation. Conventional policies with Gaussian or categorical heads are described as converging toward a single mode; Diffusion-AC instead treats action selection as structured generation over discrete maneuver combinations.

## 2. Diffusion-probabilistic policy

The policy \(\pi(a\mid s)\) is represented in logit space \(y\in\mathbb{R}^{27}\). A forward noising process corrupts clean logits \(y_0\) according to a linear schedule \(\{\beta_t\}_{t=1}^T\), with \(\alpha_t=1-\beta_t\) and \(\bar\alpha_t=\prod_{i=1}^t \alpha_i\):
\[
q(y_t\mid y_{t-1})=\mathcal N(\sqrt{\alpha_t}\,y_{t-1},\beta_t I),\qquad
q(y_t\mid y_0)=\mathcal N(\sqrt{\bar\alpha_t}\,y_0,(1-\bar\alpha_t)I).
\]
A denoiser \(D_\theta(s,y_t,t)\) predicts clean logits, and the reverse kernel is parameterized as
\[
p_\theta(y_{t-1}\mid y_t,s)=\mathcal N(\mu_\theta(y_t,s,t),\tilde\beta_t I)
\]
[2509.03550].

Value guidance is introduced through a soft teacher distribution over feasible actions,
\[
p^*(k\mid s)=\mathrm{softmax}\!\left[\frac{Q_{\min}(s,k)}{\tau}\right]_{k\in\mathcal A_{\mathrm{feasible}}(s)},
\]
where \(Q_{\min}=\min(Q_1,Q_2)\), \(\tau>0\) is a temperature, and \(\mathcal A_{\mathrm{feasible}}(s)\) masks out envelope-violating classes. The clean logits used for training are
\[
y_0=\ln\!\bigl(p^*(\cdot\mid s)+\epsilon/27\bigr).
\]
The diffusion objective is
\[
L_{\mathrm{diff}}=\mathbb E_{s,k,t,\epsilon}\!\left[\|D_\theta(s,y_t,t)-y_0\|^2\right],
\]
optionally augmented by
\[
L_{\mathrm{term}}=\mathbb E_s\!\left[\mathrm{CE}\bigl(p^*(\cdot\mid s),\mathrm{softmax}(D_\theta(s,y_0,0))\bigr)\right],
\]
so that the total actor loss is
\[
L_\pi=L_{\mathrm{diff}}+\lambda_{\mathrm{CE}}L_{\mathrm{term}}.
\]

A recurrent misconception about diffusion policies is that they are inherently tied to continuous action generation. Diffusion-AC is an explicit counterexample: the stochastic process is continuous in logit space, but the decision domain is the 27-class discrete joint action set. The final action is obtained only after denoising, masking, softmax evaluation, and discrete selection.

## 3. Actor–critic implementation and decision process

The actor receives a state embedding \(h=\mathrm{Enc}(s)\), noisy logits \(y_t\), and a sinusoidal timestep embedding. Its backbone is a U-Net-style denoiser with residual blocks arranged as \(128\rightarrow256\rightarrow512\rightarrow256\rightarrow128\) channels and 8-head self-attention at the bottleneck. The critics \(Q_1\) and \(Q_2\) are separate 3-layer MLPs with hidden dimension 128, each taking \((s,k)\) as input and outputting \(Q(s,k)\) for \(k\in\{1,\dots,27\}\) [2509.03550].

At inference time, the procedure initializes \(y_T\sim\mathcal N(0,I)\), iterates the reverse diffusion process from \(t=T\) to \(1\), applies a masked softmax over \(\mathcal A_{\mathrm{feasible}}(s)\), and selects \(a=\arg\max\). The mask enforces flight envelope and procedural rules upstream. This design is central to the framework’s safety posture: feasibility is not left entirely to learned behavior, but is partially enforced by structural constraints.

Training uses a replay buffer of size \(10^6\), batch size 128, and one gradient update per 4 environment steps. The optimizer is Adam with \(lr_\pi=3\times10^{-4}\) and \(lr_Q=10^{-3}\); the discount is \(\gamma=0.99\), and target networks are updated with \(\tau_{\mathrm{polyak}}=5\times10^{-3}\). The diffusion model uses \(T=10\) steps with a linear \(\beta_t\) schedule from \(1\times10^{-4}\) to \(2\times10^{-2}\); teacher temperature is chosen from \([1.0,5.0]\), smoothing uses \(\epsilon=1\times10^{-4}\), and \(\lambda_{\mathrm{CE}}=1.0\).

## 4. Density-Progressive Safety Curriculum

DPSC is a 12-stage curriculum designed to stabilize learning as the environment shifts from sparse to dense traffic. At stage \(k\), the environment uses \(R(k)=k\) routes and \(A(k)=3k\) intruders, while the Loss-of-Separation threshold is scheduled as
\[
d_{\mathrm{LOS}}(k)=4.5+0.5\cdot(k-1)\ \mathrm{km},
\]
rising from \(4.5\) to \(10\,\mathrm{km}\). The NMAC threshold remains fixed at \(d_{\mathrm{NMAC}}=0.2\,\mathrm{km}\). The policy head and the feasibility mask are kept fixed across stages [2509.03550].

Promotion is based on a rolling success statistic. After each episode, \(G=1\) if the aircraft reaches the goal without LoS or NMAC; otherwise \(G=0\). Let \(S_k(t)\) denote the average of \(G\) over the last 100 episodes at stage \(k\). Advancement occurs when \(S_k\ge 0.90\). The training loop alternates between environment interaction, replay-buffer storage, periodic Diffusion-AC updates, and target-network updates.

The curriculum has a specific methodological role. Rather than presenting the full high-density safety-constrained problem from the outset, it increases both traffic density and safety strictness in tandem. This organization is directly tied to the framework’s reported stability and efficiency in training.

## 5. Empirical performance and ablations

The evaluation environment uses a \(2000\times2000\) unit airspace, approximately \(400\times400\,\mathrm{km}\), with 3 flight levels spaced by \(300\,\mathrm{m}\), 12 fixed crossing airways, a time step of approximately \(30\,\mathrm{s}\), and a maximum of 1000 steps per episode. Initial separation is at least 100 units, or \(20\,\mathrm{km}\). Reported metrics are Success Rate, LoS Rate, NMAC Rate, Timeout Rate, and Average Steps to goal [2509.03550].

Under high-density inference over 1,000 episodes with 48 intruders, Diffusion-AC is reported as follows:

| Method | Success Rate | NMAC Rate |
|---|---:|---:|
| Diffusion-AC | 94.1% | 1.2% |
| PPO | 93.3% | 6.4% |
| Rainbow | 90.3% | 5.4% |
| Safe-DQN-X | 91.2% | 5.8% |
| TD3 | \(\approx 4.3\%\) | — |

In the same evaluation, Diffusion-AC records \(2.8\%\) LoS, \(4.7\%\) Timeout, and 302 average steps. PPO records \(10.7\%\) LoS, \(0.3\%\) Timeout, and 310 average steps; Rainbow records \(11.3\%\) LoS, \(4.3\%\) Timeout, and 368 average steps; Safe-DQN-X records \(7.2\%\) LoS, \(4.4\%\) Timeout, and 383 average steps; TD3 reports Timeouts of approximately \(87.7\%\) and average steps greater than 900.

Ablation studies isolate the contribution of each major component. Removing the diffusion policy reduces Success to \(89.2\%\) and increases NMAC to \(6.4\%\). Removing dual-Q critics reduces Success to \(91.3\%\) and increases NMAC to \(2.1\%\). Removing DPSC reduces Success to \(87.6\%\) and increases Timeout to \(9.5\%\). Removing value guidance is described as catastrophic, with Success dropping to \(4.1\%\) and NMAC rising to \(11.2\%\). These results support the paper’s claim that multimodal diffusion policy learning, conservative dual-Q evaluation, value guidance, and progressive curriculum are all essential to robust performance.

## 6. Interpretation, limitations, and future directions

Diffusion-AC is distinguished from conventional DRL by its refusal to collapse the action policy into a single optimal maneuver. In the paper’s formulation, the policy is a reverse denoising process guided by a value function, producing a multimodal action distribution over feasible heading, speed, and flight-level adjustments [2509.03550].

This design clarifies two points that are often blurred in discussions of diffusion-based control. First, multimodality here is operational rather than merely descriptive: the policy can switch among multiple effective alternatives when a single mode would lead to a decision deadlock. Second, the diffusion model does not replace safety logic wholesale; it operates together with feasibility masking, explicit separation thresholds, and critic guidance.

The reported limitations are also specific. Single-step decision latency is approximately \(8.3\,\mathrm{ms}\), which is higher than that of lightweight baselines. The authors identify several directions for extension: 4D trajectory-based operations with uncertain intruder intent and communication delays, decentralized multi-aircraft coordination, and the integration of reachability analysis or barrier certificates for formal safety guarantees. These limitations place Diffusion-AC within a broader line of work on expressive but computationally heavier policies for safety-critical decision making.

Source: https://www.emergentmind.com/topics/diffusion-ac