Online Action-Stacking for ATC Efficiency

• Online Action-Stacking is an inference-time methodology that aggregates repeated reinforcement learning actions into compound directives mimicking human ATC commands.
• It leverages a minimal discrete action space with action-damping rewards to enhance policy stability and reduce training sample complexity.
• Empirical results in digital twin simulations demonstrate up to a 50% reduction in command frequency, boosting operational efficiency.

Online Action-Stacking is an inference-time methodology designed to bridge the gap between standard reinforcement learning (RL) formulations and operational requirements in Air Traffic Control (ATC). It constructs realistic, compound air traffic commands from bursts of primitive RL policy actions obtained during training with a minimal discrete action set. This allows for efficient and stable RL policy learning while yielding operationally relevant clearances at deployment. The method has demonstrated significant workload and sample complexity reductions in air traffic digital twin simulations (Carvell et al., 7 Jan 2026).

1. Motivation and Conceptual Overview

ATC operators typically issue concise, domain-appropriate clearances (e.g., "turn left 30°") as opposed to a high frequency of small adjustments (e.g., repeated "turn left 10°"). Standard RL pipelines for ATC, however, are constrained by action spaces comprising such small increments, leading to oscillatory agent behavior, increased command frequency, and excessive controller workload. Online Action-Stacking addresses this by introducing an inference-time procedure that aggregates consecutive, identical primitive actions into single, compound clearances, closely resembling human ATC directives in practice.

The methodology consists of two components:

• Training with a minimal discrete action space (e.g., {no-op, turn ±10°}) and an action-damping reward to incentivize action bursts rather than sporadic command issuance.
• At test time, a wrapper identifies bursts of identical primitive actions and compiles them into domain-appropriate macro-actions (e.g., "turn right 70°"), thus emulating operational ATC command granularity without retraining or modifying the underlying RL Markov Decision Process (MDP).

2. MDP Specification and Reward Structure

Online Action-Stacking is formulated in a classic MDP framework $(\mathcal{S}, \mathcal{A}, P, R, \gamma)$:

• State Spaces:
  • Lateral Navigation, Single Aircraft: $S_t = (\theta_f, \theta_{fs}, d_c, \Delta_t)$ where $\theta_f$ (relative bearing to next fix), $\theta_{fs}$ (bearing to subsequent fix), $d_c$ (signed centerline deviation in nautical miles), and $\Delta_t$ (time since last non-noop action).
  • Two-Aircraft Lateral & Avoidance: State is concatenated for both aircraft, including inter-aircraft features ($\theta_{1,2}$: relative bearing; $d_{1,2}$: lateral separation).
  • Vertical Navigation: $S_t = (d_{xfl}, FL_{sx}, FL_{nx}, \Delta_t)$, with $d_{xfl}$ indicating distance required for level change, $FL_{sx}$, and $FL_{nx}$ denoting selected and next exit flight level differences.
  • States are normalized and clipped to prespecified ranges.
• Action Spaces:

See Table 1 for a succinct summary.

| Scenario | Primitive Action Space | Example Compound Action | |------------------|----------------------------------------------------|-------------------------------| | Lateral, single | $\{\varnothing, h\text{–}, h\text{+}\}$ | "turn left 70°" | | Lateral, two-ac | $$\{\varnothing, h^1\text{–}, h^1\text{+}, h^2\text{–}, h^2\text{+}\}$$ | "Aircraft 1 turn left 30°" | | Vertical | $\{\varnothing, FL\text{–}, FL\text{+}\}$ | "climb 150 FL" |

• Transition Dynamics:

Executed within the BluebirdDT digital twin ATC environment, the dynamics are Markovian, advancing in 6 s steps via a simplified BADA model.

• Reward Function:
  • Route-keeping: $r_c(s) = \exp[-(d_c/\lambda_c)^2] - 1$, $\lambda_c = 6$ nm
  • Action-damping: $r_a(n_s)$ penalizes each new non-noop action, with decay over $n_\text{max}=10$ steps
  • Separation: For two-aircraft, $r_s(s)$ is a function of $d_{1,2}$
  • Vertical deviation: $r_v(s) = \exp[-|\Delta_{FL}|/\lambda_v] - 1$, $\lambda_v = 40$ FL
  • Terminal bonuses: Allocated for correct exit, route compliance, and command economy

3. Online Action-Stacking Procedure

At inference, the Online Action-Stacking wrapper transforms sequences of identical primitives into single, compiled macro instructions. The simulation time is frozen during this process to prevent unobserved state evolution:

Input: π(s): trained primitive-action policy s₀: environment state ac: aircraft index Procedure ONLINE_STACK(s₀, ac): cmd_sum ← 0 s ← s₀ repeat: a ← π(s) if a == φ or action_on_other_ac(a, ac): break Δ ← increment_degrees_or_FL(a) cmd_sum ← cmd_sum + Δ s ← update_state_with_new_heading(s, ac, cmd_sum) end repeat if cmd_sum == 0: return φ else: return compile_macro(ac, cmd_sum)

Key characteristics:

• No environment dynamics advance while stacking, avoiding partial observability.
• The original MDP and reward signals are unaltered; only inference is wrapped.
• Policies, under action-damping, emit bursts of non-noop primitives, which are efficiently converted into single operational clearances.

4. Training Protocols and Implementation Details

Learning is conducted using Proximal Policy Optimization (PPO, actor-critic) with the following setup:

• Learning rate: $1 \times 10^{-4}$; discount: $\gamma=0.99$; entropy regularization: $\beta=0.01$
• Network: Two hidden layers, 64 ReLU neurons each
• Episode length: 300 steps (30 min simulated time at 6 s/step)
• Data: Two million steps for single-aircraft tasks, curriculum for two-aircraft (2 M route-following, followed by 30 M with added separation rewards)
• Action set during training: 3 primitives per aircraft ($\varnothing, \pm 10^\circ$ or $\pm 10$ FL), leading to a 5-element action space for two aircraft

This low-dimensional action space expedites learning and policy stability. Action-damping, by construction, encourages temporal clustering of identical commands, anticipating the inference-time stacking wrapper.

5. Empirical Results

Evaluations leverage BluebirdDT in the X-Plus en-route sector, with metrics over 100 test episodes per scenario:

• Lateral Navigation (Two Aircraft):
  • Undamped: $113.0 \pm 15.9$ actions/episode; high oscillatory frequency.
  • Damped: $14.5 \pm 6.6$ actions; actions clustered in bursts.
  • Damped + Stacked: $7.2 \pm 3.5$ macro-commands; ≈50% fewer than damped alone.
  • Large action-space (37 actions): $6.3 \pm 3.0$ actions; slower convergence than small action space with stacking.
• Vertical Navigation (Single Aircraft):
  • Policy issues up to 15 consecutive "climb 10 FL" primitives, which stack into a single "climb 150 FL" macro-command.
• Lateral Navigation & Avoidance (Two Aircraft):
  • Damped + Stacked: $81.2 \pm 13.2$ actions; separation losses (distance < 5 nm): 1/100 episodes.
  • Large action-space: $126.3 \pm 12.3$ actions; separation losses: 9/100 episodes.
  • Demonstrates increased robustness and collision-avoidance efficiency with stacked actions in small action spaces.

6. Ablative Analysis and Methodological Insights

• Action damping ($r_a$) substantially reduces command rate (–87%), consolidating actions into bursts.
• Stacking at inference further halves the number of issued clearances, closely matching the efficiency (in clearance count) of the large action-space baseline.
• Training with a minimal action set and stacking reaches operational success (correct exit, in bounds, ≤30 actions) after ~1 million steps; the high-dimensional action baseline requires ~2 million steps.
• Stacking confers greatest benefit in scenarios involving prolonged identical increment sequences (e.g., large heading changes, extended climbs).

7. Limitations, Extensions, and Implications

Limitations:

• Current stacking operates only on consecutive, identical primitives, precluding mixed-action or temporally interleaved macro-actions.
• All compilation occurs at a single inference point; the impact of real-time environment evolution during macro-actions is not addressed.
• The method presently handles only heading and flight-level changes, omitting speed or complex routing clearances.

Potential extensions include:

• Stacking that merges mixed command sequences (e.g., lateral and vertical together or integrating speed clearances).
• Learning a library of macro-options during training for use at inference.
• Integrating stacking into hierarchical RL architectures to account for environment evolution across long-horizon macros.

Implications for ATC operations:

• Significantly reduces pilot and controller workload via lower command frequency.
• Enhances radio efficiency by minimizing transmission count per aircraft.
• Stringently maintains operational constraints (5 nm separation, route adherence).
• Facilitates scalable training in low-dimensional action spaces, freeing resources for higher traffic density or additional safety features.

Online Action-Stacking synthesizes the efficiency of minimal action-space RL policy learning with the realism of operational command structures, effectively aligning RL-based automation with real-world ATC standards and requirements (Carvell et al., 7 Jan 2026).