Deep RL for Active Flow Control

• Deep reinforcement learning for active flow control is a technique that uses neural networks to learn optimal actuation strategies for managing fluid flow behaviors in real time.
• It integrates high-fidelity CFD simulations and policy gradient methods such as PPO and SAC to translate sensor data into reliable control commands.
• Recent applications demonstrate significant drag reduction and vortex shedding suppression, highlighting the method's potential in both 2D and 3D turbulent flow scenarios.

Deep reinforcement learning (DRL) for active flow control (AFC) refers to the use of neural network-based agents trained via reinforcement learning algorithms to autonomously discover optimal strategies for manipulating fluid flows in real time. The targeted objective in AFC is typically the reduction of aerodynamic drag, suppression of vortex-induced forces, or other modifications of the flow field, where classical control laws are insufficient due to high system dimensionality, nonlinearity, and unsteady dynamics.

1. Core Principles and Methodologies

Active flow control with DRL is predicated on representing the control law as a parameterized neural network (often deep, fully connected, or convolutional), which maps a set of fluid state observations—obtained from probes or sensors—to actuation commands, such as the mass flow rates in synthetic jets or the control signals to plasma actuators. The agent interacts with a high-fidelity computational fluid dynamics (CFD) environment that numerically solves the Navier–Stokes equations, receives feedback via a scalar “reward” function, and updates policy parameters to maximize expected returns.

Policy gradient techniques are standard; in particular, Proximal Policy Optimization (PPO), Advantage Actor-Critic (A2C), and, more recently, Soft Actor-Critic (SAC) have become central due to their robustness in continuous, high-dimensional action spaces.

The canonical DRL control pipeline for AFC:

• State $s_t$: Partial representations of the flow, e.g., probe/measured velocity or pressure values, possibly sampled at hundreds of locations.
• Action $a_t$: Controls to actuators, such as mass flow rates of synthetic jets ($Q_1$, $Q_2$), rotational speed, or burst frequency for plasma actuators.
• Reward $r_t$: A function reflecting the flow control objective; typical forms include

$$r_t = -\langle C_D \rangle_t - \beta|\langle C_L \rangle_t|$$

where angle brackets denote pseudo-period averages, $C_D$ is the drag coefficient, $C_L$ is the lift coefficient, and $\beta$ is a penalty weight (Rabault et al., 2018).

Control actions are applied either in a quasi-continuous manner (with smoothing/interpolation to avoid actuation discontinuities) or in discrete time steps synchronized to the dominant frequencies of the flow (e.g., vortex shedding).

2. Simulation Environments and Actuation Schemes

A substantial body of DRL-AFC work employs two-dimensional test cases, particularly the flow past a circular or square cylinder at a moderate Reynolds number (usually $Re = 100$–$1000$), which naturally develops Kármán vortex shedding. These configurations allow tractable but nontrivial exploration of control strategies that can suppress unsteady wakes and minimize drag.

A typical setup includes:

• Rectangular or channel computational domain.
• Bluff bodies (circular, square, or elliptical cylinders) as the main obstacle.
• Synthetic jets imposed on the body’s surface, subject to a zero-net-mass-flux constraint: $Q_1 + Q_2 = 0$.
• Actuators realized via wall boundary conditions modulated by the agent.
• Observations provided by an array of $100$–$250$ sensors placed strategically near separation or wake regions (Rabault et al., 2018, Jia et al., 18 Apr 2024).

For turbulent flow or complex three-dimensional geometries ($Re \gtrsim 1000$), high-fidelity solvers such as lattice Boltzmann methods with LES subgrid models (Ren et al., 2020) and GPU-optimized spectral element solvers (Montalà et al., 12 Sep 2025) are utilized alongside parallelized training to mitigate the extreme computational cost. Recent studies also integrate plasma actuators, windward-suction–leeward-blowing actuators, or rotary actuation for more advanced experimental and practical cases (Elhawary, 2020, Ren et al., 2020, Sababha et al., 29 Sep 2025).

3. Control Laws, Smoothing, and Reward Engineering

The effectiveness of DRL-AFC hinges on the agent’s ability to generate temporally correlated, physically valid actuation. Since naive application of the raw neural network output may lead to unphysical gradients or control noise (manifesting in high lift fluctuations or destabilization), smoothing/interpolation schemes are implemented.

Two main approaches are:

• Exponential smoothing:

$c_{s+1} = c_s + \alpha (a - c_s)$

where $c_s$ is the current actuation, $a$ is the new action, and $\alpha$ is typically $0.1$ (Rabault et al., 2018, Rabault et al., 2019).

• Linear interpolation over $N_e$ time steps:

$c_i = a_{j-1} + \frac{a_j - a_{j-1}}{N_e} \cdot n$

for $n = 1 \ldots N_e$ (Tang et al., 2020).

Reward design is critical: the reward must be informative but avoid “cheating” solutions (e.g., reducing drag at the expense of excessive lift). Penalizing both drag and the absolute value of the lift (oscillation) is a commonly used structure. For stealth or noise suppression, additional terms targeting vorticity, velocity, or sound pressure levels are applied (Ren et al., 2020, Phan et al., 2023).

4. Quantitative Performance, Robustness, and Generalization

The efficacy of DRL for AFC is consistently validated through metrics such as mean drag reduction, suppression of lift oscillations, and stabilization or elongation of the separation bubble:

$Re$	Drag Reduction (\%)	Lift Oscillation Suppression (\%)	Vortex Shedding Suppressed
100	$5.7$–$9.3$	up to $78.4$	Yes (full or partial)
400	$38.7$–$47.0$	up to $91.7$	Yes
$10^3$	$30$–$34.2$	major	Yes
$2.74 \times 10^5$	$29$	$18$	Partial
3D wings, high AoA	$65$	$>$100 (rms)	Yes (reattachment)

Typical DRL control laws require actuation intensity far below $1\%$ of the inflow mass flow rate (Rabault et al., 2018, Jia et al., 19 Apr 2024). Agents trained at discrete $Re$ values generalize to a wide $Re$ range (Tang et al., 2020, Jia et al., 18 Apr 2024). The DRL approach also demonstrates substantial robustness—trained agents operate effectively across varying boundary conditions and even with mismatches in state-space dimensionality, provided careful transfer learning mechanisms are employed (Yan et al., 23 Jan 2024).

5. Advancements: Higher Complexity and Real-World Implementation

DRL-AFC research has advanced from laminar 2D benchmark studies to:

• Multi-environment, parallelized training, enabling order-of-magnitude reductions in training time and scaling to larger and more realistic configurations (Rabault et al., 2019, Jia et al., 18 Feb 2024).
• Three-dimensional flow control, including square/circular cylinders and finite-span wings with MARL implementations and transfer learning to bridge 2D-3D state gaps (Yan et al., 23 Jan 2024, Montalà et al., 12 Sep 2025, Montalà et al., 8 Nov 2024).
• Extension to turbulent regimes up to $Re = 2.74 \times 10^5$, where DRL remains effective using only surface sensor information and zero-net mass-flux jet arrays (Chen et al., 20 Dec 2024).
• Experimental validation, such as real-time suppression of vortex-induced vibrations at $Re=3000$ despite significant actuator lag (Sababha et al., 29 Sep 2025).

A tabulation of expanding domains:

Domain Complexity	DRL Features	Notable Achievements
2D laminar cylinder	PPO, <1% actuation	$8$–$40$\% drag reduction
2D/3D square/elliptic	SAC, transfer learning	$52$\% drag reduction (3D)
3D turbulent wing	Multi-agent PPO, parallelization	$65$\% drag, $79$\% lift incr.
Experimental VIV	PPO, state augmentation	$>95$\% vibration suppression

6. Challenges and Open Problems

Despite its notable success, DRL-AFC faces several technical barriers:

• Computational cost: Direct CFD-DRL training remains bottlenecked by CFD solver time, with strong diminishing returns on CFD parallelization. Multi-environment or hybrid approaches are essential for practical scaling (Rabault et al., 2019, Jia et al., 18 Feb 2024).
• Data efficiency and reward shaping: While policy gradient methods (PPO, SAC) are comparatively stable, careful engineering of reward signals, temporal update frequencies (typically $f_\mathrm{actuate} \sim 0.1f_\mathrm{shedding}$), and smoothing are necessary to obtain physically plausible control.
• Experimental realization: Challenges include actuator/sensor delays, hardware non-idealities, and the need for minimal, physically meaningful observations. Recent studies have shown that DRL can compensate for actuator lag via state augmentation (Sababha et al., 29 Sep 2025).
• Turbulent and fully 3D flows: Vortex-dominated regimes are accessible, but large-eddy scales ($Re \gtrsim 10^6$), non-periodic forcing, and massively parallel actuation/sensing remain at the research frontier.

Future avenues likely center on further parallelization, hybrid DRL-physics-based controllers, integration of spatial invariance/symmetry into NN architectures, robust multi-agent control, and scaling to high-frequency, real-world experimental environments (Vignon et al., 2023, Jia et al., 19 Apr 2024, Sababha et al., 29 Sep 2025).

7. Significance and Outlook

The application of DRL to active flow control establishes a paradigm where adaptive, data-driven strategies are autonomously synthesized for high-dimensional, nonlinear, and unsteady systems, with minimal a priori modeling. Demonstrated performance—such as near-complete drag recovery in the Kármán vortex street case, significant enhancements in 3D wing aerodynamics, and strong generalization to new regimes—underscores the utility of DRL-AFC in classical and emerging fluid mechanics problems.

Research in this area is rapidly progressing toward industrial and experimental viability, with particular promise in complex geometries, high Reynolds number turbulent flows, and situations requiring both performance and adaptability beyond the scope of conventional control design.