Flow Gym: RL Framework for Active Flow Control

• Flow Gym is a computational framework that leverages reinforcement learning to optimize active flow control in realistic CFD and multi-physics simulations.
• It integrates Gym-style environments with external solvers using preCICE, enabling seamless data exchange and flexible coupling across diverse physics engines.
• The framework implements robust reward mechanisms—demonstrated by experiments achieving about 8% drag reduction—and supports rapid prototyping with standard RL toolkits.

Flow Gym refers to a computational framework for active flow control (AFC) tasks structured as Gym-style reinforcement learning (RL) environments, where an agent interacts with realistic CFD or multi-physics simulations via the preCICE coupling library. Gym-preCICE, a representative implementation, enables seamless RL algorithm integration with external solvers, supporting single- and multi-physics systems and facilitating the rigorous study and optimization of AFC problems (Shams et al., 2023).

1. Conceptual Basis and Framework Architecture

Flow Gym operationalizes AFC as a sequential optimization task, where RL agents manipulate fluid flow in simulated environments to maximize performance objectives such as drag reduction. Central to this paradigm is the Adapter class, which inherits from gymnasium.Env (formerly OpenAI Gym), encapsulating the requisite RL environment interface. Adapter exposes the essential lifecycle methods:

• reset(self, *, seed, options)
• step(self, action)
• close(self)

Subclassing Adapter necessitates implementation of four abstract methods:

• _get_action(self, action, write_var_list): maps RL actions to boundary data for actuators
• _get_observation(self, read_data, read_var_list): maps raw coupling data to RL-compatible observations
• _get_reward(self): computes the scalar reward
• _close_external_resources(self): resource cleanup

The action and observation spaces are explicitly declared in the subclass constructor (using gym.spaces.Box or Discrete), ensuring compatibility with standard off-the-shelf RL libraries. This architecture abstracts environment definition, isolating solver-specific logic and enabling rapid prototyping of novel AFC scenarios.

2. Coupling and Data Exchange

Integration with external solvers is realized through preCICE’s partitioned-coupling interface. Upon environment reset:

1. External solver(s) are spawned as subprocesses (e.g., OpenFOAM or deal.II).
2. A precice.Interface object is instantiated with user-specified actuation and sensor meshes, variable selections, and the coupling timestep $\Delta t$.
3. The interface is initialized, establishing the co-simulation context.

During each RL step:

• The Adapter invokes _get_action to convert the agent's action into boundary values ($u_\mathrm{act}$), which are written to the preCICE buffer.
• The advance call on the interface propagates the system forward by $\Delta t$, synchronizing all solvers and retrieving observations ($y_\mathrm{obs}$).
• Observations and rewards are computed, and termination logic is evaluated.

preCICE supports explicit or implicit partitioned coupling schemes, with interpolation strategies (e.g., nearest-node, RBF) configured in external XML files. Solver and interpolation switching is managed outside Python code, promoting modularity.

3. Reward Formulation and Evaluation

In single-physics AFC examples, the instantaneous reward is a function of the drag coefficient $C_D(t)$ and a weighted lift penalty $|C_L(t)|$:

• Drag: $$D(t) = \int_{\text{body}} \sigma \cdot n \cdot e_x\, dS$$
• Lift: $$L(t) = \int_{\text{body}} \sigma \cdot n \cdot e_y\, dS$$
• $C_D(t) = \frac{D(t)}{\frac{1}{2} \rho U^2 D}$
• $C_L(t) = \frac{L(t)}{\frac{1}{2} \rho U^2 D}$

The canonical reward is:

$r_t = -C_D(t) - \lambda |C_L(t)|$

where $\lambda$ is the lift penalty weight ($\lambda=1$ is typical). The episodic return is $R = \sum_{t=0}^{T} r_t$. In multi-physics examples (such as fluid–structure interaction, FSI), reward definition is problem-specific; for open-loop control, no reward is defined, but metrics like flap-tip deflection or material stresses could be incorporated.

4. Benchmark Environments and Empirical Results

Gym-preCICE has been instantiated for several AFC testbeds:

Environment Type	Action Space Example	Observation Space Example	RL Results
Synthetic Jet AFC	Box([–Qmax, –Qmax], [+Qmax, +Qmax]), $Q_\text{max}=250$ cm³/s (2 jets)	Box($-\infty,+\infty$; shape=(151,))	PPO yields $\sim$8% drag reduction vs. baseline
Rotating Cylinder	Box([–$\omega_{max}$, $\omega_{max}$]), $\omega_{max}=5$ rad/s (1 scalar)	Box($-\infty,+\infty$; shape=(151,))	Similar drag reduction; periodic learned control
FSI Open-loop Flap	Box($y_\mathrm{min}, y_\mathrm{max}$) (vertical jet position)	Box($-\infty,+\infty$; shape=(1,)) (tip displacement)	Flap deflection recorded for open-loop control

For synthetic-jet and rotating-cylinder AFC, agents (trained with PPO) converge within 1000 episodes, achieving stable policy signals and measurable drag reduction. Synthetic jet RL control produces a periodic actuation, reducing $C_D$ by approximately 8% in canonical tests.

5. Custom Environment Design and Solver Interoperability

Expanding Flow Gym to new AFC scenarios follows a deterministic workflow:

1. Draft a JSON configuration specifying environment, physics engines, control I/O, and script handles (e.g., gymprecice-config.json).
2. Subclass Adapter, implement Gym space declarations and the four required methods. For geometry and mesh configuration, vertex coordinates are assigned via files and passed to preCICE.
3. No modifications are required for preCICE or external solver adapters; compatibility extends to any solver supported by preCICE (OpenFOAM, FEniCS, deal.II, CalculiX, XDEM, etc.), selected via the configuration and associated scripts.
4. The environment, once registered (e.g., using gym.register), is immediately compatible with mainstream RL toolkits such as Stable-Baselines3, CleanRL, and RLlib.

This modular configuration enables rapid problem instantiation and solver swapping by altering only external config and scripts—Python code remains unchanged.

6. Significance and Directions for Research

Gym-preCICE exemplifies a "black-box" engine for AFC RL, abstracting solver couplings and environment specification to minimal Python code and intents. It facilitates robust benchmarking and comparative intervention studies across both single- and multi-physics flow systems. The demonstrated capacity for drag reduction and integration flexibility positions Flow Gym frameworks as foundational platforms for empirical RL research in fluid mechanics and coupled systems (Shams et al., 2023).

This suggests future work may address expanded multi-agent settings, hierarchical reward composition, or online adaptation to evolving simulation scenarios. A plausible implication is the adoption of Flow Gym paradigms for cross-domain RL in engineering simulation and experimental control.