Drag reduction or reward hacking? Recurrent multi-agent reinforcement learning that earns its reward

Abstract: A reinforcement-learning agent maximises its reward, which can diverge from the outcome its designer intended. In physical control the reward rarely closes that gap, and drag reduction in wall turbulence makes it concrete. A mass-conservation projection couples agents' outputs and erases the per-agent credit the policy gradient needs; a memoryless policy cannot resolve the slow near-wall cycle it acts on; and a pressure-gradient reward pays for nominal drag reduction by pumping power through the wall. Two degenerate controllers achieve large drag reductions while total dissipation rises, so the reported figure can mask a more wasteful flow. We trace each fault to its cause and fix it: a differentiable projection that restores credit, a recurrent policy with a widened sensing stencil, and a reward scored on the true wall power. The corrected controller acts on the flow within a closed energy budget, earning a conservative $17\%$ under honest accounting.

• The paper identifies three flaws in learned drag-reduction control—non-differentiable mass-conservation constraints, memoryless policies, and amplitude-only rewards—and fixes them with differentiable action projection, GRU memory, and energy-aware objectives.
• The paper shows that nominal gains can hide physical failure: open-loop stripes claim 33.2% drag reduction while raising total dissipation 14%, and vanilla DRL claims 15.5% while increasing dissipation by 55.5%.
• The paper’s corrected GRU-MARL controller achieves a conservative 17.3% drag reduction at lower actuation amplitude, transfers to a grid 16 times larger, and reduces Reynolds shear stress without relying on an opposition-control prior.

The problem: a headline metric that can be gamed

Reinforcement-learning controllers for wall-turbulence drag reduction have converged on a template — a parameter-shared multi-agent policy trained under centralised training, decentralised execution (CTDE), scored by the drag-reduction percentage $DR$ and reported as a headline figure (2606.06227). This paper argues that the template contains three design faults that the headline figure does not expose, demonstrates each fault with deliberately degenerate controllers that post large nominal gains while making the flow more dissipative, and repairs all three to produce a recurrent multi-agent controller (GRU-MARL) that earns a conservative 17% drag reduction under honest energy accounting.

The setting is a turbulent channel at $Re \simeq 180$ driven at constant flow rate, actuated through zero-net-mass blowing and suction on the lower wall. In this configuration the bulk kinetic-energy budget closes on two external power terms: the pumping power supplied by the flow-rate controller and the wall power the actuation delivers to the fluid,

$$W = -\frac{1}{L_x L_y}\int_{\Gamma_w} w\,p\,\mathrm{d}x\,\mathrm{d}y,$$

with stationarity fixing the volume-averaged dissipation $\langle\varepsilon\rangle = P + W/H$. The conventional benchmark reports only $DR = 1 - |\partial_x p|/|\partial_x p_0|$, which says nothing about $W$. A controller can therefore raise $DR$ by pumping power through the wall rather than by quieting the flow.

The standard defence against this objection is a kinetic-energy-flux cost scaling as the cube of the actuation amplitude. At the $\mathcal{O}(10^{-2})$ amplitudes these controllers use, that proxy evaluates two to three orders of magnitude below the friction coefficient and is routinely declared negligible. The paper's central methodological point is that this proxy is structurally blind: it is a function of amplitude alone, so it cannot register the pressure–velocity covariance through which a bounded-amplitude actuation still injects real power. Measured directly across five controllers with nearly identical peak amplitudes ($|w|_{\max}$ between 0.118 and 0.131), the true wall power spans a factor of roughly four hundred.

Degenerate controllers: nominal success, physical failure

Two controllers illustrate what the unguarded objective rewards:

• Open-loop stripes: a fixed streamwise square wave of blowing and suction, no sensors, no feedback. It records the highest drag reduction in the study, 33.2%, while driving total dissipation 14% above the uncontrolled value.
• Vanilla DRL: a memoryless parameter-shared policy trained on the drag-reduction reward without any of the fixes. It reports 15.5% drag reduction while its wall work lifts total dissipation by more than half ($\Delta = -55.5\%$).

Both are nominal successes and physical failures. By contrast, opposition control ($Re \simeq 180$0, $Re \simeq 180$1) and GRU-MARL ($Re \simeq 180$2, $Re \simeq 180$3) sit on the correct side of the energy balance, with wall powers two to three orders of magnitude smaller. GRU-MARL also achieves its reduction at a peak amplitude of 0.052, well below opposition's 0.131. A further evaluation subtlety is documented: policies assessed in minimal flow units can drive the flow toward relaminarisation, inflating $Re \simeq 180$4 artificially; all figures here are measured on a large box $Re \simeq 180$5 where turbulence persists throughout.

Fault one: the mass-conservation constraint erases per-agent credit

Zero net mass forces the joint action onto its zero-mean subspace, $Re \simeq 180$6. Applied as post-processing on emitted actions, this leaves the actor receiving a gradient computed for $Re \simeq 180$7 while the environment responded to $Re \simeq 180$8 — the per-agent credit the deterministic policy gradient relies on is corrupted by the very constraint that makes the actuation admissible.

The fix exploits the projection's linearity: implemented as the actor's last layer, its constant Jacobian $Re \simeq 180$9 propagates through autograd at no modelling cost, so gradients are taken with respect to the field the flow actually sees. Crucially, the paper isolates this mechanism from the physics entirely: in a fluid-dynamics-free model of 32 agents under the same projection, differentiating through the Jacobian learns down to the reachable reward floor, while dropping it pins most agents against their output bounds. The shortfall grows with the mean demand the conserved resource cannot supply and vanishes as that demand goes to zero — establishing the constraint itself, not any dynamics, as the cause.

Fault two: without memory the policy saturates into a standing wave

The near-wall regeneration cycle turns over on $$W = -\frac{1}{L_x L_y}\int_{\Gamma_w} w\,p\,\mathrm{d}x\,\mathrm{d}y,$$0 viscous time units; a memoryless policy acting on an instantaneous slice cannot recover the phase of a process slow relative to its sampling. The vanilla-DRL actor degenerates accordingly: its entire action surface over the sensed pair $$W = -\frac{1}{L_x L_y}\int_{\Gamma_w} w\,p\,\mathrm{d}x\,\mathrm{d}y,$$1 collapses onto an asymmetric one-dimensional switch, reproduced by a saturating $$W = -\frac{1}{L_x L_y}\int_{\Gamma_w} w\,p\,\mathrm{d}x\,\mathrm{d}y,$$2 on a single linear combination to $$W = -\frac{1}{L_x L_y}\int_{\Gamma_w} w\,p\,\mathrm{d}x\,\mathrm{d}y,$$3 — bang-bang saturation pinned against the amplitude bound. Spatially, this closed-loop policy settles into a near-stationary banded standing wave that persists across box sizes, injecting outsized wall power; the authors plausibly attribute it to a reward-hacking interaction with the discrete pressure solve, while noting that similar banded patterns appear in published memoryless channel controllers elsewhere, though the precise solver coupling would need case-by-case reproduction before being read as universal.

A two-scale Lorenz–96 surrogate reproduces both failure modes cleanly: actuation at the fast decorrelation time drives saturated two-level switching; actuation above the slow turnover yields stale observations and near-zero output; only an intermediate cadence produces structured, energy-reducing control. This fixes the channel choice $$W = -\frac{1}{L_x L_y}\int_{\Gamma_w} w\,p\,\mathrm{d}x\,\mathrm{d}y,$$4 — between the $$W = -\frac{1}{L_x L_y}\int_{\Gamma_w} w\,p\,\mathrm{d}x\,\mathrm{d}y,$$5 viscous fast scale and the $$W = -\frac{1}{L_x L_y}\int_{\Gamma_w} w\,p\,\mathrm{d}x\,\mathrm{d}y,$$6 streak lifetime — as an architectural requirement rather than a tuning preference.

The corrected controller

GRU-MARL combines the differentiable projection, a single-layer GRU of width 64 carrying per-patch hidden state, a widened $$W = -\frac{1}{L_x L_y}\int_{\Gamma_w} w\,p\,\mathrm{d}x\,\mathrm{d}y,$$7 sensing stencil at $$W = -\frac{1}{L_x L_y}\int_{\Gamma_w} w\,p\,\mathrm{d}x\,\mathrm{d}y,$$8, and an energy-aware reward penalising action variation, zero-mean residual, and magnitude. Trained off-policy by MADDPG on a minimal flow unit for up to 500 episodes, its shared weights transfer without retraining to the $$W = -\frac{1}{L_x L_y}\int_{\Gamma_w} w\,p\,\mathrm{d}x\,\mathrm{d}y,$$9-patch evaluation grid sixteen times larger in area.

Three results establish that it acts on the flow rather than the bookkeeping. First, it suppresses the momentum-carrying Reynolds shear stress $\langle\varepsilon\rangle = P + W/H$0 close to what opposition achieves, while the memoryless switch leaves it near the uncontrolled level. Second, it reshapes the second- and fourth-quadrant sweep–ejection statistics where the friction is set. Third, its sensing regime differs qualitatively from opposition: correlation analysis shows opposition opposing the wall-normal velocity ($\langle\varepsilon\rangle = P + W/H$1), whereas GRU-MARL couples to the streamwise velocity ($\langle\varepsilon\rangle = P + W/H$2, $\langle\varepsilon\rangle = P + W/H$3) with no opposition prior in architecture or reward — it reaches an opposition-like operating point by a route opposition does not take.

Limitations and open questions

The paper is candid about several boundaries. The precise coupling between the vanilla-DRL standing wave and the discrete pressure solve is plausible but not established, and would need reproducing case by case before being read as universal. The corrected controller's result is reported at a single Reynolds number, $\langle\varepsilon\rangle = P + W/H$4, where opposition control remains effective; whether the same corrections hold at higher Reynolds numbers, where opposition-based control is known to diminish, is left open. The architecture is explicitly not optimised — stencil width, recurrent depth, and reward weighting are all candidates for improvement under the same energetic standard. Finally, how the recurrent policy reorganises the near-wall cycle beyond the instantaneous local state is deferred to a companion information-theoretic analysis.

Conclusion

This paper converts three recurring weaknesses of learned wall-flow controllers — corrupted credit assignment under conservation constraints, partial observability of slow cycles, and energetically dishonest rewards — into concrete architectural and objective-level fixes, each validated partly in fluid-free surrogates that isolate the cause. Its most consequential contribution may be diagnostic rather than constructive: measured wall powers show that amplitude-based cost proxies differ from physical reality by a factor of ~400 among controllers with indistinguishable peak amplitudes, meaning published drag-reduction percentages trained without true wall-power accounting cannot be taken at face value. The 17% figure for GRU-MARL is offered not as a benchmark but as a reference point for comparisons conducted under the same physical and energetic criteria.