Reinforcement Learning for Execution under Dynamic Fees in a Closed-Loop DEX Simulator

Abstract: Trader-facing dynamic fees are increasingly proposed for automated market makers (AMMs), but historical data do not identify how order flow would respond: trader-facing fees do not vary, trader types are latent, and a replayed tape is not a sequential decision environment. We therefore construct a minimal closed-loop simulator in which the missing signal exists by construction: two constant-product pools repriced by an equilibrium-inspired dynamic-fee rule, fee-sensitive noise flow, and closed-form CEX--AMM arbitrage. Equilibrium is used as a closure principle, not as an object the trader learns. Against a tuned benchmark ladder of schedule, planning, lookahead, and tabular policies, a small DQN is the only evaluated valid policy whose paired improvement over tuned one-step routing excludes zero. On a reserved final block of 1{,}000 seeds with completion forced to 1.0 for every policy, it reduces implementation shortfall under every tested intra-step ordering, by $13.3\bps$ of order notional under the pre-specified agent-last ordering, and the edge is concentrated in, and learned from, dynamic-fee environments: under constant fees the paired difference is indistinguishable from zero. The result is model-conditioned counterfactual evidence about execution control in AMMs, not evidence about historical traders, equilibrium play, or deployable profit.

• The paper’s main finding is that a DQN-based policy reduces implementation shortfall by up to 13.3 bps compared to state-of-the-art heuristics in dynamic fee regimes.
• The study utilizes a finite-horizon MDP simulation framework capturing order flow, dynamic fee mechanisms, and arbitrage to replicate closed-loop DEX behavior.
• The analysis shows that model-free learning exploits dynamic fee adaptations effectively, with performance advantages disappearing in static-fee environments.

Reinforcement Learning for Execution under Dynamic Fees in Closed-Loop DEXs

Overview and Motivation

The paper "Reinforcement Learning for Execution under Dynamic Fees in a Closed-Loop DEX Simulator" (2607.10960) addresses a central challenge in decentralized exchange (DEX) design: understanding execution strategies when automated market makers (AMMs) adapt trader-facing fees dynamically. Contemporary empirical analysis is limited by the stationarity of fee regimes and the latent nature of trader types in historical data. The authors circumvent this by constructing a simulation-based experimental framework, wherein order flow, dynamic-fee mechanisms, noise traders, and arbitrage interact endogenously in a closed market loop.

Their key research question is narrow but significant: within an equilibrium-motivated dynamic-fee DEX environment, can a model-free reinforcement learning (RL) agent achieve lower implementation shortfall relative to strong tuned heuristic and planning-based execution strategies? The answer is rigorously conditional: only in dynamic-fee environments, and after exhaustive evaluation protocol, is there clear, statistically significant evidence that a DQN-based policy outperforms current state-of-the-art baselines.

Figure 1: The closed-loop market with two constant-product AMM pools, dynamic-fee repricing, fee-sensitive noise, and CEX–AMM arbitrage.

Simulation Framework and Methodology

The environment is formalized as a finite-horizon Markov Decision Process (MDP) where trader actions induce state transitions encompassing not only the trader’s executions, but also endogenous fee updates, noise trader routing, and arbitrageur responses. Crucially, the AMM pools update fees dynamically via a clipped linear rule inspired by equilibrium analysis in CFMM competition, incorporating own-pool inventory, rival-pool state, and oracle price deviation.

Key components of the simulation:

• Dynamic-Fee Rule: Pool fees adjust as

$$f^{\mathrm{buy}} = [\bar{f} + a_{\text{own}} z_i + a_{\text{riv}} z_{-i} + a_{\text{orc}} m_i]_{f_\text{min}}^{f_\text{max}},$$

with $z_i$ (inventory imbalance) and $m_i$ (mid-oracle gap).

• Order Flow: Buy/sell noise traders route orders by a cost-sensitive softmax, incentivizing flow towards cheaper pools and faithfully modeling non-deterministic routing.
• Arbitrage: A closed-form arbitrageur (with configurable frequency and participation) closes deviations between CEX and AMM prices above cost thresholds.
• Completion Semantics: A forced-terminal completion rule is imposed universally, ensuring all policies achieve full order execution, and distinguishing true execution quality from partial fulfillment artifacts.

Figure 2: Mean shortfall by ordering—quantitative impact of agent priority on execution costs for DQN, TWAP, and lookahead baselines.

Evaluation is performed in a rigorously controlled setting. Baselines are not limited to naïve or static rules, but include:

• Schedule policies (e.g., TWAP, fastest-feasible liquidation)
• Lookahead heuristic: A validation-tuned, per-step optimal routing with urgency penalization.
• Multi-step deterministic/stochastic planners: Certainty-equivalent and Monte Carlo rollout planners.
• Tabular RL learners: On-policy MC and Q-learning.
• DQN: A compressed-observation deep Q-network trained via Huber loss and experience replay.

A key design feature is the use of a "frozen reserved" seed block for headline evaluation, separate from all tuning and development. Statistical comparisons are paired and bootstrapped, with no post-hoc baseline selection, ensuring the credibility of effect size claims.

Main Empirical Results

On the reserved evaluation block (1,000 seeds, strict forced-terminal semantics), the DQN policy demonstrates a statistically robust reduction in implementation shortfall versus all evaluated policies under all tested intra-step priority orderings, but only under dynamic-fee regimes. The effects are detailed:

• Agent-last ordering (deterministic headline): DQN reduces shortfall by $-13.3$ bps versus lookahead (95% CI $[-14.22, -12.32]$), with mean shortfall of 84.7 bps for DQN versus 97.9 bps for lookahead.
• Randomized intra-step: DQN maintains a smaller but still significant edge ($-5.6$ bps, CI $[-7.03, -4.18]$).
• Static-fee regime: The paired performance difference is statistically indistinguishable from zero, indicating that DQN’s advantage is specific to learning and exploiting dynamic fee structure.

Figure 3: DQN wins on a representative seed, illustrating superior inventory pacing by adaptive deferral during adverse price movement.

Figure 4: DQN wait probability as a function of pool-Oracle price gap—demonstrates learned passive execution when AMMs are expensive relative to fundamental value.

Benchmark results also show:

• Planning methods (deterministic and stochastic, depth 2/3) do not close the DQN–lookahead gap; expectation-based search fails to exploit the realized fee and quote path stochasticity that a model-free learner captures.
• Tabular RL approaches, on fine discretization, eliminate most but not all of the DQN's advantage; DQN is the only learner whose confidence interval for improvement excludes zero.
• Headroom calibration: DQN achieves performance within approximately 7.5 bps of a hindsight, achieved (not theoretically optimal) reference. This places the DQN solidly ahead of practical baselines, but not at unattainable lower bounds.

Figure 5: DQN shortfall across different intra-step priority regimes—performance is robust to order specifications when retrained per regime.

Mechanistic Analysis and Behavioral Observations

Fee savings and opportunistic execution are the dominant contributors to DQN’s edge. Policy inspection shows the DQN:

• Times purchases to coincide with both low-fee and low-price states, favoring small inventory clips to preserve flexibility.
• Defers execution when pools are quoted expensively (relative to oracle), only executing as horizons shrink or favorable states emerge.
• Exhibits a failure mode (as per Figure 3) when pools never cheapen sufficiently in sample, as aggressive deferral chases a dip that does not materialize—losing to a more myopic lookahead strategy in some paths.

Routing statistics confirm nearly logistic conditioning on AMM fee gap, aligning with cost-minimizing rational behavior.

Robustness, Attribution, and Sensitivity Analyses

Several critical robustness and attribution studies are performed:

• Completion Protocols: The advantage persists under forced-completion; unexecution is not a source of outperformance.
• Orderings: DQN exhibits sensitivity to intra-step action ordering; retraining under agent-last or randomized ordering restores its advantage. Failure of frozen policies to transfer across priority schemes reflects inherent environment–policy interaction.
• Fee Regime Attribution: Retrained DQN in static-fee mode shows no significant gain over lookahead, confirming edge is due to learning dynamic fee exploitation, not generic market timing.
• Stress tests: Edge persists under wide perturbations—fee coefficients, gas cost, arbitrageur speed, noise particle properties, and additive LP exit or searcher sandwich stress layers.

  Figure 6: Agent-first benchmark ladder, detailing quantitative position of DQN relative to planners, heuristics, and hindsight on the reserved seeds.

  

  Figure 7: Robustness—the DQN–lookahead edge persists even with aggressive LP-adaptation and adversarial sandwich searchers in force.

Theoretical and Practical Implications

The findings have discrete, policy-relevant implications for both DEX mechanism design and optimal execution research:

• Model-Free Learning Utility: Model-free policies can exploit the endogeneity of dynamic fee environments, uncovering execution strategies unavailable to model-based planners due to inherent path dependencies in AMM pricing and fee updates.
• Environment Design Matters: The RL advantage vanishes in static environments; only wherein the execution agent's actions propagate through endogenous fee updates, noise trader rerouting, and arbitrage, does learning uncover systematic value.
• Methodological Rigor: Explicit treatment of paired evaluation, randomized seeds, completion semantics, and separation of transfer from re-solving is critical to credible RL application in market simulation studies.
• Limitations: Policy optimality is environment-specific; results are not directly transferable to production DEXs or historical execution. Claims are strictly conditional on the simulation model, observation compression, and discretized action space. Empirical gains do not imply real-world arbitrage or profit.

Conclusion

This work delivers model-conditioned, causal evidence that in a closed-loop AMM market with dynamic, trader-facing fees, model-free RL-based execution policies—specifically DQN—achieve consistent, statistically significant reductions in implementation shortfall over strong heuristic and planning-based execution strategies. However, these advantages are concentrated in, and learned from, dynamic-fee regimes; no such advantage manifests in static-fee environments. Planning methods evaluated fail to capture this value, highlighting the complexity added by dynamic fee adaptation. The rigorous protocol and simulation environment serve as a template for similar studies in both AMM design and RL-based market interaction.

Avenues for future work include identification of minimal observation sets, continuous policy/action spaces, risk-sensitive (non-risk-neutral) controls, and the study of nonlinear or adversarial fee mechanisms under more intricate liquidity and searcher dynamic models. The discipline of closed-loop simulation and careful evaluation established here is likely to remain critical as AMM research progresses.