FinFlowRL: Adaptive RL for Market Making

• The paper introduces FinFlowRL, an imitation–reinforcement learning framework that consolidates diverse expert market-making strategies in volatile, non-Markovian environments.
• It employs a two-stage process: initial imitation pretraining with a MeanFlow network followed by PPO-based fine-tuning on a latent noise policy to generate action chunks.
• Experimental results across simulated regimes demonstrate superior Sharpe ratios, improved PnL, and lower maximum drawdowns compared to traditional benchmarks.

FinFlowRL is an imitation–reinforcement learning framework for adaptive stochastic control in finance, introduced for market making under non-stationary and non-Markovian market conditions. In its stated formulation, the framework first pretrains an adaptive meta-policy from a pool of expert strategies and then fine-tunes that policy through reinforcement learning in the noise space, while using action chunking to generate short action sequences rather than isolated one-step decisions. The core claim is that this combination allows the policy to consolidate heterogeneous expert behavior, adapt online through PPO-based optimization, and outperform individually optimized experts across diverse market conditions (Li et al., 30 Aug 2025).

1. Formal stochastic-control formulation

FinFlowRL poses the market-making problem over discrete time steps $t=0,1,\dots,T$ as an MDP

$\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma).$

In the more explicit specification, the state at time $t$ is

$$s_t=\bigl(t,\;X_t,\;q_t,\;S_t,\;\text{spread}_{t-1}\bigr)\in\mathbb{R}^5,$$

where $X_t$ is cash, $q_t$ is inventory, $S_t$ is the mid-price, and $\text{spread}_{t-1}=p^{ask}_{t-1}-p^{bid}_{t-1}$. In practice, the last $T_{obs}$ states are stacked into an observation buffer $\mathcal{O}\in\mathbb{R}^{T_{obs}\times 5}$. A broader formulation also states that the observation may encode mid-price, order-book depths, Hawkes intensities, fractional-Brownian signals, inventory $\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma).$0, and cash position $\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma).$1 (Li et al., 30 Aug 2025).

The action space is continuous. The agent selects bid and ask offsets

$\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma).$2

The market dynamics combine a jump–diffusion mid-price process with mutually exciting Hawkes order arrivals. The mid-price evolves as

$\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma).$3

where $\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma).$4 is fractional Brownian motion, $\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma).$5, and $\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma).$6 is a Poisson jump process. Buy and sell arrival intensities are specified through Hawkes processes, for example

$\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma).$7

with execution intensity then modulated by the chosen spread as $\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma).$8, yielding buy fills with probability $\mathcal{M}=(\mathcal{S},\mathcal{A},P,r,\gamma).$9 (Li et al., 30 Aug 2025).

The reward is defined as profit-and-loss minus an inventory penalty. One formulation gives

$t$0

while an alternate summary expresses it as

$t$1

with $t$2 typically quadratic. The control objective is the discounted return

$t$3

Within the paper’s framing, this formalization is used to argue that classical stochastic-control methods are limited by simplifying assumptions and stylized environments, especially when market conditions shift (Li et al., 22 Sep 2025).

2. Meta-policy construction and imitation pretraining

The central policy object in FinFlowRL is a flow-matching “meta-policy,” described as a MeanFlow network $t$4. It maps a standard normal noise variable together with state conditioning to an action sequence. In the concise formulation,

$t$5

where $t$6 and $t$7 is the flattened observation (Li et al., 30 Aug 2025).

Pretraining uses a Mixture of Experts pool comprising Avellaneda–Stoikov, GLFT, GLFT-drift, and PPO across 108 simulated market scenarios. One version states that this produces $t$8 million state–action pairs,

$t$9

with scenarios varying $$s_t=\bigl(t,\;X_t,\;q_t,\;S_t,\;\text{spread}_{t-1}\bigr)\in\mathbb{R}^5,$$0, $$s_t=\bigl(t,\;X_t,\;q_t,\;S_t,\;\text{spread}_{t-1}\bigr)\in\mathbb{R}^5,$$1, and jump intensity (Li et al., 22 Sep 2025). Another summary describes the collection procedure more selectively: for each scenario, the best expert $$s_t=\bigl(t,\;X_t,\;q_t,\;S_t,\;\text{spread}_{t-1}\bigr)\in\mathbb{R}^5,$$2 is chosen and corresponding state–action pairs are retained.

The imitation objective is given in two closely related forms. In one, the network is trained by behavioral cloning: $$s_t=\bigl(t,\;X_t,\;q_t,\;S_t,\;\text{spread}_{t-1}\bigr)\in\mathbb{R}^5,$$3 where $$s_t=\bigl(t,\;X_t,\;q_t,\;S_t,\;\text{spread}_{t-1}\bigr)\in\mathbb{R}^5,$$4 is the one-step flow-matching likelihood under $$s_t=\bigl(t,\;X_t,\;q_t,\;S_t,\;\text{spread}_{t-1}\bigr)\in\mathbb{R}^5,$$5 (Li et al., 30 Aug 2025). In the fuller MeanFlow presentation, the model learns to denoise a Gaussian latent $$s_t=\bigl(t,\;X_t,\;q_t,\;S_t,\;\text{spread}_{t-1}\bigr)\in\mathbb{R}^5,$$6 into an expert action by minimizing

$$s_t=\bigl(t,\;X_t,\;q_t,\;S_t,\;\text{spread}_{t-1}\bigr)\in\mathbb{R}^5,$$7

with

$$s_t=\bigl(t,\;X_t,\;q_t,\;S_t,\;\text{spread}_{t-1}\bigr)\in\mathbb{R}^5,$$8

and

$$s_t=\bigl(t,\;X_t,\;q_t,\;S_t,\;\text{spread}_{t-1}\bigr)\in\mathbb{R}^5,$$9

At generation time, one-step inference is

$X_t$0

which the paper states meets microsecond latency requirements in HFT (Li et al., 22 Sep 2025).

This pretraining stage is presented as knowledge consolidation rather than single-expert replication. The pretrained MeanFlow alone is reported to match its strongest expert, GLFT, in Sharpe ratio, which the paper interprets as successful consolidation of diverse expert behavior (Li et al., 22 Sep 2025).

3. Reinforcement-learning fine-tuning in noise space

After imitation pretraining, FinFlowRL freezes the MeanFlow meta-policy and introduces a separate stochastic noise policy

$X_t$1

along with a value network $X_t$2. The latent-action construction is

$X_t$3

with reward

$X_t$4

Fine-tuning therefore updates $X_t$5 and $X_t$6 while leaving $X_t$7 frozen (Li et al., 30 Aug 2025).

The RL stage uses PPO. The clipped surrogate objective is written as

$X_t$8

where

$X_t$9

The total optimization objective includes value fitting and entropy regularization,

$q_t$0

A parallel description states the RL objective as

$q_t$1

with advantage estimates obtained from $q_t$2 (Li et al., 22 Sep 2025).

Algorithmically, the pipeline is explicitly two-stage. Phase I simulates diverse market scenarios, selects or aggregates expert behavior, and trains $q_t$3 under the imitation objective. Phase II freezes $q_t$4, initializes $q_t$5, samples latent noise $q_t$6, generates action chunks $q_t$7, executes the first $q_t$8 actions, stores $q_t$9, computes advantages via GAE, and performs minibatch gradient updates on the PPO objective. At inference time, the policy uses $S_t$0, forms an action chunk, and executes the first $S_t$1 actions before replanning (Li et al., 30 Aug 2025).

A recurrent misconception is to treat FinFlowRL as end-to-end PPO over raw actions. The framework is not described that way. PPO appears in three different roles: as one expert in the imitation pool, as an end-to-end baseline in evaluation, and as the optimizer of the separate noise policy in the latent space while the pretrained MeanFlow module remains frozen (Li et al., 30 Aug 2025).

4. Action chunking and non-Markovian market structure

A defining design choice in FinFlowRL is action chunking. Rather than outputting a single action $S_t$2, the model generates a short sequence

$S_t$3

where $S_t$4, and only the first $S_t$5 actions are executed before the agent re-observes the environment (Li et al., 30 Aug 2025).

The fuller notation introduces three time scales: $S_t$6 as the look-back window, $S_t$7 as the prediction horizon or chunk length, and $S_t$8 as the number of actions actually executed before replanning. One summary writes the chunk output as

$S_t$9

after which only the first $\text{spread}_{t-1}=p^{ask}_{t-1}-p^{bid}_{t-1}$0 actions are carried out (Li et al., 22 Sep 2025).

The stated motivation is to address the non-Markovian nature of markets. The paper attributes this non-Markovianity to path dependence, volatility clustering, order-flow autocorrelations, and limit-order-book microstructure. Chunking is said to encode a short “plan” into each decision, integrate history over $\text{spread}_{t-1}=p^{ask}_{t-1}-p^{bid}_{t-1}$1, and mitigate single-step compounding errors. This suggests that the framework treats temporal abstraction not merely as an efficiency device but as part of the model class for financial control under persistent microstructural effects (Li et al., 30 Aug 2025).

5. Experimental setup, baselines, and reported outcomes

The reported evaluation uses four simulated market regimes: High Vol/High Demand (HH), High Vol/Low Demand (HL), Low Vol/High Demand (LH), and Low Vol/Low Demand (LL), with

$\text{spread}_{t-1}=p^{ask}_{t-1}-p^{bid}_{t-1}$2

and episodes of length 100 (Li et al., 30 Aug 2025). Out-of-sample performance is summarized over 1 million trials in the later preprint (Li et al., 22 Sep 2025).

The baselines are Random, Avellaneda–Stoikov (AS), Guéant–Lehalle–Fernandez–Tapia (GLFT), GLFT-drift, end-to-end PPO, and pretrained MeanFlow. The metrics are Profit & Loss (PnL), Sharpe ratio (SR), and maximum drawdown (MDD %) (Li et al., 30 Aug 2025).

The principal numerical claims reported for FinFlowRL are the following:

• In all four regimes, it yields the highest Sharpe ratio.
• In the HH regime, SR is approximately $\text{spread}_{t-1}=p^{ask}_{t-1}-p^{bid}_{t-1}$3, compared with $\text{spread}_{t-1}=p^{ask}_{t-1}-p^{bid}_{t-1}$4 for GLFT and $\text{spread}_{t-1}=p^{ask}_{t-1}-p^{bid}_{t-1}$5 for AS.
• In the LL regime, PnL is approximately $\text{spread}_{t-1}=p^{ask}_{t-1}-p^{bid}_{t-1}$6, compared with $\text{spread}_{t-1}=p^{ask}_{t-1}-p^{bid}_{t-1}$7 for GLFT, and SR is approximately $\text{spread}_{t-1}=p^{ask}_{t-1}-p^{bid}_{t-1}$8, compared with $\text{spread}_{t-1}=p^{ask}_{t-1}-p^{bid}_{t-1}$9.
• MDD is uniformly lower than PPO and AS in every regime (Li et al., 30 Aug 2025).

A more detailed version gives regime-wise FinFlowRL values:

• HH: PnL $T_{obs}$0, SR $T_{obs}$1, MDD $T_{obs}$2
• HL: PnL $T_{obs}$3, SR $T_{obs}$4, MDD $T_{obs}$5
• LH: PnL $T_{obs}$6, SR $T_{obs}$7, MDD $T_{obs}$8
• LL: PnL $T_{obs}$9, SR $\mathcal{O}\in\mathbb{R}^{T_{obs}\times 5}$0, MDD $\mathcal{O}\in\mathbb{R}^{T_{obs}\times 5}$1 (Li et al., 22 Sep 2025)

The same source states that the FlowRL fine-tuning stage consistently outperforms all baselines, including the experts and the frozen model, especially under sudden jumps, and that the framework achieves the highest risk-adjusted returns and lowest drawdowns across all four regimes (Li et al., 22 Sep 2025).

Training-efficiency claims are also central. FlowRL is reported to train with only approximately $\mathcal{O}\in\mathbb{R}^{T_{obs}\times 5}$2 million trainable parameters, converge in $\mathcal{O}\in\mathbb{R}^{T_{obs}\times 5}$3 steps, run at approximately $\mathcal{O}\in\mathbb{R}^{T_{obs}\times 5}$4 the time of PPO, and achieve approximately $\mathcal{O}\in\mathbb{R}^{T_{obs}\times 5}$5 inference latency over $\mathcal{O}\in\mathbb{R}^{T_{obs}\times 5}$6 steps, which is described as “fully suitable for real-time HFT” (Li et al., 30 Aug 2025).

6. Scope, related formulations, and interpretive issues

Within the provided sources, FinFlowRL is consistently defined as a finance-specific framework for adaptive stochastic control that combines imitation learning, flow matching, noise-space reinforcement learning, and action chunking (Li et al., 30 Aug 2025). Its stated target problem is market making under jump–diffusion price dynamics and Hawkes order-flow dynamics, rather than generic continuous control.

A source of potential confusion is nomenclature. A separate paper, “ReinFlow: Fine-tuning Flow Matching Policy with Online Reinforcement Learning,” is about continuous robotic control and the associated summary explicitly states that “FinFlowRL” refers to the ReinFlow algorithm (Zhang et al., 28 May 2025). That summary also describes a different application domain—locomotion and manipulation tasks rather than financial stochastic control. By contrast, the two preprints titled “FinFlowRL: An Imitation-Reinforcement Learning Framework for Adaptive Stochastic Control in Finance” define FinFlowRL as a market-making framework in finance (Li et al., 22 Sep 2025). The overlap therefore appears to be methodological—flow-matching policies fine-tuned with RL—rather than domain-specific identity.

Another interpretive issue concerns the evaluation scope. The abstract states that traditional methods struggle in “real world markets,” and one summary concludes that experiments span “diverse simulated and real HFT environments” (Li et al., 30 Aug 2025). The detailed quantitative setup, however, is explicitly given in terms of four simulated regimes with specified volatility and demand parameters. This suggests that the framework is positioned as relevant to real-world market adaptation, while the numerical evidence provided in the summaries is simulation-based.

In the paper’s own synthesis, FinFlowRL unifies imitation of multiple expert market-making strategies through a fast one-step MeanFlow policy, adapts that policy through lightweight noise-space PPO fine-tuning, and uses action chunking to address path dependence. The conceptual significance lies in how these components are combined: expert consolidation through flow matching, adaptation through latent stochastic control, and temporal planning through chunked quoting policies (Li et al., 30 Aug 2025).