Zero-shot adaptation to order book dynamics

Abstract: We describe an adaptive market-making architecture that preserves the analytical structure of the Avellaneda--Stoikov framework while introducing a successor measure-style adaptation mechanism. In our paper we keep Avellaneda--Stoikov fast Hamilton--Jacobi--Bellman structure and make it adaptive to changing market regimes and trading objectives. The central idea is to separate market dynamics from the trading objective. The market state determines a low-dimensional set of Avellaneda--Stoikov parameters, while recent realized rewards determine a low-dimensional objective vector. The HJB forward map then converts this objective into optimal bid and ask quotes through a scalarization of future reward features.

• The paper presents a novel framework unifying the AS model with zero-shot successor-feature-based RL to adapt order book dynamics in real time.
• It decouples market regime extraction from reward-driven objective inference using HJB-forward recursion for dynamic, explainable quote selection.
• Empirical results on BTCUSDT data show improved Sharpe ratios, positive returns, and lower drawdowns compared to fixed-objective baseline strategies.

Zero-Shot Adaptation to Order Book Dynamics: An Expert Analysis

Introduction and Motivation

This paper presents a novel adaptive market-making framework that synthesizes the analytical tractability of the Avellaneda–Stoikov (AS) model with zero-shot adaptation mechanisms drawn from recent advancements in successor feature-driven reinforcement learning. The central innovation is the explicit separation of market dynamics from trading objectives, enabling the system to adapt both to evolving market microstructure and dynamically inferred trading priorities—objectives are extracted from realized recent rewards, while regime detection is based on the contemporaneous limit order book (LOB) state. By leveraging a Hamilton–Jacobi–Bellman (HJB)-based forward solution, the method ensures that adaptation remains analytically solid, low-latency, and interpretable.

Model Architecture: Bridging Analytical Finance and RL

The framework remains anchored to the AS dynamics but decouples the previously static reward structure, equipping it with an adaptive "objective vector" $z_t$ and a dynamic market regime parameter $\xi_t$. The regime inference, $\xi_t$, is extracted from structured LOB features, capturing local volatility, fill intensities, nonlinear adverse-selection proxies, and more, while the objective vector is inferred via successor-measure regression from recent marked reward labels.

The policy mapping proceeds as follows:

1. Market regime extraction: $L_t \mapsto \xi_t$ is modeled via a function $\Xi_\psi$ on the current LOB state;
2. Objective inference: Recent realized reward windows $\mathcal{W}_t$ label reward-feature pairs, yielding $z_t$ by a (possibly regularized) backward representation;
3. HJB mapping and quote selection: The pair $(\xi_t, z_t)$ is processed via HJB recursion to deliver optimal quote distances $(\delta_t^{b,\star}, \delta_t^{a,\star})$.

The immediate benefit is the preservation of the closed-form quoting structure with theory-backed interpretability and real-time feasibility, now augmented to support on-the-fly switches in trading goals or risk tolerances.

Theoretical Foundations

Adaptive Linear Reward Representation

To facilitate adaptation, rewards are defined as linear functionals over feature vectors $\phi(s_t, a_t, s_{t+1})$ consisting of key risk and PnL metrics (e.g., $\xi_t$0, $\xi_t$1, $\xi_t$2, and adverse selection). Target inventory or risk aversion can still be imposed as in classical models, but the framework also allows backward inference: the observed vector $\xi_t$3 can be projected back to implied inventory preferences and risk penalties, closing the loop between empirical outcomes and model specification.

HJB-Forward Successor Features

The central policy mapping employs an HJB-based successor feature formulation:

$\xi_t$4

where $\xi_t$5 captures the expected feature occupancy. The model purposely eschews the typical assumption that a single, objective-independent successor representation suffices for all possible reward structures; rather, the forward map is indexed by $\xi_t$6, preserving nonlinearity and full dependence on the current objective.

Closed-form Quoting Rule

The quoting equations maintain the AS archetype but dynamically link to the current adaptive objective:

$\xi_t$7

where $\xi_t$8 projects the HJB value with the current inferred $\xi_t$9 and market parameter $\xi_t$0. This enables real-time adjustment: if the inference module detects, for example, recent rewards favoring long inventory, quoting becomes more aggressive on the bid side, while adverse-selection terms $\xi_t$1, $\xi_t$2 adaptively repel quotes from toxicity as detected from LOB metrics.

Empirical Evaluation

An extensive backtest is performed on a standardized, high-fidelity reconstruction of BTCUSDT limit order book data using a shared exchange simulation and latency model. Three policies are contrasted:

• Grid-variant Guéant–Lehalle–Fernandez-Tapia (GLFT)
• Grid-variant Avellaneda–Stoikov (AS)
• Rolling Forward-Backward AS-HJB (proposed)

Both GLFT and AS baselines are fixed-latency, static-objective implementations utilizing rolling estimates for volatility and fill intensities, ensuring fairness in comparison.

Diagnostic Results

The diagnostic plots illustrate equity-curve and position behavior:

Figure 1: Diagnostic summary for the classic Avellaneda–Stoikov grid, showing equity and position evolution under static-objective quoting.

The rolling FB-AS HJB approach further reveals objective-variable and quoting adaption:

Figure 2: Diagnostic summary for the adaptive rolling Forward-Backward AS-HJB, demonstrating real-time evolution of inventory, objective vector, and quoting policy.

Numerical Comparisons

Key summary statistics are notable. The grid-based GLFT and AS baselines both yield negative returns and high drawdowns, while the proposed rolling FB-AS HJB delivers positive returns, improved Sharpe ratio, and substantially lower drawdown—with a dramatic reduction in trading frequency and turnover, indicating superior filtering of spurious short-term noise and more controlled risk-taking. Specifically, the FB-AS HJB method achieves:

• Return: $\xi_t$3
• Sharpe ratio: $\xi_t$4
• Max drawdown: $\xi_t$5

Contrast this with both baselines, which post negative returns, negative Sharpe, and two to three times higher drawdown.

Implementation Considerations

The empirical deployment leverages rolling windows for reward labeling, exponential- and regime-conditioned weighting, and ridge regularization to stabilize the objective estimation under noisy or thin data. Safety projections and smoothing guarantee that risk-seeking objective drift is bounded, and the adaptive procedure can be further tuned by incorporating out-of-distribution penalties and hard inventory/quote limits.

Moreover, the HJB recursion is performed over a finite inventory grid and safe action set, maintaining analytical precision and computational viability for sub-second decision loops.

Implications and Future Directions

The proposed architecture demonstrates that successor measure-based adaptation—well-studied in RL for zero-shot goal generalization—can be married to the best-in-class analytical models of market microstructure. Immediate practical impacts include rapid reactivity to evolving market regimes, explainable adaptation to recent realized outcomes, and robust control over inventory and adverse-selection exposure, all while preserving a regime of analytical transparency that is critical for financial risk management.

Theoretically, this approach narrows the gap between model-based reinforcement learning and real-world HFT application, moving towards a future where trading strategies can systematically balance interpretability, tractability, and empirical adaptability. Several extensions are noted:

• Deep neural representations for both the backward (objective inference) and market-state encoders, admitting higher-dimensional and more complex market features
• Extension to multi-asset and multi-objective settings
• Enhanced robustness to market regime shifts through more nuanced regime clustering and regularization

Conclusion

By unifying adaptive RL architectures with the analytical rigor of Avellaneda–Stoikov, this paper provides a principled and empirically validated framework for zero-shot reward adaptation in market making. The demonstrated risk-adjusted performance improvements and dynamic behavior underscore the utility of decomposing trading policy into regime-aware market state modeling and objective-driven control, all filtered through an HJB-based successor feature pipeline. This work forms a concrete benchmark for future efforts at the interface of modern RL and real-time quantitative finance.