---
title: 'MacroHFT: Memory-Augmented Crypto HFT'
url: https://www.emergentmind.com/topics/macrohft-framework
type: topic
---

# MacroHFT: Memory-Augmented Crypto HFT

MacroHFT is a memory-augmented, context-aware reinforcement learning (RL) framework for high-frequency trading (HFT) on cryptocurrency markets, designed to address conventional RL vulnerabilities such as overfitting and limited adaptability to non-stationary financial conditions. The system combines multiple specialized sub-agents—each conditioned on distinct market regimes of trend and volatility—with a hyper-agent via soft policy mixing and an episodic memory module, yielding a meta-policy capable of robust decision-making under rapid market fluctuations and rare events. Extensive empirical evaluation demonstrates state-of-the-art performance in minute-level trading across major cryptocurrency pairs, with strong improvements in both profitability and risk-adjusted metrics [2406.14537].

## 1. Problem Formulation in MacroHFT

MacroHFT targets discrete-time, minute-level HFT in high-dimensional financial microstructures. At each time step $t$, the system observes an $M$-level limit order book
$$b_t = \{ (p_t^{b_i}, q_t^{b_i}), (p_t^{a_i}, q_t^{a_i}) \}_{i=1}^M$$
where $p^{b_i}, q^{b_i}$ and $p^{a_i}, q^{a_i}$ are the $i$-th best bid/ask prices and sizes. The state further comprises an OHLCV snapshot $x_t = (p_t^o, p_t^h, p_t^l, p_t^c, v_t)$ and technical indicators $y_t = \phi(x_{t-h+1:t}, b_{t-h+1:t})$ calculated over a lookback window of $h$ minutes.

The agent's position at each step is $P_t \in \{0, 1\}$, denoting holding either no coin or one unit. The net account value is $V_t = V_{ct} + P_t \cdot p_t^c$, where $V_{ct}$ is cash.

MacroHFT formulates the trading process as a two-level Markov Decision Process (MDP):
- **Low-level MDP (sub-agents):** States $s_{lt} = (s_{lt}^1, s_{lt}^2, P_t)$, where $s_{lt}^1$ are static features, $s_{lt}^2$ are context features (historical sequences).
- **High-level MDP (hyper-agent):** States $s_{ht} = (s_{ht}^1, s_{ht}^2, P_t)$, with $s_{ht}^1$ including all information from $s_{lt}$ and $s_{ht}^2$ aggregating context (trend slope, volatility) over a global window $h_c$.

Actions $a_{lt}, a_{ht} \in \{0, 1\}$ represent target positions; the reward for both levels is
$$r_{lt} = [a_{lt}(p_{t+1}^c-p_t^c) - \delta|a_{lt}-P_t|] \cdot m$$
with $\delta$ as transaction cost and $m$ the trade size.

## 2. Sub-Agent Specialization and Conditional Context Adaptation

### 2.1 Market Regime Decomposition

The input time series is segmented into chunks of length $l_\text{chunk}$ (e.g., 360 or 4320 minutes). For each chunk:
- Compute (i) de-noised slope (trend label); (ii) average realized volatility (volatility label).
- Partition the data into three slope quantiles (Bear, Flat, Bull) and three volatility quantiles (Stable, Medium, Volatile), producing six disjoint subsets.
- Train a dedicated sub-agent for each subset, selecting the best epoch according to validation performance.

### 2.2 Conditional-Adapter DDQN Architecture

Each sub-agent adopts a dueling DDQN backbone with a *conditional adapter* that injects context embedding into policy computation. At time $t$:
- Encode $s_{lt}^1, s_{lt}^2, P_t$ using neural functions $\psi_1, \psi_2, \psi_3$.
- Aggregate context and position: $c = \psi_2(s_{lt}^2) + \psi_3(P_t)$.
- Compute scale and shift factors $(\beta, \gamma) = \psi_c(c)$.
- After layer normalization: $h = \text{LayerNorm}(h_s) \odot \beta + \gamma$.
- Output dueling Q-values:
  $$Q^{sub}(h,a) = V(h) + \left( A(h,a) - \frac{1}{|A|} \sum_{a'} A(h,a') \right)$$

The sub-agent loss combines DDQN temporal-difference and a KL divergence to an optimal Q-distribution:
$$
L_\text{sub} = \left(r_t + \gamma Q^{sub}_t(h', \arg\max_{a'} Q^{sub}(h',a')) - Q^{sub}(h,a)\right)^2 + \alpha_l \text{KL}(Q^{sub}(h, \cdot) \parallel Q^*(h, \cdot))
$$

## 3. Hyper-Agent Meta-Policy and Memory Mechanism

### 3.1 Meta-Policy via Soft Policy Mixing

The hyper-agent encodes $s_{ht}$ and applies a conditional adapter analogous to that of sub-agents. It produces logit scores $u \in \mathbb{R}^N$ across $N$ sub-agents, which are transformed into softmax weights $w_i$. The meta-Q value is a convex combination:
$$
Q^{hyper}(s_{ht}, a) = \sum_{i=1}^N w_i Q_i^{sub}(s_{ht}, a)
$$
The hyper-agent selects $a_{ht} = \arg\max_a Q^{hyper}(s_{ht}, a)$.

### 3.2 Episodic Memory Module

The memory module consists of a table $M = \{ (k_i, (s_i, a_i), v_i) \}$, where:
- Keys $k_i = \psi_{enc}(s_i) \in \mathbb{R}^K$, generated by the hyper-agent encoder.
- Value $v_i = r_i + \gamma \max_{a'} Q^{hyper}(s'_i, a')$ is the TD target.
- Query with the current state $s$ to find top-$m$ keys by L2 similarity, filter experiences where $a_i = a$, and aggregate using attention.

The memory-based Q is:
$$
Q_M(s,a) = \sum_{i=1}^m w_i v_i
$$

### 3.3 Hyper-Agent Loss

The hyper-agent's loss incorporates (i) TD error, (ii) KL divergence to optimal Q-distribution, and (iii) memory regularization:
$$
L_\text{hyper} = \left( r_t + \gamma Q^{hyper}_t(s', \arg\max_{a'} Q^{hyper}(s', a')) - Q^{hyper}(s,a) \right)^2 + \alpha_h \text{KL}(Q^{hyper}(s, \cdot)\|Q^*(s, \cdot)) + \beta \left(Q^{hyper}(s, a) - Q_M(s, a)\right)^2
$$

## 4. Training Pipeline

MacroHFT uses a two-phase curriculum:
1. **Sub-agent training:**
   - Each of the 6 sub-agents is trained for 15 epochs on its respective regime-specific subset using Adam (learning rate $1 \times 10^{-4}$), with embedding dimension 64 and network width 128.
   - Chunk lengths $l_{chunk} \in \{360, 4320\}$ and regularization $\alpha_l \in \{0,1,4\}$ selected by validation.
   - The optimal checkpoint for each regime is retained.
2. **Hyper-agent training:**
   - Training for 15 epochs on the full dataset, embedding dimension 32, network width 128, same optimizer.
   - Memory capacity is set large enough to include latest experiences; regularization $\beta \in \{1,5\}$ adjusted by validation.

## 5. Empirical Evaluation

### 5.1 Datasets and Feature Construction

MacroHFT is evaluated on four cryptocurrency markets: BTC/USDT, ETH/USDT, DOT/USDT, LTC/USDT, using minute-level price and 5-level order book data. Feature engineering includes raw LOB and OHLCV snapshots, low-level context (window $h=60$ minutes), high-level context (trend and volatility), and technical indicators $\phi$ as detailed in the paper appendix.

### 5.2 Baselines and Metrics

Comparison baselines include value-based (DQN, DDQN, CDQNRP), policy-based (PPO, CLSTM-PPO), hierarchical (EarnHFT), and rule-based (IV, MACD) approaches. Evaluation uses:
- Total return (TR): $\frac{V_T-V_1}{V_1}$
- Annualized volatility (AVOL): $\sigma[r]\sqrt{m}$
- Maximum drawdown (MDD), Annualized Sharpe Ratio (ASR): $\frac{\mathbb{E}[r]}{\sigma[r]}\sqrt{m}$,
- Annualized Calmar Ratio (ACR), and Annualized Sortino Ratio (ASoR);
where $m=525600$ minutes/year.

### 5.3 Test Performance

On the test set:
| Ticker     | TR (%) | ASR  | MDD (%) |
|------------|--------|------|---------|
| BTC/USDT   | 3.03   | 0.61 | 5.41    |
| ETH/USDT   | 39.28  | 3.89 | 9.67    |
| DOT/USDT   | 13.79  | 0.97 | 15.89   |
| LTC/USDT   | 18.16  | 1.50 | 14.24   |

MacroHFT leads in profit and most risk-adjusted metrics across all assets.

## 6. Architectural Insights, Limitations, and Prospects

MacroHFT's context-aware conditional adapter lets sub-agents avoid overfitting to static features, instead adapting rapidly to regime changes driven by $s_{lt}^2$ and $P_t$. Soft-mixing yields more balanced policies than hard selection, handling bear/bull/volatile trends concurrently. The episodic memory module enforces Q-estimate consistency among similar states and supports prompt adaptation to atypical events by leveraging trajectory-attached targets.

Ablation results indicate removing either the conditional adapter or the memory mechanism reduces returns by 20–50%, with particularly adverse effects on drawdown in stressed markets (e.g., DOT, LTC).

Current limitations include restriction to a single long position per trade, lack of explicit latency/slippage modeling beyond a static transaction fee $\delta$, and sensitivity to hyperparameters ($l_{chunk}$, $\alpha_l$, $\beta$). Practical deployment would benefit from incorporating slippage-aware execution, multi-asset extensions, and latency-resilient order handling.

In summary, MacroHFT's two-phase pipeline—specialized regime-adaptive sub-agents augmented by memory-regularized policy mixing—establishes new performance benchmarks in minute-level cryptocurrency HFT tasks, outperforming both standard and hierarchical RL, as well as traditional rule-based approaches [2406.14537].

Source: https://www.emergentmind.com/topics/macrohft-framework