Deep reinforcement learning for optimal trading with partial information

Abstract: Reinforcement Learning (RL) applied to financial problems has been the subject of a lively area of research. The use of RL for optimal trading strategies that exploit latent information in the market is, to the best of our knowledge, not widely tackled. In this paper we study an optimal trading problem, where a trading signal follows an Ornstein-Uhlenbeck process with regime-switching dynamics. We employ a blend of RL and Recurrent Neural Networks (RNN) in order to make the most at extracting underlying information from the trading signal with latent parameters. The latent parameters driving mean reversion, speed, and volatility are filtered from observations of the signal, and trading strategies are derived via RL. To address this problem, we propose three Deep Deterministic Policy Gradient (DDPG)-based algorithms that integrate Gated Recurrent Unit (GRU) networks to capture temporal dependencies in the signal. The first, a one -step approach (hid-DDPG), directly encodes hidden states from the GRU into the RL trader. The second and third are two-step methods: one (prob-DDPG) makes use of posterior regime probability estimates, while the other (reg-DDPG) relies on forecasts of the next signal value. Through extensive simulations with increasingly complex Markovian regime dynamics for the trading signal's parameters, as well as an empirical application to equity pair trading, we find that prob-DDPG achieves superior cumulative rewards and exhibits more interpretable strategies. By contrast, reg-DDPG provides limited benefits, while hid-DDPG offers intermediate performance with less interpretable strategies. Our results show that the quality and structure of the information supplied to the agent are crucial: embedding probabilistic insights into latent regimes substantially improves both profitability and robustness of reinforcement learning-based trading strategies.

• The paper demonstrates that incorporating probabilistic regime information using GRU-enhanced DDPG models significantly improves trading performance.
• It compares three algorithms – hid-DDPG, prob-DDPG, and reg-DDPG – under simulated Markovian regimes to evaluate efficacy with cumulative rewards as a performance metric.
• The study validates findings with NASDAQ pair-trading experiments, showing that the prob-DDPG approach achieves superior profitability and interpretability.

Deep Reinforcement Learning for Optimal Trading with Partial Information

Introduction

The use of reinforcement learning (RL) in financial applications, particularly for devising optimal trading strategies based on latent market information, presents a significant advancement in the field. The focal point of this research is the application of RL to scenarios where a trading signal is characterized by an Ornstein-Uhlenbeck process with regime-switching dynamics. Leveraging recurrent neural networks (RNNs), specifically Gated Recurrent Units (GRUs), this study aims to enhance the extraction of latent information from market signals to inform trading decisions.

Methodology

This research explores the problem of deriving optimal trading strategies from partially observable market information through three distinct Deep Deterministic Policy Gradient (DDPG) algorithms enhanced by GRUs. These algorithms address the challenges posed by latent parameters influencing mean reversion, speed, and volatility, which are commonly encountered in financial markets.

1. hid-DDPG: This approach directly integrates the hidden states from the GRU into the RL framework, allowing the model to intuitively learn trading strategies without explicitly identifying market regimes.
2. prob-DDPG: This two-step method involves initially estimating the probabilities of different market regimes using a structured classification approach, followed by using these probabilities to guide trading strategy development.
3. reg-DDPG: Similar to prob-DDPG, but instead of regime classification, it predicts the next signal value to assist in optimizing trading actions.

Through extensive simulations, each algorithm was tested under varying complexity levels of Markovian regime dynamics. The empirical results in synthetic environments provided insights into the efficacy of embedding probabilistic regime information in improving profitability and interpretability of trading strategies.

Results

The results highlight that the prob-DDPG approach consistently outperforms the other methods, particularly in scenarios with complex regime dynamics. The ability to incorporate probabilistic insights about latent regimes led to superior cumulative rewards and more interpretable strategies. In contrast, the reg-DDPG demonstrated limited additional benefits over the one-step hid-DDPG, primarily due to the added complexity in predicting the next trading signal without explicit regime information.

Figure 1: Comparison of the cumulative rewards for the different DDPG approaches when theta_t is a Markov chain.

The analysis demonstrated that the quality and structure of the information provided to the RL agent are paramount. The implementation of regime probabilities as an input feature substantially enhances the effectiveness of RL-based trading models, a finding supported by both simulated and real-world market data.

Application to Real Market Data

To evaluate the real-world applicability, the algorithms were applied to a pair-trading strategy using historical data from the NASDAQ for Intel (INTC) and SMH. The research found that utilizing RL to trade co-integrated assets based on a mean-reverting signal derived from their combined price dynamics resulted in positive performance, with the prob-DDPG method achieving the highest cumulative rewards.

Figure 2: Comparison of realised cumulative rewards for the hid-DDPG, prob-DDPG and rolling Z-score strategy on the testing set.

Conclusion

The study provides comprehensive evidence that embedding structured regime information into RL architectures significantly enhances performance in trading applications. The prob-DDPG method, by leveraging probabilities of regime presence, achieves higher accuracy and profitability, demonstrating its practical superiority. These findings suggest promising avenues for future research, particularly in exploring multi-agent scenarios and further integrating financial domain knowledge into RL frameworks to improve robustness and adaptability in dynamic market environments.