Reinforcement Learning Pair Trading: A Dynamic Scaling approach

Abstract: Cryptocurrency is a cryptography-based digital asset with extremely volatile prices. Around USD 70 billion worth of cryptocurrency is traded daily on exchanges. Trading cryptocurrency is difficult due to the inherent volatility of the crypto market. This study investigates whether Reinforcement Learning (RL) can enhance decision-making in cryptocurrency algorithmic trading compared to traditional methods. In order to address this question, we combined reinforcement learning with a statistical arbitrage trading technique, pair trading, which exploits the price difference between statistically correlated assets. We constructed RL environments and trained RL agents to determine when and how to trade pairs of cryptocurrencies. We developed new reward shaping and observation/action spaces for reinforcement learning. We performed experiments with the developed reinforcement learner on pairs of BTC-GBP and BTC-EUR data separated by 1 min intervals (n=263,520). The traditional non-RL pair trading technique achieved an annualized profit of 8.33%, while the proposed RL-based pair trading technique achieved annualized profits from 9.94% to 31.53%, depending upon the RL learner. Our results show that RL can significantly outperform manual and traditional pair trading techniques when applied to volatile markets such as~cryptocurrencies.

• The paper introduces a dynamic scaling approach leveraging reinforcement learning to adaptively manage pair trading in volatile crypto markets, achieving annualized profits between 9.94% and 31.53%.
• The methodology employs actor-critic RL algorithms with a continuous action space to optimize both trade timing and quantity based on real-time market signals.
• The RL-based strategy outperforms traditional static models by enhancing profitability and risk control, highlighting its potential for adaptive financial decision-making.

Reinforcement Learning Pair Trading: A Dynamic Scaling Approach

The paper employs RL for pair trading in the volatile cryptocurrency market, introducing a novel RL-based algorithmic trading strategy outperforming traditional methods by leveraging dynamic scaling approaches. In essence, the research adapts trading execution through RL to capitalize on volatility while managing risk inherent in crypto-assets.

Pair Trading in Financial Markets

Pair trading is a popular statistical arbitrage strategy focusing on exploiting price discrepancies between correlated assets. Typically, these strategies are structured to be market-neutral by simultaneously opening a long position on one asset while shorting the other within highly correlated pairs. Traditional approaches using static rules have shown profit generation capabilities but may lag in adaptability to frequent market condition changes (Figure 1).

Figure 1: Architecture of Trading Strategies.

Reinforcement Learning in Algorithmic Trading

RL, particularly in the form of Deep RL, has been increasingly used in algorithmic trading due to its capability to learn optimal strategies through interaction with the environment. By modeling financial markets as a Markov Decision Process (MDP), the RL agent makes investment decisions aimed at maximizing profits. Notably, RL's incorporation into trading strategies allows for adaptive timing and quantity decisions, previously unattainable with fixed-rule systems. Among the RL algorithms, those utilizing actor-critic architectures, such as PPO and A2C, are preferred due to better facilitate policy improvements.

Observation and Action Space

The observation space in RL-based pair trading consists of three elements: Position, Spread, and Zone. Each component provides real-time trading signals, which the RL agent interprets to make informed trading decisions (Figure 2).

Figure 2: The value of position observation based on investment.

Action space in RL pair trading is continuous, encompassing the decision not only about the timing of trades but also the quantity. This involves deciding the degree of investment based on opportunity quality, thereby scaling trades to the assessed potential of profit.

Dynamic Scaling and Investment Strategies

The primary innovation in this research is the integration of dynamic scaling methods through RL, contrasted against static models of traditional pair trading. Experimentation shows this method produces significantly higher profits, achieving a more robust performance metric across variable market conditions (Figure 3).

Figure 3: Prices of BTCEUR and BTCGBP.

Performance Metrics

Experiments demonstrate the RL-based approach yields annualized profits ranging from 9.94% to 31.53%, surpassing the traditional pair trading technique's 8.33% by a substantial margin. The dynamic scalability of investment decisions allows for improved responsiveness to market shifts, thereby optimizing both profitability and risk exposure.

Conclusion

The research successfully introduces a dynamic component to pair trading via RL, providing a scalable approach well-suited to the cryptocurrency market's inherent volatility. The combination of sophisticated RL methods, sophisticated reward shaping, and carefully designed observation/action spaces culminates in a trading strategy capable of outperforming traditional systems. Future work might explore expanding the RL approach to more complex trading strategies, incorporating additional legs or dimensions in trading, or adapting RL models to other volatile markets. This paper signifies a definitive shift towards more flexible, adaptive trading strategies empowered by AI in financial applications.