Revisiting Ensemble Methods for Stock Trading and Crypto Trading Tasks at ACM ICAIF FinRL Contest 2023-2024

Abstract: Reinforcement learning has demonstrated great potential for performing financial tasks. However, it faces two major challenges: policy instability and sampling bottlenecks. In this paper, we revisit ensemble methods with massively parallel simulations on graphics processing units (GPUs), significantly enhancing the computational efficiency and robustness of trained models in volatile financial markets. Our approach leverages the parallel processing capability of GPUs to significantly improve the sampling speed for training ensemble models. The ensemble models combine the strengths of component agents to improve the robustness of financial decision-making strategies. We conduct experiments in both stock and cryptocurrency trading tasks to evaluate the effectiveness of our approach. Massively parallel simulation on a single GPU improves the sampling speed by up to $1,746\times$ using $2,048$ parallel environments compared to a single environment. The ensemble models have high cumulative returns and outperform some individual agents, reducing maximum drawdown by up to $4.17\%$ and improving the Sharpe ratio by up to $0.21$. This paper describes trading tasks at ACM ICAIF FinRL Contests in 2023 and 2024.

• The paper demonstrates that combining ensemble learning with massively parallel GPU simulations significantly enhances reinforcement learning performance in financial trading.
• It details how diversity enhancement via KL divergence and weighted or majority voting strategies mitigates policy instability and sampling bottlenecks.
• Experiments reveal higher Sharpe ratios, reduced drawdowns, and improved cumulative returns, underscoring the practical viability of the approach in volatile markets.

Revisiting Ensemble Methods for Stock Trading and Crypto Trading Tasks

Introduction

The paper discusses the challenges and opportunities in applying reinforcement learning (RL) to financial tasks, particularly stock and cryptocurrency trading. RL, despite its potential for financial applications, suffers from policy instability and sampling bottlenecks. The authors propose the use of ensemble methods, combined with massively parallel simulations on GPUs, to address these issues effectively. The primary goal is to enhance computational efficiency and model robustness in volatile financial markets.

Challenges in Financial Reinforcement Learning

The authors identify two critical challenges in applying RL to finance:

• Policy Instability: Due to value function approximation errors and sensitivity to hyperparameters, RL models often suffer from instability, which can significantly affect their performance.
• Sampling Bottleneck: Complex financial tasks require massive data sampling, which can be inefficient on standard CPUs due to the high computational demands.

To tackle these problems, the paper revisits ensemble methods and employs massively parallel simulations to improve sampling speed and model robustness. By running simulations on GPUs, the authors achieve substantial performance gains, making the approach well-suited for high-frequency financial data.

Ensemble Learning Approach

The ensemble approach combines multiple RL agents, each leveraging its unique strengths to form a robust decision-making model. It mitigates individual weaknesses, leading to improved overall performance:

• Diversity Enhancement: Using KL divergence, the training loss function penalizes similarities among the component agents, promoting diverse trading strategies.
• Weighted Ensemble for Stock Trading: Combines PPO, SAC, and DDPG agents by averaging action probabilities weighted by Sharpe ratios, ensuring adaptability to market changes.
• Majority Voting for Crypto Trading: Employs DQN, Double DQN, and Dueling DQN in a discrete action space, achieving quick response times and lower latency, crucial for high-frequency crypto markets.

  Figure 1: Performance deviation for different RL algorithms and a simple ensemble method.

Massively Parallel Simulation

The paper highlights the benefits of massively parallel simulations:

• Sampling Efficiency: By simulating 2,048 parallel market environments on a single GPU, the sampling speed improves up to $1,746\times$ compared to a single environment.
• Data Management: Samples are efficiently organized in tensors stored on GPU memory, reducing communication bottleneck between CPU and GPU, crucial for high-throughput data processing.

  Figure 2: Producer-Consumer model for RL.

Performance Evaluation

Experiments conducted in both stock and cryptocurrency markets reveal that ensemble models outperform individual agents on various performance metrics, including cumulative return and Sharpe ratio. Important findings include:

• Stock Trading: Ensemble models show high returns and reduced maximum drawdown, indicating robustness in volatile markets.
• Cryptocurrency Trading: Ensemble methods achieve a higher Sharpe ratio and better risk management compared to individual agents, demonstrating the effectiveness of the majority voting mechanism.

  Figure 3: Samples per second for the stock trading task and the cryptocurrency trading task. NVIDIA A100 GPU is used.

  

  Figure 4: Cumulative returns of different strategies for the stock trading task and cryptocurrency trading task.

Reflections on Practical Implications

The paper's integration into the ACM ICAIF FinRL contests elevates its real-world applicability. The use of ensemble methods in live competitions underscores a practical adoption path, while the potential extension to zero-knowledge proofs (ZKPs) for secure model validation offers innovative collaboration frameworks between model producers and institutional funds. This approach not only promises enhanced security for trading algorithms but also fosters trust and transparency in financial model applications.

Conclusion

By combining ensemble methods with massively parallel simulations on GPUs, the paper successfully addresses significant challenges in financial RL. The results highlight improved model performance and robustness, paving the way for applying these advanced techniques across various financial markets. Future work may explore broader financial instruments and integrate real-time market conditions, potentially leveraging secure validation frameworks like zero-knowledge proofs for further advancements.