QuantFactor REINFORCE: Mining Steady Formulaic Alpha Factors with Variance-bounded REINFORCE (2409.05144v3)

Abstract: Alpha factor mining aims to discover investment signals from the historical financial market data, which can be used to predict asset returns and gain excess profits. Powerful deep learning methods for alpha factor mining lack interpretability, making them unacceptable in the risk-sensitive real markets. Formulaic alpha factors are preferred for their interpretability, while the search space is complex and powerful explorative methods are urged. Recently, a promising framework is proposed for generating formulaic alpha factors using deep reinforcement learning, and quickly gained research focuses from both academia and industries. This paper first argues that the originally employed policy training method, i.e., Proximal Policy Optimization (PPO), faces several important issues in the context of alpha factors mining. Herein, a novel reinforcement learning algorithm based on the well-known REINFORCE algorithm is proposed. REINFORCE employs Monte Carlo sampling to estimate the policy gradient-yielding unbiased but high variance estimates. The minimal environmental variability inherent in the underlying state transition function, which adheres to the Dirac distribution, can help alleviate this high variance issue, making REINFORCE algorithm more appropriate than PPO. A new dedicated baseline is designed to theoretically reduce the commonly suffered high variance of REINFORCE. Moreover, the information ratio is introduced as a reward shaping mechanism to encourage the generation of steady alpha factors that can better adapt to changes in market volatility. Evaluations on real assets data indicate the proposed algorithm boosts correlation with returns by 3.83\%, and a stronger ability to obtain excess returns compared to the latest alpha factors mining methods, which meets the theoretical results well.

• The paper introduces the QFR algorithm, which replaces the actor-critic framework with a variance-bounded REINFORCE approach to enhance interpretability and convergence.
• The paper demonstrates that QFR achieves a 3.83% improvement in Rank IC and superior cumulative returns in real trading simulations compared to methods like PPO.
• The paper provides theoretical guarantees by using a greedy baseline and IR reward shaping to substantially reduce gradient variance, ensuring robust policy updates.

QuantFactor REINFORCE: Mining Steady Formulaic Alpha Factors with Variance-bounded REINFORCE

The paper "QuantFactor REINFORCE: Mining Steady Formulaic Alpha Factors with Variance-bounded REINFORCE" introduces an innovative approach to mining interpretable formulaic alpha factors using reinforcement learning (RL). This research addresses the limitations of existing methods that generate alpha factors either as complex, non-interpretable deep models or through heuristic and tree-based methods that suffer from performance and exploration limitations.

Proposed Approach

The authors propose a novel RL algorithm termed QuantFactor REINFORCE (QFR) that leverages the well-known REINFORCE algorithm for policy optimization. They identify several critical issues with the Proximal Policy Optimization (PPO) method used in prior works, specifically its inefficacy in handling the specific characteristics of the alpha factor mining task, such as deterministic state transitions and trajectory-level rewards. To rectify this, the QFR algorithm introduces the following key innovations:

1. Abandoning the Actor-Critic Framework: By discarding the value network related to the actor-critic model, QFR circumvents the inefficiencies associated with value network sampling in PPO, enabling faster convergence.
2. Greedy Baseline to Reduce Variance: To mitigate the high variance problem inherent in the REINFORCE algorithm, the QFR algorithm employs a novel baseline based on a greedy policy that stabilizes the policy gradient estimate.
3. Information Ratio (IR) Reward Shaping: The paper introduces IR as a supplementary reward mechanism. This mechanism steers the policy network to produce alpha factors that not only predict returns accurately but also maintain stability against market volatility.

Experimental Evaluation

The authors conduct extensive experiments on real asset data comprising stocks from CSI300, CSI500, CSI1000, SPX, DJI, and NDX indices. The experimental setup rigorously evaluates the performance of QFR by comparing it with various state-of-the-art RL algorithms (PPO, TRPO, A3C) and traditional alpha factor mining methods.

Key findings from the experiments are highlighted as follows:

• Enhanced Convergence and Performance: QFR significantly outperforms traditional RL algorithms in the alpha factor mining task, showing a 3.83% improvement in Rank Information Coefficient (Rank IC) over PPO, which is evidence of its better convergence properties.
• Outperformance in Real Trading Simulations: The cumulative return achieved using the factors generated by QFR is consistently superior compared to other methods across different volatile market conditions. This underscores the practical applicability of QFR-generated alpha factors in dynamic trading environments.

Theoretical Contributions

The paper provides a theoretical analysis substantiating the efficacy of QFR. The authors derive upper bounds on the variance of the gradient estimates for QFR, demonstrating that its variance is significantly less than REINFORCE when dealing with deterministic state transitions. The analysis confirms that greedy baseline stabilization effectively reduces variance, ensuring more robust and stable policy updates.

Implications and Future Directions

The primary contribution of this research is the development of a more efficient and stable methodology for mining interpretable formulaic alpha factors using RL. This advancement holds practical value in quant finance sectors, enabling the generation of sophisticated, yet transparent trading signals that can adapt to fluctuating market conditions.

Potential future work could expand this framework by exploring other sophisticated reward shaping mechanisms or adapting the QFR approach to additional financial applications, such as portfolio optimization and risk management. Moreover, integrating this RL-based approach with other machine learning techniques may further enhance the algorithm’s robustness and effectiveness in various financial markets.

Conclusion

In summary, "QuantFactor REINFORCE" presents a significant methodological improvement for mining formulaic alpha factors. The proposed QFR algorithm addresses critical limitations of prior methods, exhibits superior performance in practical applications, and offers theoretical backing for its variance reduction and efficiency. This research paves the way for more transparent and effective RL-driven solutions in the domain of quantitative finance.