FinRL-DeepSeek: LLM-Infused Risk Trading

• FinRL-DeepSeek is a hybrid reinforcement learning framework that integrates CPPO risk adjustments with LLM-derived financial news signals for advanced trading strategies.
• It employs a modular three-layer FinRL architecture to simulate markets, aggregate multi-modal data, and deploy deep RL algorithms for dynamic decision-making.
• The system incorporates a meta-search component (DeepSeek) for optimal feature and hyperparameter tuning, improving model calibration and risk management.

FinRL-DeepSeek is a hybrid reinforcement learning (RL) framework for quantitative trading that integrates risk-sensitive policy optimization with real-time LLM derived signals, specifically leveraging financial news for both trading recommendations and risk assessment. It extends the FinRL and FNSPID research lines and serves as a reference implementation of LLM-infused RL agents for financial tasks, with applications to both single-agent trading and retrieval-augmented event-driven prediction (Liu et al., 2021, Benhenda, 11 Feb 2025, Zhao et al., 29 May 2026).

1. Core FinRL Framework and Layered Architecture

FinRL provides a modular three-layer pipeline—environment, feature, and agent layers—each configurable for practical trading and research. The environment ingests historical and live market data (e.g., OHLCV, corporate fundamentals, alternative data) via modular connectors (Yahoo Finance, Alpaca, CCXT, and others), and simulates market mechanics (transaction costs, liquidity, bid-ask spread). Time-driven OpenAI Gym-style APIs enable fine-grained simulation at user-selected granularities (minute, hourly, daily).

The feature layer assembles composite states encompassing both agent-level (portfolio allocations, account balances) and market-level (price vectors, technical/fundamental indicators, alternative signals including LLM-derived news sentiment or risk scores). Reward shaping and constraint hooks—such as turbulence indices for risk control—permit the construction of custom reward functions and the enforcement of market-relevant constraints.

The agent layer supports out-of-the-box integration of libraries such as Stable-Baselines3, RLlib, and ElegantRL, giving access to deep RL algorithms including DQN, DDPG, PPO, A2C/A3C, TD3, and SAC. The unified API allows for environment instantiation, model selection (e.g., PPO, CPPO), training, evaluation, and straightforward extensibility for custom architectures (Liu et al., 2021).

2. Risk-Sensitive RL via CPPO and Integration of LLM Signals

FinRL-DeepSeek introduces a risk-adjusted variant of Proximal Policy Optimization (PPO) called Conditional Value-at-Risk Proximal Policy Optimization (CPPO). The PPO surrogate objective

$$L_{\mathrm{PPO}}(\theta) = \mathbb{E}_t\left[\min(r_t(\theta)A_t, \mathrm{clip}(r_t(\theta),1-\epsilon,1+\epsilon)A_t)\right]$$

is augmented by a tail-risk control Lagrangian: $$L_{\mathrm{CPPO}}(\theta, \eta, \lambda) = L_{\mathrm{PPO}}(\theta) + \lambda \left[ \frac{1}{1-\alpha}\,\mathbb{E}[(\eta - D(\pi_\theta))^+] - \eta + \beta \right]$$ where $D(\pi_\theta)$ is the total trajectory return, $\alpha$ is the risk quantile, $\beta$ the allowed CVaR, and $\lambda$ is a Lagrange multiplier. The algorithm alternates between stochastic gradient steps on $\theta$, minimization over $\eta$, and $\lambda$ ascent (Benhenda, 11 Feb 2025).

LLM-generated signals enter the RL loop as follows:

• Recommendation inputs: Each financial news article per stock per day (from the FNSPID dataset) is processed by an LLM (DeepSeek V3, Qwen 2.5 72B, Llama 3.3 70B) via explicit prompts for recommendation and risk. For recommendations, the model outputs a score $S_f\in\{1,2,3,4,5\}$, mapped to a multiplicative action perturbation ($$L_{\mathrm{CPPO}}(\theta, \eta, \lambda) = L_{\mathrm{PPO}}(\theta) + \lambda \left[ \frac{1}{1-\alpha}\,\mathbb{E}[(\eta - D(\pi_\theta))^+] - \eta + \beta \right]$$0), which scales the agent’s action intensity: $$L_{\mathrm{CPPO}}(\theta, \eta, \lambda) = L_{\mathrm{PPO}}(\theta) + \lambda \left[ \frac{1}{1-\alpha}\,\mathbb{E}[(\eta - D(\pi_\theta))^+] - \eta + \beta \right]$$1.
• Risk assessment inputs: Risk scores $$L_{\mathrm{CPPO}}(\theta, \eta, \lambda) = L_{\mathrm{PPO}}(\theta) + \lambda \left[ \frac{1}{1-\alpha}\,\mathbb{E}[(\eta - D(\pi_\theta))^+] - \eta + \beta \right]$$2 are computed per asset. The aggregate, portfolio-weighted risk factor $$L_{\mathrm{CPPO}}(\theta, \eta, \lambda) = L_{\mathrm{PPO}}(\theta) + \lambda \left[ \frac{1}{1-\alpha}\,\mathbb{E}[(\eta - D(\pi_\theta))^+] - \eta + \beta \right]$$3 is used to inflate or deflate realized rewards; i.e., $$L_{\mathrm{CPPO}}(\theta, \eta, \lambda) = L_{\mathrm{PPO}}(\theta) + \lambda \left[ \frac{1}{1-\alpha}\,\mathbb{E}[(\eta - D(\pi_\theta))^+] - \eta + \beta \right]$$4 enters the CPPO tail-penalty term. High risk inflates the loss attributed to the negative tail, steering optimization toward risk aversion.

This mechanism is designed to infuse real-time, event-conditioned market signals produced by LLMs directly into the RL policy and value update, providing a performance and stability margin in adverse market regimes (Benhenda, 11 Feb 2025).

3. DeepSeek Meta-Search Extension

A key extension, "DeepSeek" (Editor's term), proposes meta-search over feature selection, hyperparameter tuning, and reward design within the FinRL ecosystem. DeepSeek’s controller—a compact RNN or transformer—outputs sampling distributions over feature subsets and hyperparameters. Each sample defines a distinct FinRL experiment (feature set, agent configuration, reward form). On each iteration, a child agent is trained and evaluated, with validation Sharpe ratio used as the meta-reward. Controller parameters are updated (e.g., via REINFORCE) to maximize expected Sharpe. Convergence yields optimal feature-hyperparameter configurations for downstream deployment. DeepSeek interfaces only at the data, feature, and agent layers—core FinRL code is unmodified—and outputs best configurations as structured JSON for reproducible workflow integration (Liu et al., 2021).

4. FinRL-DeepSeek/FNSPID Dataset and Event-Driven RAG

FinRL-DeepSeek/FNSPID benchmarks use a fixed universe of 89 Nasdaq-oriented stocks, with news-event anchoring and multi-modal evidence streams (15.7 million timestamped news, 29.7 million daily prices, SEC filings). For RAG-style event-driven prediction and trading, each news anchor is paired point-in-time with eligible evidence: original, deduplicated news items and EDGAR-segmented filings, filtered strictly both temporally and by sectoral caps. Market context (momentum, volatility, ETF returns) is appended per anchor. Online evaluation splits and label/feature construction mirror real-time finance workflows: models are never permitted leak-susceptible or post-hoc data (Zhao et al., 29 May 2026).

5. Bayesian Source-Memory for Event-Driven RAG

Extensions to retrieval-augmented generation (RAG) in the FinRL-DeepSeek context utilize Bayesian source-memory adaptation. For each news-event anchor, the retrieval pool (BM25 plus dense retriever, normalized and capped) is reranked not only by textual relevance but by learned market-feedback, with separate memory cells for (source family, event type, forecast horizon). After financial horizons mature, actual market impact is compared to LLM prediction, and source-memory parameters (Naïve Bayes-Beta posterior, with shrinkage and bounded gain) are updated via fractional, class-balanced increments.

The reranking score per candidate becomes: $$L_{\mathrm{CPPO}}(\theta, \eta, \lambda) = L_{\mathrm{PPO}}(\theta) + \lambda \left[ \frac{1}{1-\alpha}\,\mathbb{E}[(\eta - D(\pi_\theta))^+] - \eta + \beta \right]$$5 with $$L_{\mathrm{CPPO}}(\theta, \eta, \lambda) = L_{\mathrm{PPO}}(\theta) + \lambda \left[ \frac{1}{1-\alpha}\,\mathbb{E}[(\eta - D(\pi_\theta))^+] - \eta + \beta \right]$$6 as tuning hyperparameters, and utility mean $$L_{\mathrm{CPPO}}(\theta, \eta, \lambda) = L_{\mathrm{PPO}}(\theta) + \lambda \left[ \frac{1}{1-\alpha}\,\mathbb{E}[(\eta - D(\pi_\theta))^+] - \eta + \beta \right]$$7 reflecting actual outcome alignment per evidence source. This adaptive mechanism demonstrably raises event-level Macro-F1 (from 0.438 to 0.471) and downstream cost-adjusted Sharpe (from 0.52 to 0.84) on held-out test sets, outperforming LoRA-adapted reader models and static retrieval (Zhao et al., 29 May 2026).

6. Backtesting Results, Performance, and Insights

Extensive backtesting (2013–2023) demonstrates that CPPO+DeepSeek architectures achieve positive information ratios and tighter tail losses during bear markets, with LLM infusion showing marginal or negative effects for PPO (risk-blind) but significant gains for CPPO (risk-aware). Tail-risk measures (CVaR, Rachev Ratio), Macro-F1, non-neutral F1, and realized Sharpe are standard evaluation metrics.

Empirical findings include:

• CPPO + DeepSeek produces favorable risk-return tradeoffs even under adverse market conditions.
• Stronger dependence (up to 10%) on LLM-based action/reward perturbations helps CPPO but can degrade PPO.
• LLM-derived risk inputs effectively penalize adverse market behaviors without requiring explicit encoder retraining.
• Bayesian source-memory for event-driven RAG yields larger improvements than incremental LLM fine-tuning on noisy return labels (Zhao et al., 29 May 2026, Benhenda, 11 Feb 2025).

7. Limitations, Implementation, and Prospects

Current FinRL-DeepSeek pipelines are computationally demanding (up to 128 GB RAM for 2 million CPPO steps), with significant LLM inference latency and cost. Single-article-per-day sampling constrains sensitivity to intraday event cascades. No formal tests of statistical significance have been published. Memory-efficient RL and intraday extensions are significant future directions; advanced LLM architectures or distilled task-specific summarizers may further reduce costs.

A plausible implication is that the modular FinRL-DeepSeek formulation, when fused with source-adaptive event retrieval and risk-aware RL, establishes a performance ceiling for LLM-driven trading agents in contemporary quantitative finance research. Reproducibility and extension via meta-learning controllers (DeepSeek) and event-driven source memory are central to ongoing advances in this domain (Liu et al., 2021, Benhenda, 11 Feb 2025, Zhao et al., 29 May 2026).