LLM-Infused Risk-Sensitive RL

• LLM-infused risk-sensitive RL is the integration of LLM-derived financial news signals into reinforcement learning algorithms using CPPO for dynamic risk modulation.
• The approach fuses action perturbations and risk assessments derived from financial data to adjust policy actions and returns for enhanced market responsiveness.
• Empirical evaluations demonstrate improved returns and risk management in volatile conditions, highlighting the importance of careful calibration and signal fidelity.

LLM-infused risk-sensitive reinforcement learning refers to the integration of LLM-derived signals—specifically financial news-based recommendations and risk assessments—into risk-sensitive reinforcement learning (RL) algorithms, with Conditional Value-at-Risk Proximal Policy Optimization (CPPO) representing the canonical framework. The central thesis is that natural language data, parsed and distilled by state-of-the-art LLMs, can be formally incorporated as additional control and risk-modulation signals, yielding enhanced agent performance under adverse market conditions. The FinRL-DeepSeek framework exemplifies this paradigm and provides systematic methodology, empirical evaluation, and analysis for LLM-driven RL trading agents (Benhenda, 11 Feb 2025).

1. Algorithmic Foundation: CPPO and LLM Signal Fusion

The baseline is the Proximal Policy Optimization (PPO) algorithm, with clipped objective:

$$L_{ppo}(\theta) = \mathbb{E}_t \left[ \min\Big( r_t(\theta)A_t, \ \mathrm{clip}(r_t(\theta),1-\epsilon,1+\epsilon)A_t \Big) \right]$$

where $$r_t(\theta) = \pi_{\theta}(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)$$.

Risk-sensitivity is incorporated via CPPO, using Conditional Value-at-Risk (CVaR) constraints. Let $D(\pi_\theta)$ denote the discounted trajectory return. At confidence level $\alpha$,

$$\text{CVaR}_\alpha[D] = \eta + \frac{1}{1-\alpha} \mathbb{E}\left[(D-\eta)^-\right], \quad (x)^- = \max(0, -x)$$

The Lagrangian loss for CPPO is:

$$L_{CPPO}(\theta, \eta, \lambda) = L_{ppo}(\theta) + \lambda \Big[ \frac{1}{1-\alpha} \mathbb{E}[(\eta - D(\pi_\theta))^+] - \eta + \beta \Big]$$

LLM infusion introduces two signals per stock per day, derived from financial news:

• Recommendation score $S_f^i \in [0.9, 1.1]$: Encodes buy/sell preference as a multiplicative action perturbation.
• Risk assessment $R_f^i \in [0.9, 1.1]$: Encodes tail-risk sentiment.

These are integrated as:

• Action fusion (PPO step): $a_t' = S_f \cdot a_t$
• Return fusion (CPPO step): $R_f = \sum_i w_i R_f^i$ with $$r_t(\theta) = \pi_{\theta}(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)$$0; $$r_t(\theta) = \pi_{\theta}(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)$$1

This yields the LLM-extended objective,

$$r_t(\theta) = \pi_{\theta}(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)$$2

The principal training loop alternates stochastic gradient steps for the policy network and dual-ascent steps for the CVaR penalty multipliers. The fusion preserves PPO stability with $$r_t(\theta) = \pi_{\theta}(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)$$3 and enables risk modulation via $$r_t(\theta) = \pi_{\theta}(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)$$4.

2. LLM-Based Signal Generation from Financial News

LLM signals are extracted from financial news using the FNSPID dataset (2M articles, one per stock/day). Three LLMs—DeepSeek V3, Qwen 2.5, and Llama 3.3—are queried per instance with engineered prompts:

• Stock recommendation: "Score 1–5, 1=very negative…5=very positive"
• Risk assessment: "Score 1–5, 1=very low risk…5=very high risk"

Prompted texts are preprocessed (prompt as prefix; short completion), and post-processed by regex to extract the first integer in $$r_t(\theta) = \pi_{\theta}(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)$$5. The result is mapped via a linear scale: $$r_t(\theta) = \pi_{\theta}(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)$$6, $$r_t(\theta) = \pi_{\theta}(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)$$7, $$r_t(\theta) = \pi_{\theta}(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)$$8, $$r_t(\theta) = \pi_{\theta}(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)$$9, $D(\pi_\theta)$0.

This explicit and fixed mapping from discrete LLM output to continuous scaling is crucial for model stability.

3. Architecture and Training Paradigm

State vectors $D(\pi_\theta)$1 (comprising recent price windows and technical indicators) are concatenated with LLM-derived features $D(\pi_\theta)$2 (all $D(\pi_\theta)$3 and $D(\pi_\theta)$4 for the equity universe). The resulting input is processed by a unified neural network with fully connected layers ([128 → ReLU → 64 → ReLU]), outputting both the policy logits $D(\pi_\theta)$5 and the state value $D(\pi_\theta)$6.

Key training protocols:

• Data split: Training on 2013–2018 (or 2019–2022); backtest on 2019–2023 or calendar 2023.
• Hyperparameters: Learning rates $D(\pi_\theta)$7 (policy), $D(\pi_\theta)$8 (dual variables); batch size 64; CVaR level $D(\pi_\theta)$9; PPO clip $\alpha$0; discount $\alpha$1.
• Infusion: LLM signals are fused only at the final trajectory level, with no reward shaping at intermediate steps.
• Early stopping is based on out-of-sample CVaR or Sharpe ratio after 400–500K steps.

4. Empirical Evaluation and Quantitative Results

Backtesting is conducted on Nasdaq-100 constituents with daily rebalancing. Performance is benchmarked using three key metrics: Information Ratio, CVaR (5%), and Rachev Ratio. Results after 100 epochs (≈2M steps) are summarized as:

Model	Information Ratio	CVaR (5%)	Rachev Ratio
PPO	0.0100	–3.94%	1.0637
CPPO	–0.0148	–4.39%	1.0404
PPO-DeepSeek	–0.0093	–3.38%	0.9890
CPPO-DeepSeek	0.0078	–4.37%	0.9818

Out-of-sample equity curves indicate:

• PPO agents with LLM recommendations demonstrate higher returns in bull regimes, albeit at greater volatility.
• CPPO-DeepSeek achieves parity with PPO in bullish markets and exhibits outperformance in bearish conditions (notably post end-2021).

Ablation studies on LLM infusion strength (0.1%–10% perturbations) show:

• Excessive action perturbation degrades PPO returns; minimal fusion is less destructive.
• CPPO-DeepSeek achieves best Information Ratio with the strongest (10%) LLM infusion.

5. Portfolio Dynamics, Limitations, and Sensitivity

LLM-driven recommendation multipliers ($\alpha$2) can amplify the agent’s net market exposure—moving in sync with high-confidence language signals. Risk multipliers ($\alpha$3) dynamically scale the CVaR penalty, prompting CPPO-DeepSeek to reallocate capital into assets with more favorable risk profiles during periods of elevated tail risk.

Observed effects include:

• Defensive posture in downturns, with substantial drawdown reduction.
• Responsiveness to rapid shifts in news sentiment.

However, several limitations are acknowledged:

• The fidelity of LLM output is pivotal; poor or inconsistent scoring (especially for PPO-DeepSeek) introduces noise and can damage performance.
• RAM requirements are substantial (e.g., 2M steps ≈ 128GB), notably due to unrolled trajectory buffers.
• Sensitivity is observed with respect to prompt design and the post-processing mapping from scores to multipliers.

6. Deployment Considerations and Practical Implications

Real-time application of LLM-infused RL is subject to operational constraints:

• LLM inference latency and associated API costs necessitate caching strategies or the deployment of local, lightweight LLMs.
• A plausible implication is that reducing the decision timescale (e.g., moving to intraday frequencies) could improve timeliness of responses, but would require highly efficient LLM pipelines for actionable latency.

The empirical findings suggest that LLM-augmented, risk-sensitive RL agents can drive measurable improvements in financial trading, particularly during adverse market conditions. Nevertheless, model stability, signal fidelity, and computational logistics represent persistent challenges that necessitate further research and applied engineering.