LoRA-adapted GPT-DT for Offline RL

• The paper demonstrates that DT-LoRA-GPT2 achieves competitive returns and Sharpe ratios by fine-tuning only efficient LoRA adapters.
• The model freezes GPT-2 weights and updates just 0.7% of parameters, reducing overfitting and computational cost.
• Empirical evaluations highlight improved sample efficiency and reproducibility, outperforming standard offline RL baselines in quantitative trading.

LoRA-adapted GPT-DT (DT-LoRA-GPT2) is a Decision Transformer architecture for offline reinforcement learning (RL), specifically offline quantitative trading, which leverages a pretrained GPT-2 backbone with parameter-efficient Low-Rank Adaptation (LoRA) to enable effective policy learning from historical expert trajectories. By freezing all pretrained weights and restricting fine-tuning to LoRA-injected adapters, this approach achieves strong sample efficiency, minimizes overfitting, and secures competitive or superior results compared to existing offline RL baselines, with significant reductions in trainable parameter count and computational overhead (Yun, 2024).

1. Model Architecture and LoRA Mechanism

DT-LoRA-GPT2 employs GPT-2-small (12 layers, 12 heads, hidden size 768) as its frozen transformer backbone. Reinforcement learning trajectories are reshaped into input token sequences of the form

$$(\hat R_{t-K+1},\,s_{t-K+1},\,a_{t-K+1},\,\dots,\,\hat R_t,\,s_t,\,a_t)$$

where $\hat R_\tau$ is the return-to-go, $s_\tau$ denotes market state features, and $a_\tau$ portfolio actions. All modalities are projected into $\mathbb{R}^{768}$ via small residual MLPs and positional embeddings, then LayerNorm is applied.

The core adaptation method is LoRA, which augments the self-attention query, key, and value projections with trainable low-rank matrices: $$W' = W_0 + \Delta W, \qquad \Delta W = B A, \quad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},\; r\ll \min(d, k)$$ For DT-LoRA-GPT2, LoRA rank is set to $r=16$, yielding approximately 900,000 trainable parameters ($\approx$0.7% of GPT-2-small). Only these adapters, as well as embedding MLPs, are updated during fine-tuning; all GPT-2 weights are frozen.

The decoder operates autoregressively, using a causal attention mask automatically aligned to the $(\hat R, s, a)$ token order to prevent information leakage from future timesteps. The prediction head is an MLP mapping the embedding of the final state token to the continuous action space.

2. RL Objective and Offline Data Pipeline

DT-LoRA-GPT2 is trained under a behavioral cloning regime with mean squared error (MSE) loss over continuous action predictions: $$L = \frac{1}{N T}\sum_{n=1}^N \sum_{t=1}^T \|a_t^{(n)} - \hat a_t^{(n)}\|^2$$ where $\hat R_\tau$0 are expert actions and $\hat R_\tau$1 are model predictions. RL returns-to-go $\hat R_\tau$2 are computed exactly.

The offline dataset consists of daily trajectories collected from expert RL algorithms (A2C, PPO, SAC, TD3, DDPG) on DJIA stocks between 2009–2020. States are computed from normalized prices and industry-standard technical indicators; actions are 29-dimensional portfolio weights in $\hat R_\tau$3. Rewards are daily returns clipped to $\hat R_\tau$4. Data is segmented into sliding windows of length $\hat R_\tau$5 (sequence length 60). Trajectories shorter than $\hat R_\tau$6 are masked with zero-padding.

3. Comparative Performance and Ablation Analysis

Evaluation metrics include total return percent, maximum drawdown, and Sharpe ratio. Baselines comprise Behavior Cloning (BC), Implicit Q-Learning (IQL), Conservative Q-Learning (CQL), and a Decision Transformer with randomly initialized GPT-2 plus LoRA (parameters matched at 900 K).

Empirical highlights:

• On TD3 trajectories, DT-LoRA-GPT2 achieves 46.62% return (vs. 43.37% random-init DT and 45.05% CQL) and matched-best Sharpe (2.14).
• On SAC data, DT-LoRA-GPT2 achieves best Sharpe (1.69) and drawdown (–8.54%).
• Across all experts, pre-trained DT-LoRA-GPT2 outperforms the random-init counterpart by 3–5 percentage points in return and 0.05–0.1 in Sharpe.
• Pretraining gains are statistically significant at $\hat R_\tau$7 over five seeds.
• Ablations confirm diminishing returns for LoRA ranks $\hat R_\tau$8; using only 6 GPT-2 layers degrades return by approximately 2 percentage points.

Regularization includes weight decay ($\hat R_\tau$9), early stopping by validation-set MSE (patience 50), and LoRA’s intrinsic low-rank constraint.

4. Practical Training Guidelines and Reproducibility

Recommended hyperparameters:

• Optimizer: Adam; learning rate $s_\tau$0 for LoRA and embedding MLPs.
• Batch size: 64 trajectories; context length $s_\tau$1 (60 tokens).
• Training: 1,000 iterations (circa 3 epochs across the dataset).

All code, data, splits, seeds, and baseline scripts are provided at https://github.com/syyunn/finrl-dt, ensuring full replicability. To reproduce results, the provided scripts allow switching among expert policies and direct evaluation.

5. Parameter Efficiency, Regularization, and LoRA Extensions

DT-LoRA-GPT2 exemplifies extreme parameter efficiency characteristic of LoRA methods (Hu et al., 2021). By freezing the full GPT-2 model (124M parameters) and training only 0.7% of parameters, substantial reductions in memory and compute demand are realized with no inference latency penalty. LoRA adapters effectively regularize the adaptation process, reducing overfitting in the low-data regime.

Recent LoRA variants, including Dual LoRA (Xu et al., 3 Dec 2025) and LoRTA (Hounie et al., 2024), further generalize this parameter-efficient paradigm. Dual LoRA decomposes parameter updates into separate low-rank magnitude and direction factors—using elementwise ReLU and Sign, respectively—achieving higher effective rank and closer emulation of full fine-tuning. LoRTA applies tensor CP decomposition across layers, heads, and projections, minimizing redundancy and potentially cutting adapter parameter count by additional orders of magnitude while retaining adaptation quality. For model developers targeting even greater efficiency, these extensions are directly applicable to the Decision Transformer architecture.

6. Context within Offline RL and Quantitative Trading

The LoRA-adapted GPT-DT model situates itself among sequence-model–based offline RL methods for quantitative finance. Decision Transformers provide a unified autoregressive approach to trajectory modeling; leveraging advanced LLMs allows for better generalization across long temporal horizons observed in financial time series. DT-LoRA-GPT2 surpasses conventional RL baselines and non-pretrained Decision Transformers by exploiting LLM priors and restricting adaptation via LoRA to guard against overfitting.

This architecture is particularly apt for environments—such as financial markets—where online exploration is infeasible or risky, and large repositories of expert-annotated demonstrations are available. Its empirical results and reproducibility profile make it a reference implementation for parameter-efficient decision transformer deployment in finance and related time-series domains.

7. Limitations, Extensions, and Open Challenges

While DT-LoRA-GPT2 demonstrates robust performance and parameter efficiency, limitations persist:

• Performance is bounded by the expressivity of the low-rank adaptation and quality/diversity of expert trajectories.
• Diminishing returns from increasing LoRA rank indicate a practical ceiling for adapter-based fine-tuning.
• The model currently applies adapters only to attention projections; extending to MLP layers or adopting advanced tensorized PEFT (e.g., LoRTA) could further reduce parameter requirements or enhance adaptation.
• As with all offline RL models, generalization to non-stationary or adversarial regimes not observed in the historical data is untested.

Recent advancements in PEFT (LoRA, Dual LoRA, LoRTA) supply a modular toolkit for adapting large Decision Transformer backbones to novel reward profiles and market conditions with minimal retraining cost. The parameter-efficient adaptation framework established by DT-LoRA-GPT2 is a foundation for future work in scaling offline RL to high-frequency, multi-asset, and cross-market trading scenarios (Yun, 2024, Hu et al., 2021, Xu et al., 3 Dec 2025, Hounie et al., 2024).