Impact of encoder-only versus decoder-only LLM representations on stock return forecasting

Determine how the text representations produced by encoder-only large language models such as DeBERTa, compared to those produced by decoder-only large language models such as Mistral and Llama, affect stock return forecasting performance when fine-tuned to predict forward returns from concatenated financial news sequences.

Background

The paper fine-tunes LLMs to predict forward stock returns directly from financial news, replacing traditional feature engineering with learned text representations.

Encoder-only (e.g., DeBERTa) and decoder-only (e.g., Mistral, Llama) models differ in their pre-training objectives (masked language modeling versus autoregressive next-token prediction), which leads to distinct token-level embeddings and potentially different sequence representations for forecasting.

The paper compares these families of models as the text representation module within a news-to-return forecasting framework, but explicitly notes the uncertainty about how these different representations impact forecasting performance.