Impact of encoder-only versus decoder-only LLM text representations on forecasting performance

Determine how text representations produced by encoder-only large language models such as DeBERTa compared with decoder-only large language models such as Mistral and Llama3 affect the accuracy of stock return forecasting when the models are fine-tuned end-to-end on concatenated financial news sequences for n-step forward return prediction.

Background

The paper fine-tunes LLMs to predict forward stock returns directly from financial newsflow, positioning the LLM as a text representation module that feeds into a forecasting module.

Encoder-only models (e.g., DeBERTa) and decoder-only models (e.g., Mistral, Llama3) differ in pre-training objectives (masked-language modeling vs autoregressive next-token prediction), which can yield different token- and sequence-level representations.

The authors highlight that how these representational differences translate into forecasting performance in stock return prediction is not yet established, motivating a systematic comparison.