Extent of memorization effects in LLM-based embedding forecasts

Determine the extent to which memorization affects forecasts that rely on embeddings generated by large language models such as GPT-4o in financial prediction tasks, by quantifying how much embedding representations encode training-period outcomes or lookahead information that contaminates forecasting evaluations.

Background

The paper surveys approaches that use LLM-derived embeddings combined with supervised steps for financial prediction. While embeddings have been proposed as a powerful representation for return forecasting, the authors point out that the degree to which memorization contaminates such forecasts during the training period is not established.

Given the broader evidence of selective memorization across economic indicators, indices, and textual data, the unknown extent of contamination in embedding-based forecasts poses a risk to interpreting apparent predictive performance as genuine insight rather than recall.

References

Finally, using embeddings along with a supervised step has been proposed by \citet{chenExpectedReturnsLarge2022}, though it remains unknown to what extent the memorization problem affects forecasts using LLMs embeddings.

— The Memorization Problem: Can We Trust LLMs' Economic Forecasts?  (2504.14765 - Lopez-Lira et al., 20 Apr 2025) in Subsection: Related Literature

These results do not reveal a clear difference between cohorts, but they do not rule out contamination.

— Same Day, Same Story; One Day Ahead, a Different Signal: The Dual Validity of Financial Sentiment  (2609.11144 - Aravinthkakshan et al., 10 Sep 2026) in Section 7, “Robustness,” paragraph “Look ahead contamination”; Appendix, Section “Look Ahead Contamination”