Residual pre-training bias and hidden prompt effects in LLM-based trading evaluation

Ascertain whether using evaluation periods outside an LLM’s knowledge cut-off suffices to eliminate pre-training-induced biases in LLM-based trading agents, and determine the extent to which undisclosed system prompts in proprietary APIs modify model outputs in ways that could affect trading decisions.

Background

The authors note that pre-training data may bias LLM-driven trading even when evaluations use data outside the models’ knowledge cut-offs. They further point out that non-transparent system prompts in some APIs may alter model behavior in ways evaluators cannot observe.

To mitigate these confounders, the paper employs Simudyne Horizon to generate synthetic price paths and news cycles, but the text explicitly states uncertainty about fully discounting pre-training biases and undisclosed system prompt effects.