Relevance of memorization to LLM-elicited survey responses and expectations

Ascertain whether and to what extent memorization affects applications that use large language models such as GPT-4o to elicit survey-style responses or generate expectations in economic and finance research, particularly when knowledge of future outcomes could influence the model’s answers.

Background

Beyond forecasting tasks, LLMs are increasingly used to simulate survey responses and generate expectations. The paper highlights uncertainty about whether the memorization problem meaningfully affects these applications, which may be sensitive to future knowledge in ways that are difficult to detect.

Clarifying the relevance and magnitude of memorization effects in survey-style elicitation is important for the validity of research that seeks to model beliefs, expectations, or preferences using LLM outputs.