Intrinsic transformer mechanisms behind narrative-prompt prediction gains

Investigate whether an intrinsic property of transformer-based language models produces improved predictive accuracy under narrative prompting independent of OpenAI usage policy constraints, and, if so, characterize the underlying mechanism—potentially involving hallucination processes within attention.

Background

The authors consider whether the superior performance under narrative prompts is merely a byproduct of policy constraints on direct prediction or instead reflects deeper, architecture-level dynamics within transformers.

They hypothesize that interactions between hallucination tendencies and attention mechanisms may play a role but explicitly acknowledge they cannot substantiate this beyond speculation given the limited scope of models examined.

References

Another explanation, though, is that there is something intrinsic to the narrative prompting that allows the Transformer architecture to make more accurate predictions even outside of the confounding set by OpenAI's terms of service. This may be related to how the hallucination fabricrations work within the machine learning environment of attention mechanisms. But as we only studied the two OpenAI GPT models, we are unable to provide more than just speculation as these terms of use violations are always present if that is indeed the case.

— Can Base ChatGPT be Used for Forecasting without Additional Optimization? (2404.07396 - Pham et al., 11 Apr 2024) in Section: Conjecture on ChatGPT-4's Predictive Abilities in Narrative Form