Cause of GPT-4’s notably strong performance in the Matter at Extremes module

Investigate the factors underlying GPT-4’s strong performance on assessments in the Matter at Extremes module (covering topics such as particle colliders and superconductivity), specifically testing whether this is due to a limited variety of canonical question forms or to fortuitous alignment with the distributions present in GPT-4’s training data.

Background

In the Matter at Extremes module, GPT-4 achieved a first-class overall mark (85%), performing strongly on computational and explanatory tasks despite persistent issues with diagrammatic questions and some equations embedded as images. This contrasts with weaker performance on multi-step reasoning tasks in other modules.

The authors explicitly state uncertainty about why GPT-4’s answers were of unusually high quality in this module, suggesting possible explanations such as limited question diversity or alignment with the model’s training data, and thereby motivating a targeted investigation.