Explain the absence of a task-diversity trend in MLP representations

Determine why increasing the number of pretraining tasks does not produce a similar improvement in linear probe performance for the MLP models in IntWorld and PermWorld, despite the trend observed for Transformers.

Background

The paper evaluates whether mathematical properties excluded from pretraining can be linearly decoded from model representations. For Transformers, probe performance generally improves as the number of pretraining tasks increases, suggesting that broader task diversity produces richer or more accessible internal representations.

The same trend is not observed for the MLP architecture. The authors suggest that the transformer's token-based structure may facilitate representations compatible across tasks, but they explicitly state that the reason for the MLP's different behavior is unclear. Resolving this issue would clarify whether the observed task-diversity effect depends on architectural features specific to Transformers.