Determine whether the transformer has learned a general algorithm

Determine whether the transformer models trained on words in smaller symmetric groups, including the reported models trained on S_10 and tested on S_16 or S_25, have discovered a general algorithm for predicting the corresponding permutations from transposition words.

Background

The paper trains transformer neural networks on permutation words from the smaller symmetric group S_10 and evaluates their ability to predict permutations in the larger groups S_16 and S_25. Although the models achieve nearly perfect test accuracy, the authors report that they did not observe grokking, the transition commonly interpreted as evidence that a model has moved from memorization to learning an underlying algorithm.

Consequently, the observed out-of-distribution success does not establish whether the models have learned a size-independent procedure for evaluating transposition words or have instead exploited another mechanism that happens to generalize to the tested group sizes. The authors explicitly leave this question unresolved.