Logical reasoning capability of Transformer-based language models

Ascertain whether Transformer-based language models possess genuine logical reasoning abilities—including multi-step reasoning, handling long contexts, learning abstractions, and hierarchical planning—when applied to formal mathematical tasks.

Background

While Transformers excel at memorization due to large-scale pretraining, the paper catalogs evidence of reasoning failures and limited generalization, raising doubts about their intrinsic logical reasoning capabilities.

The authors propose formal mathematics as a controlled domain to investigate and decisively test whether Transformers can truly reason logically rather than merely imitate patterns.