The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers (2108.12284v4)

Abstract: Recently, many datasets have been proposed to test the systematic generalization ability of neural networks. The companion baseline Transformers, typically trained with default hyper-parameters from standard tasks, are shown to fail dramatically. Here we demonstrate that by revisiting model configurations as basic as scaling of embeddings, early stopping, relative positional embedding, and Universal Transformer variants, we can drastically improve the performance of Transformers on systematic generalization. We report improvements on five popular datasets: SCAN, CFQ, PCFG, COGS, and Mathematics dataset. Our models improve accuracy from 50% to 85% on the PCFG productivity split, and from 35% to 81% on COGS. On SCAN, relative positional embedding largely mitigates the EOS decision problem (Newman et al., 2020), yielding 100% accuracy on the length split with a cutoff at 26. Importantly, performance differences between these models are typically invisible on the IID data split. This calls for proper generalization validation sets for developing neural networks that generalize systematically. We publicly release the code to reproduce our results.

• The paper demonstrates that revising embedding scaling, early stopping, and relative positional embeddings can significantly raise accuracy, with improvements such as COGS accuracy from 35% to 81%.
• The paper reveals that employing Universal Transformer variants and shared weight strategies enhances systematic generalization across diverse benchmarks.
• The paper underscores that detailed adjustments in model configurations provide practical pathways for improving compositional reasoning in Transformer architectures.

Insights into Systematic Generalization of Transformers

The paper "The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers" by Robert Csordas, Kazuki Irie, and Jürgen Schmidhuber presents an investigation into the systematic generalization capabilities of Transformer models. The paper reveals that the underperformance of Transformers on systematic generalization tasks can be attributed to neglected model configurations primarily designed for standard tasks like machine translation.

Key Findings

The paper demonstrates that by revisiting fundamental model configurations, including scaling of embeddings, early stopping strategies, relative positional embedding, and Universal Transformer variants, the performance of Transformers can be significantly enhanced across various datasets. The researchers focused on five datasets: SCAN, CFQ, PCFG, COGS, and a Mathematics dataset, showing substantial improvements in model accuracy.

1. Relative Positional Embeddings: The paper highlights the efficacy of relative positional embeddings in addressing the End-Of-Sequence (EOS) decision problem, notably improving accuracy in tasks involving longer sequence outputs, such as SCAN's length split.
2. Improved Training Configurations: The paper details how modifications in training protocols, such as eliminating premature termination of training (early stopping), significantly elevate generalization performance. This adjustment proved particularly impactful on datasets like COGS, with an increase in generalization accuracy from baseline results of 35% to 81%.
3. Universal Transformers: The authors find that sharing weights across layers in Universal Transformers can improve systematic generalization, underscoring a potential area for leveraging recurrent knowledge in recursive tasks.
4. Impact of Embedding Scaling: Different strategies for scaling positional and word embeddings significantly affect model training. Among several techniques evaluated, Position Embedding Downscaling (PED) consistently produced superior results across tasks.

Implications

The implications of these findings are multifaceted:

• Model Design for Generalization: For developing neural models that not only excel in distribution-specific performance but also generalize to compositional scenarios, incorporating relative positional embeddings and considering Universal Transformer architectures are critical.
• Training Strategies: The research underlines the necessity for validation sets that reflect the systematic generalization scenarios to avoid misleading evaluations based on IID data splits — a shift that could refine early stopping strategies to better suit generalization objectives.
• Scaling Guidelines: The paper proposes reconsidering traditional scaling techniques in models, suggesting that adjustments at this level can profoundly impact compositional generalization capabilities.

Future Directions

Future developments could explore advanced architectural innovations leveraging the insights from this paper. Specifically, incorporating analogical reasoning into Transformers could further enhance their ability to generalize compositions. Additionally, refining existing models with the insights regarding configuration sensitivity might yield new state-of-the-art results in solving algorithmic reasoning problems.

In conclusion, the paper provides an illuminating view into how seemingly simplistic modifications in Transformer configurations can lead to substantial improvements in performance on systematic generalization tasks, outlining critical paths forward for research and practical applications in neural network design.