Enigme: Generative Text Puzzles for Evaluating Reasoning in Language Models (2505.04914v1)

Abstract: Transformer-decoder LLMs are a core innovation in text based generative artificial intelligence. These models are being deployed as general-purpose intelligence systems in many applications. Central to their utility is the capacity to understand natural language commands and exploit the reasoning embedded in human text corpora to apply some form of reasoning process to a wide variety of novel tasks. To understand the limitations of this approach to generating reasoning we argue that we need to consider the architectural constraints of these systems. Consideration of the latent variable structure of transformer-decoder models allows us to design reasoning tasks that should probe the boundary of their capacity to reason. We present enigme, an open-source library for generating text-based puzzles to be used in training and evaluating reasoning skills within transformer-decoder models and future AI architectures.

Enigme: Evaluating Reasoning in LLMs with Generative Text Puzzles

The paper "Enigme: Generative Text Puzzles for Evaluating Reasoning in LLMs" presents an innovative methodological approach to assessing the reasoning capabilities of transformer-decoder LLMs. The research explores the architectural constraints of LLMs and posits a library of generative text puzzles designed to serve as benchmarks in evaluating these models' reasoning abilities. This is particularly critical as the models transition to playing increasingly central roles in diverse AI applications.

Overview

The foundational proposition of the paper is that by examining the latent variable structures in LLMs, researchers can develop tests that probe the boundaries of reasoning capabilities in these models. The authors introduce enigme, an open-source library that generates various text-based puzzles intended to challenge LLMs and future AI systems. These puzzles are procedural in nature, focusing on abstract reasoning and world-model inference through three distinct categories: numeric puzzles, sequence puzzles, and physics puzzles.

Methodology

The methodology involves generating puzzles that exploit the known strengths and limitations of the transformer architecture; this involves understanding its embedding space and sequential token processing. Each class of puzzles targets different aspects of reasoning:

1. Numeric Puzzles: Require agents to engage in arithmetic operations with self-referential tasks derived from text blocks, challenging the model's numerical reasoning.
2. Sequence Puzzles: Test the model's ability to recognize patterns and infer sequences through tasks resembling those in traditional IQ tests.
3. Physics Puzzles: Engage concepts of intuitive physics such as momentum or collision, demanding that models extrapolate behavioral patterns in objects over time and space.

These variations are enhanced by randomization and parametrization to mitigate possible memorization by the LLMs.

Implications and Future Directions

The implications of this research are significant for several reasons. Firstly, it provides an alternative benchmark to assess the reasoning capabilities of LLMs without reliance on traditional textual content that may have been part of models' training data. This approach addresses the ongoing challenges in distinguishing between emergent reasoning and memorization. Secondly, by focusing on abstract and numeric pattern recognition, enigme could advance methodologies for training LLMs in tasks requiring higher cognitive functionality.

Future developments could explore additional puzzle types and investigate the implications of these benchmarks on different AI system architectures. As LLMs evolve, extending the toolkit to assess reasoning will be pivotal in ensuring robust evaluations of AI systems' cognitive abilities.

Conclusion

The paper refrains from dramatizing its contributions but provides a robust framework for the objective evaluation of reasoning in LLMs. By deploying precisely structured challenges, the enigme library aims to shed light on the capabilities and limitations of LLMs in reasoning tasks—a necessary step in realizing general-purpose AI. The authors also underline the need for continued adaptation and expansion of evaluative tools, advocating for a more nuanced exploration of AI architectures and their emergent properties. This work is, therefore, an essential reference for researchers focused on the theoretical and practical aspects of AI reasoning.