Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning (2004.12193v1)

Abstract: As a comprehensive indicator of mathematical thinking and intelligence, the number sense (Dehaene 2011) bridges the induction of symbolic concepts and the competence of problem-solving. To endow such a crucial cognitive ability to machine intelligence, we propose a dataset, Machine Number Sense (MNS), consisting of visual arithmetic problems automatically generated using a grammar model--And-Or Graph (AOG). These visual arithmetic problems are in the form of geometric figures: each problem has a set of geometric shapes as its context and embedded number symbols. Solving such problems is not trivial; the machine not only has to recognize the number, but also to interpret the number with its contexts, shapes, and relations (e.g., symmetry) together with proper operations. We benchmark the MNS dataset using four predominant neural network models as baselines in this visual reasoning task. Comprehensive experiments show that current neural-network-based models still struggle to understand number concepts and relational operations. We show that a simple brute-force search algorithm could work out some of the problems without context information. Crucially, taking geometric context into account by an additional perception module would provide a sharp performance gain with fewer search steps. Altogether, we call for attention in fusing the classic search-based algorithms with modern neural networks to discover the essential number concepts in future research.

• The paper introduces the Machine Number Sense (MNS) dataset, generated using AOG, featuring visual arithmetic problems to evaluate machine abstract and relational reasoning.
• Experiments show current neural networks and symbolic search models struggle with associating visual context and abstract mathematical meaning compared to human performance.
• The dataset highlights the need for future hybrid AI approaches combining data-driven methods with knowledge-based search to improve abstract reasoning.

The paper "Machine Number Sense: A Dataset of Visual Arithmetic Problems for Abstract and Relational Reasoning" presents an innovative dataset designed to evaluate machine intelligence in the context of mathematical reasoning, specifically emphasizing visual arithmetic tasks. This dataset, referred to as MNS (Machine Number Sense), is composed of visual problems that are automatically generated using AOG (And-Or Graph), a context-free grammar framework commonly applied to hierarchical and compositional data.

Dataset Structure and Problem Types

The MNS dataset comprises arithmetic problems structured in the form of geometric figures, where problems are contextualized by geometric shapes and contain embedded number symbols. The types of problems included are:

1. Combination Problems: These involve grouping two or three geometric shapes using spatial relations such as overlapping or including.
2. Composition Problems: These include arrangement of geometric shapes to form larger shapes, allowing interpretations based on holistic or analytic problem-solving perspectives.
3. Partition Problems: These involve dividing a shape into several parts, with assessment based on how the shapes are partitioned.

The dataset generation involves two core components: the layout component, which provides the context through geometric shapes, and the algebra component, defining the arithmetic operations and integer constants involved.

Methodology and Experiments

Comprehensive experiments were conducted using four predominant baseline models: CNN, LSTM, ResNet, and RN. These models were used to evaluate their ability to interpret visual patterns and reason about number symbols and operations. The findings highlighted the limitations of current neural networks, which largely fail to bridge the cognitive gap between recognizing visual symbols and understanding their abstract mathematical meanings. Specifically, models struggled with forming associations between visual contexts and numerical problem-solving.

In addition, the paper employs search-based algorithms to explore a different approach to problem-solving. Two symbolic search models were implemented: pure symbolic search and context-guided search. While pure symbolic search relies on processing only the numbers, context-guided search incorporates additional contextual information, thereby achieving better performance. Nevertheless, the performance of these search-based models exposed their inefficiency, particularly in handling problems with larger search spaces.

Human vs. Machine Performance

Human performance exhibited significantly higher accuracy compared to both neural network and search-based models, emphasizing the inadequacies in current AI methods for understanding number sense. Humans excelled in both holistic and analytic interpretations across various problem types, suggesting that machines have much to learn from human cognitive processing.

Implications and Future Directions

The paper calls for future research to focus on hybrid approaches combining data-driven methods, such as neural networks, with knowledge-based search algorithms to enhance abstract reasoning and symbolic understanding. The integration of these methods aims to leverage visual feature extraction capabilities alongside structured problem-solving strategies.

In conclusion, the MNS dataset provides a foundational step towards exploring machine intelligence in mathematical reasoning, setting the stage for developing more sophisticated AI systems capable of bridging the gap between visual perception and abstract number processing.