Deep Symbolic Regression for Recurrent Sequences (2201.04600v2)

Abstract: Symbolic regression, i.e. predicting a function from the observation of its values, is well-known to be a challenging task. In this paper, we train Transformers to infer the function or recurrence relation underlying sequences of integers or floats, a typical task in human IQ tests which has hardly been tackled in the machine learning literature. We evaluate our integer model on a subset of OEIS sequences, and show that it outperforms built-in Mathematica functions for recurrence prediction. We also demonstrate that our float model is able to yield informative approximations of out-of-vocabulary functions and constants, e.g. $\operatorname{bessel0}(x)\approx \frac{\sin(x)+\cos(x)}{\sqrt{\pi x}}$ and $1.644934\approx \pi^2/6$. An interactive demonstration of our models is provided at https://symbolicregression.metademolab.com.

• The paper introduces a Transformer-based framework that infers symbolic recurrence relations from sequences, achieving superior performance over established methods.
• It demonstrates that the models generalize well to out-of-vocabulary and out-of-domain patterns, accurately extrapolating complex recurrence relations.
• The study underscores symbolic regression's advantages over numerical approaches, promising enhanced computational tools for mathematical analysis.

An Analytical Perspective on Deep Symbolic Regression for Recurrent Sequences

The paper "Deep Symbolic Regression for Recurrent Sequences" addresses a significant challenge in machine learning, namely symbolic regression, particularly focusing on the task of predicting the underlying functions of integer or float sequences. This task, reminiscent of problems seen in human IQ tests, is approached by training Transformers to infer the recurrence relations that define these sequences. The paper distinguishes itself by evaluating the proposed models against the Online Encyclopedia of Integer Sequences (OEIS), showcasing them as outperforming established mathematical software like Mathematica in predicting recurrence relations.

The authors propose a methodological framework where neural networks, particularly Transformer architectures, are trained on large datasets composed of synthetic sequences generated by random recurrence relations. Each sequence or function representation is formulated as a symbolic expression, and the network's task is to learn these symbolic representations and use them for subsequent term predictions in a sequence. This approach allows for a deeper understanding of sequences, akin to understanding discrete time differential equations in systems.

Key Results and Findings

1. Performance Metrics:
   • On integer sequences from OEIS, the model demonstrated superiority over Mathematica's built-in functions both in recurrence prediction and sequence extrapolation. The integer model correctly predicted complex recurrence relations and made precise extrapolations beyond the initially observed terms.
2. Robustness to Out-of-Vocabulary Functions:
   • The float model presented an outstanding capability to approximate out-of-vocabulary functions and constants. For instance, the model approximated $\operatorname{bessel0}(x)$ with $\frac{\sin(x)+\cos(x)}{\sqrt{\pi x}}$ and accurately related $1.64493$ to $\pi^2/6$, showcasing its potential in expressive and complex reasoning tasks.
3. Generalization and Approximation Abilities:
   • The models were tested on several out-of-domain sequences, including OEIS integer sequences and synthetic float sequences with out-of-dictionary components. Despite the training constraints, the models generalized well, maintaining functional accuracy in predicting sequence continuations under various difficulty settings.
4. Numerical and Symbolic Contrasts:
   • A comparative analysis between symbolic and numerical models showed that symbolic models are more effective in retrieving explicit mathematical relations, whereas numerical models excel in direct sequence continuation but lack symbolic reasoning capability.

Implications and Future Directions

The implications of this paper are twofold. Practically, it suggests a novel method of sequence analysis which could improve symbolic computation tools, ultimately enhancing disciplines like mathematics education and computational intelligence. Theoretically, it opens avenues for incorporating complex symbolic reasoning tasks within machine learning frameworks, potentially extending to tasks like discovering novel mathematical identities or solving advanced differential equations.

Looking forward, the research suggests the following enhancements:

• Incorporating Higher Precision: To handle finer subdominant terms in expressions, further development could explore improved precision encoding. This might involve multi-token representation or iterative refinement techniques.
• Expansion to Multi-Dimensional Sequences: Extending the methodology to handle multi-dimensional or complex sequences could bridge gaps between discrete and continuous systems, aligning more closely with real-world data.
• Differential Equations and Dynamic Systems: The methods might be expanded to infer differential equations from dynamic systems, further bridging the gap between machine learning and classical fields of science and engineering.

In conclusion, "Deep Symbolic Regression for Recurrent Sequences" demonstrates a robust approach to disentangling sequence patterns through deep learning. By leveraging symbolic regression within the Transformer architecture, it pushes forward the capabilities of artificial intelligence to reason beyond conventional numerical tasks, engaging deeply with the symbolic domain.