- The paper introduces a symbolic regression method using vision LLMs to propose function forms from plots and KANs to extend this to multivariate functions based on visual input.
- The LLM-LEx method is competitive with classical algorithms, often finding exact function expressions in benchmark tests without requiring pre-defined basis functions.
- The method exhibits simplicity bias, handles moderate noise well, and shows potential for improvement with advanced models and domain-specific prompts.
Symbolic Regression with Multimodal LLMs and Kolmogorov–Arnold Networks
The paper "Symbolic Regression with Multimodal LLMs and Kolmogorov–Arnold Networks" introduces an innovative approach for conducting symbolic regression by leveraging the capabilities of LLMs and Kolmogorov–Arnold Networks (KANs). Symbolic regression has been a challenging task due to the vast search space of possible functions that characterize any given dataset. Traditionally, this task relies on algorithms that seek to minimize model complexity and mitigate expression "bloat." However, the authors propose a new method inspired by the intuitive capabilities humans demonstrate when inferring functions' forms from visual data.
The core methodology involves using vision-capable LLMs to make symbolic regression more effective. The LLMs are presented with plots of univariate functions and tasked with proposing an ansatz. Following this, free parameters within the proposed expressions are fitted using standard numerical methods, generating a population for a genetic algorithm. This approach uniquely does not require an initial specification of function sets for regression due to the LLM's generative capabilities conditioned through prompt engineering.
A significant innovation is the use of Kolmogorov–Arnold Networks, which facilitate extending the symbolic regression method to multivariate functions. KANs leverage the Kolmogorov-Arnold representation theorem, suggesting multivariate functions can be expressed as sums and compositions of univariate functions. This reduces the multivariate function problem to several simpler univariate problems. The LLM-LEx package implemented employs commercially available LLMs to guide the regression process.
Comparative analysis to classical methods, such as genetic algorithms or heuristic searches over symbolic expressions, demonstrated that LLM-LEx is highly competitive, frequently finding exact expressions in benchmark tests. Unique characteristics of this model include the absence of enforced basis functions, an innate simplicity bias, and quick adaptability through prompt modifications to tailor to specific needs, such as recognizing special functions.
Furthermore, an exploration of open-source LLMs showed reasonable approximation performance, albeit with increased time costs due to computational limitations compared to proprietary models like gpt-4o.
One interesting finding is the method's robustness to noise; the fusion of visual inputs and symbolic logic permits a graceful handling of moderate noise, albeit with potential overfitting risks as noise levels increase. The simplicity bias manifested itself in the method's ability to focus on the dominant functional form, even under noisy conditions.
The paper adds valuable perspectives to the field, suggesting that as AI technology advances, the limitations observed may be overcome through algorithmic enhancements such as advanced genetic algorithm designs and vision transformers specifically trained for symbolic regression tasks. Additionally, domain-specific prompt engineering could further guide the proposal of relevant function forms.
Lastly, the integration of univariate focus with KANs extends the applicability of LLM-based symbolic regression to complex multivariate function scenarios, paving the way for further exploration and potential breakthroughs in algorithm efficiency and interpretability in scientific domains.