Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MMSR: Symbolic Regression is a Multi-Modal Information Fusion Task (2402.18603v5)

Published 28 Feb 2024 in cs.LG, cs.AI, and cs.CL

Abstract: Mathematical formulas are the crystallization of human wisdom in exploring the laws of nature for thousands of years. Describing the complex laws of nature with a concise mathematical formula is a constant pursuit of scientists and a great challenge for artificial intelligence. This field is called symbolic regression (SR). Symbolic regression was originally formulated as a combinatorial optimization problem, and Genetic Programming (GP) and Reinforcement Learning algorithms were used to solve it. However, GP is sensitive to hyperparameters, and these two types of algorithms are inefficient. To solve this problem, researchers treat the mapping from data to expressions as a translation problem. And the corresponding large-scale pre-trained model is introduced. However, the data and expression skeletons do not have very clear word correspondences as the two languages do. Instead, they are more like two modalities (e.g., image and text). Therefore, in this paper, we proposed MMSR. The SR problem is solved as a pure multi-modal problem, and contrastive learning is also introduced in the training process for modal alignment to facilitate later modal feature fusion. It is worth noting that to better promote the modal feature fusion, we adopt the strategy of training contrastive learning loss and other losses at the same time, which only needs one-step training, instead of training contrastive learning loss first and then training other losses. Because our experiments prove training together can make the feature extraction module and feature fusion module wearing-in better. Experimental results show that compared with multiple large-scale pre-training baselines, MMSR achieves the most advanced results on multiple mainstream datasets including SRBench. Our code is open source at https://github.com/1716757342/MMSR

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Improving language understanding by generative pre-training. 2018.
  2. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp.  2425–2433, 2015.
  3. Multiple regression genetic programming. In Proceedings of the 2014 Annual Conference on Genetic and Evolutionary Computation, pp.  879–886, New York, NY, USA, 2014. Association for Computing Machinery. ISBN 9781450326629. doi: 10.1145/2576768.2598291. URL https://doi.org/10.1145/2576768.2598291.
  4. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1):1–43, 2012.
  5. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.  1597–1607. PMLR, 2020.
  6. Pali-x: On scaling up a multilingual vision and language model. arXiv preprint arXiv:2305.18565, 2023.
  7. Debiased contrastive learning. Advances in neural information processing systems, 33:8765–8775, 2020.
  8. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  9729–9738, 2020.
  9. Deep generative symbolic regression. arXiv preprint arXiv:2401.00282, 2023.
  10. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp.  4904–4916. PMLR, 2021.
  11. End-to-end symbolic regression with transformers. Advances in Neural Information Processing Systems, 35:10269–10281, 2022.
  12. Integration of neural network-based symbolic regression in deep learning for scientific discovery. IEEE Transactions on Neural Networks and Learning Systems, 32(9):4166–4177, 2021a. doi: 10.1109/TNNLS.2020.3017010.
  13. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning, pp.  5583–5594. PMLR, 2021b.
  14. Beam search algorithms for multilabel learning. Machine learning, 92:65–89, 2013.
  15. Contemporary symbolic regression methods and their relative performance. arXiv preprint arXiv:2107.14351, 2021.
  16. A unified framework for deep symbolic regression. Advances in Neural Information Processing Systems, 35:33985–33998, 2022.
  17. Set transformer: A framework for attention-based permutation-invariant neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.  3744–3753. PMLR, 09–15 Jun 2019. URL https://proceedings.mlr.press/v97/lee19d.html.
  18. Mopro: Webly supervised learning with momentum prototypes. arXiv preprint arXiv:2009.07995, 2020a.
  19. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04966, 2020b.
  20. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705, 2021.
  21. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp.  12888–12900. PMLR, 2022a.
  22. Transformer-based model for symbolic regression via joint supervised learning. In The Eleventh International Conference on Learning Representations, 2022b.
  23. Metasymnet: A dynamic symbolic regression network capable of evolving into arbitrary formulations. arXiv preprint arXiv:2311.07326, 2023.
  24. Discovering mathematical formulas from data via gpt-guided monte carlo tree search. arXiv preprint arXiv:2401.14424, 2024.
  25. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1-3):503–528, 1989.
  26. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
  27. Snr: Symbolic network-based rectifiable learning framework for symbolic regression. Neural Networks, 165:1021–1034, 2023.
  28. Neural symbolic regression that scales. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.  936–945. PMLR, 2021.
  29. McConaghy, T. FFX: Fast, Scalable, Deterministic Symbolic Regression Technology. Springer New York, New York, NY, 2011. ISBN 978-1-4614-1770-5. doi: 10.1007/978-1-4614-1770-5_13. URL https://doi.org/10.1007/978-1-4614-1770-5_13.
  30. Snip: Bridging mathematical symbolic and numeric realms with unified pre-training. arXiv preprint arXiv:2310.02227, 2023.
  31. Symbolic regression via neural-guided genetic programming population seeding. CoRR, abs/2111.00053, 2021. URL https://arxiv.org/abs/2111.00053.
  32. Surrogate-assisted genetic programming with simplified models for automated design of dispatching rules. IEEE Transactions on Cybernetics, 47(9):2951–2965, 2017. doi: 10.1109/TCYB.2016.2562674.
  33. Petersen, B. K. Deep symbolic regression: Recovering mathematical expressions from data via policy gradients. CoRR, abs/1912.04871, 2019. URL http://arxiv.org/abs/1912.04871.
  34. Answer-me: Multi-task open-vocabulary visual question answering. arXiv preprint arXiv:2205.00949, 2022.
  35. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.  8748–8763. PMLR, 2021.
  36. Distilling free-form natural laws from experimental data. Science, 324(5923):81–85, 2009. doi: 10.1126/science.1165893. URL https://www.science.org/doi/abs/10.1126/science.1165893.
  37. Transformer-based planning for symbolic regression. Advances in Neural Information Processing Systems, 36, 2024.
  38. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  15638–15650, 2022.
  39. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014.
  40. Tibshirani, R. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267–288, 1996. doi: https://doi.org/10.1111/j.2517-6161.1996.tb02080.x.
  41. Ai feynman: A physics-inspired method for symbolic regression. Science Advances, 6(16):eaay2631, 2020. doi: 10.1126/sciadv.aay2631.
  42. Ai feynman 2.0: Pareto-optimal symbolic regression exploiting graph modularity. In Advances in Neural Information Processing Systems, volume 33, pp.  4860–4871. Curran Associates, Inc., 2020.
  43. Symbolicgpt: A generative transformer model for symbolic regression. CoRR, abs/2106.14131, 2021. URL https://arxiv.org/abs/2106.14131.
  44. Symformer: End-to-end symbolic regression using transformer-based architecture. CoRR, abs/2205.15764, 2022. doi: 10.48550/arXiv.2205.15764. URL https://doi.org/10.48550/arXiv.2205.15764.
  45. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In International Conference on Machine Learning, pp.  23318–23340. PMLR, 2022.
  46. Symbolic regression in materials science. MRS Communications, 9(3):793––805, 2019.
  47. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904, 2021.
  48. Coca: Contrastive captioners are image-text foundation models. arxiv 2022. arXiv preprint arXiv:2205.01917.
  49. Evolving scheduling heuristics via genetic programming with feature selection in dynamic flexible job-shop scheduling. IEEE Transactions on Cybernetics, 51(4):1797–1811, 2021. doi: 10.1109/TCYB.2020.3024849.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets